Why your local LLM feels dumber than it is

文章洞見

原文內容目前無法安全取得，因此未產生文章摘要。

討論洞見

討論聚焦於「本地 LLM 為何常顯得比雲端模型笨」以及如何正確評估與部署。與談者普遍指出，差異不一定來自模型本身，也可能源於 quantization、KV cache、chat template、sampling、context 管理、inference engine、agent harness 與硬體配置；同時，本地模型在隱私、可控性與特定任務客製化方面仍具吸引力。

共識：大致共識是：不能只以「模型名稱」比較效能，必須明確記錄 exact weights、quantization 方法、chat template、sampling 參數、KV cache 設定、context 長度、runtime 與 harness，並以可重現的測試評估。惟不同硬體、任務與部署方式造成的結果差異很大，因此對「哪個 runtime 或 quantization 一定最好」沒有充分一致的證據。

實務建議

• 比較模型時記錄完整 reproducibility metadata：原始權重版本、quantization 格式與來源、chat template、sampling 參數、KV cache precision、context length、runtime、GPU/CPU 與 harness。
• 先檢查 GGUF 或 runtime 是否正確載入官方 chat template，再檢查 sampling defaults；不要只因模型輸出差就立即歸咎於模型或 quantization。
• 對長上下文與 agentic workflow，加入 parser、encoding、tool-call、context eviction 與 memory budget 的 health checks 和 regression tests。
• 不要把 BF16、Q8、Q4 或不同格式的模型視為同一個可直接比較的 artefact；每種 quantization 都應以目標任務重新測試。
• 建立小型但固定的 domain-specific evaluation set，例如實際 codebase、歷史 bug、工具呼叫與多輪 session，避免只依賴網路 benchmark。
• 若追求隱私或成本效益，可採用 hybrid workflow：雲端模型負責分析或產生規格，本地模型負責測試、驗證或處理敏感資料。
• 若本地部署的設定複雜，應優先查閱官方 model settings 與 runtime 文件，並明確區分 global 與 per-model configuration。

待釐清問題

• 雲端供應商是否會動態調整 KV cache eviction、memory budget、batching 或 model serving configuration，以及這些變化對跨時段品質的實際影響，討論中沒有可靠答案。
• 不同 inference engine 與同一模型權重在品質上的差異，尚缺乏控制 quantization、template、sampling 與硬體後的大規模 head-to-head benchmark。
• KV cache compression/quantization 對長上下文 reasoning 的影響仍需要更多公開、可重現的評測，尤其是不同模型是否曾接受 quantization-aware training。
• 針對特定公司 codebase 以 RLVR 或其他 post-training 方法客製化模型，與單純改善 harness、context retrieval 或 prompt 的成本效益比較，尚未定論。
• 本地模型在特定任務上超越 frontier closed models 的條件，以及這種優勢能否穩定泛化到不同資料與工作流程，仍缺少一致證據。

代表性留言

1. I thought it was a link too because of the line under the with math but it isn’t. :/

   a1o · HN #49404548

2. Especially when there are many ways to do the quantization. You can get quite different results from different methods.

   RevEng · HN #49411947

3. There's quite a few tangential features that must be implemented correctly or risk affecting the LLM output in significant ways.Parsing/encoding is one example: A couple of months ago I've debugged a reasoning loop bug in Step 3.7 Flash on llama.cpp that was caused by the parser capturing an extra `\n` as part of a reasoning block. It was something that only manifested at longer multi-turn agentic sessions, and the extra linefeed was steering the model into making reasoning self corrections that only got worse with longer sessions (more details about this issue: https://github.com/ggml-org/llama.cpp/issues/24181#issuecomm...)No inference engine is perfect, but I feel that llama.cpp is the most reliable way to run language models locally.

   tarruda · HN #49407638

4. Meanwhile, entirely too frequently I see unit tests that have atol / rtol values which permit errors of one part in 200 or even higher, where people bother with unit tests at all. At some point someone in the pytorch ecosystem used rtol=5e-03 in a unit test and crowds of people started passing that around unquestioned.With attention matrix sizes being what they are, that's high enough that you can literally zero out a row or two and still have the test suite pass. Guess how I know!Check your numbers, folks.

   tsukikage · HN #49410005

5. Debugging any LLM output when you also have done substantial harness engineering is a total pita and I wish there were better tools for it to isolate issues.I spent ages tracking down start appears to be an issue with the current Deepseek v4 flash 0731 version that would cause it to output giant walls of gibberish in Hermes with reasoning turned on.

   shostack · HN #49409020

6. I saw some guy streaming how he was deploying qwen3.8 37B on his local setup. Well, he was asking Claude to do it. It took him two hours of passing errors to Claude for the endpoint to start working, he then started testing it against DS4 Flash when Qwen had thinking disabled and Claude messed up sampling parameters, it was an absolute pain to watch

   big-chungus4 · HN #49406454

7. Wow. I tried to get Qwen3.8 4B to parse song lyrics and analyse them. Getting ollama running was a minute or two.However coming up with a prompt that didn't turn out total garbage was impossible. After wasting over an hour and I ended up getting Qwen side by side with Llama 3.2 3B, just to see if I was being stupid. Nope, it just looks like Llama is orders of magnitude better at this specific task for some reason).If you think I'm doing it wrong, you're probably right, I don't know a ton about local LLMs. But I hand selected 50 songs, set up ollama with both LLMs, and for each iteration on the prompt text, ran both LLMs 10x times per song. Side-by-side comparisons showed that Qwen 3 4B was so bad that I actually downloaded Qwen again, thinking there must have been some mistake and I accidentally grabbed an old 1B model.

   raffraffraff · HN #49407025

8. You could do this for sure but its quite a lot of effort and a normal software stack is quite well known due to the massive amount of github projects and other opensource code.You can also get a lot out of a harness in this case or your Agents.md or Claude.md file by just enhancing the context.You might even question yourself if you are doing something wrong if a modern LLM really struggles with your code.We do the finetuning only on small semantic data were it helps a lot.I'm still wondering when we see smaller models (faster and cheaper) for more specific stacks like spring boot + java + angular + english only or so. Interstingly enough, i assumed LLMs are really good in any language but it seems that non english languages do reduce the ooutput quality of an LLM. At least last years GTC there was a talk about it.There are companies though which ahve this exact problem with programming languages you normally don't see. ABAP for example is a very well known SAP language.

   Zylokloto · HN #49408082

9. I wonder the same thing. Code is pretty open ended though so I wonder if it’s not the best example.On the one hand it would definitely be useful for something like classifying support requests into priority. But would it be worth it to just train your own model? I guess one advantage is you could give it some well informed guidelines without training something on lots of data.There is probably a better example between discrete labeling and code though.

   nonethewiser · HN #49408282

10. That's funny, because I just went through the opposite.I got llama.cpp working with qwen3.6 and qwen3.8 by Googling and manually adjusting things according to reddit posts and Google not-really-helpful AI suggestions.I tried settings up per-model stuff in settings.json, but again Google got in my way, and llama.cpp having 2 different settings.json (and Google lying about where 1 goes) made it far too difficulty to figure out. I spent hours on it.Then I got fed up and asked Claude.Immediately, it told me that the winget version of llama.cpp is for Vulkan, and I needed a different one and pointed at it. It doubled my speed.Then it figured out what I was doing wrong with settings.json (wrong spot, global settings can't go in the per-model file, etc etc) and fixed all that, and got it working.Then it tuned it somewhat.Then I showed it the official settings pages for both models, and it undid the tuning and all the damage I had done with my tinkering, and got everything working.In 30 minutes.It was absolutely amazing.Every time I see people recommending Qwen locally with llama.cpp, they just say "download it" and act like anyone that can't get it running is an idiot. But if there's a "using this settings.json" tutorial somewhere, I didn't find it, and neither did Google over a week of searching.But Claude got it done for me.Now, I admit, I haven't played with it much. Just before all this, I ran out of Claude on the $20 plan and bumped up to $100, and It's been so amazing that it's really hard to work on the local. Especially since it feels like Qwen3.8 35b a3b is probably around the corner, and why mess with 3.6 when 3.8 will probably release soon?

    wccrawford · HN #49407702

11. I just got qwen 3.8 27b mlx running on my Macbook Pro and honestly I’m pretty blown away by how not-dumb it is.

    jonplackett · HN #49403412

12. It was actually great. I have like a non-AI box so to speak 8GB VRAM, co-incidentally from a gaming PC ...All the previous models that were "frontier level, just try it!" but wouldn't run at all in agentic mode, including previous Qwens, just disappointed, period.Then I ran then Qwen 3.8 27b and while it was super slow (4t/s) it literally one-shotted creating a usable "web search/pull" skill for `pi.dev`. while any other model previously just entirely failed to create anything usable even with actual guidance.Since then I have actually gotten a gemma-4 12B qat 4bit quantized with a ~250MB MTP from unsloth to work with a 32k context "working" on this setup at 80-120 t/s. That's usable for private stuff on a co-incidental box!It's still only 32k context and it's entirely dumb vs. our API paid at-work Claude Opus. But for entirely private local stuff it's totally workable without breaking the bank even after all these AI price hikes!. I bought this rig literally just for gaming a month ago.

    tharkun__ · HN #49405624

13. That's funny, I downloaded the same model on my 48GB M4 Pro and gave it a problem to solve in an existing codebase, it spun its wheels for twenty minutes and then fell over dead. This was using LMStudio and pi as a harness; I never use pi for anything else, so maybe I'm holding it wrong.

    velcrovan · HN #49405907

14. Even a 4-bit quant of Qwen3.8 27b is indistinguishable from Gemini 3.7 flash in our internal tests. With an RTX5090 card and ninfer, you can get ~800 TPS token generation (c=8) and ~140 Tokens per second single stream.

    a11r · HN #49405215

15. Much of this is why I stick to the rule of:a) Don't quantize your KV cacheb) Don't run quantizations of the LLM that are worse than the best available Q8 (the largest possible file size unsloth GGUF for a given model like qwen 3.8 27B as an example). I would rather things go slowly but I have confidence that it's doing things more accurately.

    walrus01 · HN #49405060

16. Comments are mostly showing off M5s and 5090s without addressing the article.

    utopiah · HN #49406476

17. I’m running Qwen3.8 aggressive uncensored Q4_K_P on a 4090 in a loop against the 2026 CrackMe CTF challenges.Using oh-my-pi in a prebuilt environment that I let Qwen build too.Codex wouldn’t even look at the files - literally, as soon as it read something with CTF it shut down. Didn’t even offer to fall back to a dumber model.

    InvertedRhodium · HN #49404557

18. This is similar to my use case as well. I’ll use a cloud model to identify security issues in a codebase and then use a report/spec output as input to the local model to build tests or verify the issue is real.

    treesknees · HN #49406009

19. How's it performing on the challenges?

    CamperBob2 · HN #49404687

20. At least I'd be in control of model quality vs. when Anthropic decides to randomly drop the quality of their offering

    nullpoint420 · HN #49405385

21. It is good that someone is having such a deep look. This is not exclusive to LLMs in the least. Every non-trivial program depends on hundreds of little details being correct.That is also why I recommend including health checks in such programs. Some function that checks that all assumptions hold. That could be an endpoint, an automated test, a periodic diagnostic job, etc...

    ThouYS · HN #49407986

22. For people who want to quantize kv caches for longer context, basically, if the model checkpoint you use didn't trained to use quantized kv caches (and all most all of model checkpoints you can access didn't), you shouldn't quantize kv caches during inference. A major benefit to do quantization-aware training/distillation on official checkpoints is to let the model to be familiar with quantized activations/weights/kv caches.

    RandyOrion · HN #49414932

23. Somewhat off topic, but I've started wondering if we can actually make local LLMs feel smarter than the frontier closed models, by post-training it for your specific use case.Say Company X has a software product which consists of a million lines of code, including a ticket for every bug and new feature for this piece of software. Then wouldn't it make sense to try using an open weight model but post-train it on that specific code base while using the tickets to teach the model about past bugs and features. Not sure exactly how, but it could involve doing reinforcement learning solving a past bug on that historic version of the code base, and rewarding the model if it comes up with the correct solution (as defined by the linked PR which fixed the bug).So essentially post-training your local open-weight model using reinforcement Learning with Verifiable Rewards (RLVR) on your software products history of bug reports and their ultimate solution. And the same for new features.

    runeks · HN #49407881

24. > I will make you read the really long unpleasant version with math.This is the version I want to read :)I assume it is unpleasant in spite of the math, not because of it?

    catlifeonmars · HN #49404221

25. So to what extent does this apply to cloud hosted LLM’s? Are there benchmarks that score models across cloud providers? My experience using LLM’s during day time vs evening sessions has felt “night and day” and I’ve chalked that mostly up to it must be my imagination or just the general indeterministic nature of LLM’s. Sessions resumed after a day away also feel “dumb” sometimes so I can see an aggressive kv cache eviction policy playing a role if it’s reasonable to extrapolate what the article is saying about local inference.Are things like kv cache eviction policies and memory budgets shipped with recommended configurations based on the hardware and software serving the inference requests? or are they configured dynamically by the cloud provider hosting the model to manage multi-tenant load?

    heywoods · HN #49406678

26. > And the comparisons in this post are not going to be running some 2.58-bit-gguf-in-ollama with a couple test prompts.Genuine question : is there something fundamentally wrong with Ollama ?I use Ollama because it is easy to set up and manage (and also because VLLM is not super Windows friendly).I thought the main advantage of VLLM was better concurrency management (better batching).But if the quality of the interference itself is an issue, then maybe I should reconsider my choice.

    JacobJack · HN #49404004

27. People who use Ollama generally (not everyone obviously) don't always clearly understand what quantization they use when running models, so people end up saying "I tried running Qwen 3.8 27b locally and it was dumb" while Ollama would default to a Q4 version of the model, which has very different results from the BF16 weights, doesn't really speak to the model itself because it's been so quantized in that case.Sure, makes things easier, but tons of people misunderstand what they're using, then base and share their experiences on that, without really specifying what exact weights they use too.For a single local user, using llama.cpp directly shouldn't be a problem if you're already using Ollama's CLI, it works basically the same except you manage weights yourself, and if you put your favorite agent to make sense of the faux "registry + image layers" Ollama has prepared locally for you, you can reuse the files you've already downloaded with Ollama.

    embedding-shape · HN #49404223

28. From what I've heard, Ollama has a bad reputation because it's a thin wrapper around llama.cpp without attributing it properly, thereby stealing recognition from the maintainers doing most of the work

    kangalioo · HN #49404065

29. > Both the NVFP4 and AWQ W4A16 failed to properly close their tool calls ...If I understand correctly, this failure mode is just not possible with llama.cpp / ik_llama.cpp, which enforces token generation to follow the grammar once a tool call is detected.> ... and botched Cisco command line syntax (the correct command was ‘show arp’, while they executed ‘show run’)But this failure mode can still happen.Anyway, NVFP4 and AWQ W4A16 are generally regarded as low quality quants. IQK/Trellis quants from ik_llama.cpp and EXL3 quants from exllama should work better.So, perhaps the lesson here is "don't use vllm at home"?

    throwdbaaway · HN #49405490

30. As for KV cache quantization, Q8_0 from llama.cpp / ik_llama.cpp should also work better than FP8 from vllm (see https://github.com/vllm-project/vllm/issues/33480#issuecomme...).

    throwdbaaway · HN #49405512

31. "Why LLMs ARE dumber than they appear" is much closer to the reality I live in.

    mkhalil · HN #49406396

32. The section on system prompts and context window management is spot on; most people don't realize how much the default quantization in popular runners degrades logic compared to full FP16. I'd be curious to see if the author has benchmarked the impact of KV cache compression on longer context reasoning, as that usually seems to be where my local Llama 3 setup starts to fall apart.

    fenestella · HN #49405481

33. I have been using open source LLMs locally for the past 5 months like Qwen, Llama, DeepSeek and others, and I have also noticed that current local models feel significantly dumber than closed source commercial models like ChatGPT, Claude, and Gemini. One main reason I think is the amount, variety, and quality of original authentic data on which they are being trained on, and also the training method plays a significant role in the performance difference between local open source models and closed source commercial models.

    synthrakx · HN #49407916

34. They are also an order of magnitude difference in number of parameters and that matters a lot.

    RevEng · HN #49411929

35. sglang, 150+ tok/s on a 5090 in ubuntu 26.04 via wsl. gittensor-model-hub/Qwen3.8-27B-NVFP4-RTX5090, dspark, medium reasoning, 96k context.Using opencode and it built a old fashioned arcade vertical shooter with no issues.Images are ok'ish, just had grok create updated images, and it came out great.

    IronWolve · HN #49404821

36. After 3 years of running local models I think the model which fits unquantized (BF16) in the VRAM is the best model for general purpose tasks; fine-tuned SLMs or utilities based on non language models for solving a specific problem (e.g. TTS,STT,RMBG etc.) have been the best use of local AI for me.Local models for coding, is just not worth the effort IMO; unless of course you have the hardware to fit it unquantized in your VRAM.

    Abishek_Muthian · HN #49407454

37. I believe whoever lays down the best structure around LLM will take the lead. Proven in Anthropic vs OpenAI.

    OsamaMustafaa · HN #49407674

38. The problem is benchmarking. Not everyone has a 500k token workstream of the model they are setting up for the first time to run it against 10 different config and compare differences.And if you download benchmarks from the net they are likely poisoned by models being trained on them.

    Roark66 · HN #49407892

39. Jeez, I thought I could get away with q8_0 KV cache. Guess not.

    arcanemachiner · HN #49406517

40. I think it's maybe because we:a) Load it up over time with skills and mcp servers and other junk b) We start to ask it ridiculously complicated tasks because we've normalized the power so we scale our expectations.

    freepiai · HN #49410526

41. "I can't wait to run this new sota model locally. I'll just use the quantized version that is certain to be better than [other model I'm running]."This is fast becoming one of my top old-man-yells-at-clouds pet peeves.Reported performance metrics are ONLY good for the exact model weights.Quantizing a model, or changing it in any way, requires new evaluation to know how well it performs.Quantizing a good model doesn't mean the quantized version is good.

    djoldman · HN #49408373

42. And here i am with a unsloth UD Q4_K_XL quant. Its a good model still.

    redbear2026 · HN #49406955

43. Any DGX Spark users in this thread? What's your favourite model to run on it?

    mrgaro · HN #49406087

44. Without doubt, dsv4-flash-0731. Original weights; needs two connected DGX.

    pama · HN #49408761

45. Rate limiting on free LLM APIs is usually where the pain lies. I've seen 5 concurrent reqs hit 20K/day limit in under 2 hours. Does anyone know a free API that still allows some reasonable concurrent requests?

    happybox2016 · HN #49406612

46. Asking for free inference is like asking for free gold in today's economy. :)

    serbuvlad · HN #49406663

47. Honestly expected yet another post dunking on local LLMs with some comparisons, got setup advice and benchmarking methodology instead.

    dowonseo · HN #49408994

48. RTN quantization of weights

    woadwarrior01 · HN #49406227

49. what a beautiful non-slop article!!! (i’m not ironic)

    giuscri · HN #49408041

50. most of the time when a local model feels dumb its not the quant, its the chat template. a lot of gguf mints just drop the template from the metadata and the runtime silently falls back to chatml. model still talks fine so nobody notices, it just gets noticeably dumber. got burned by this myself serving qwen, now i grep the gguf for the template tokens before i blame anything else. second place is sampling, people run whatever defaults their ui ships instead of what the vendor recommends and then compare that to benchmark numbers that were run greedy or with the official settings

    anotherCodder · HN #49403502

51. I've been comparing against TextGen and llama.cpp while I port to LocalAI and have been surprised by what's happening over the API, even with the defaults and jinja. It's been a fair reminder not to eschew familiarizing myself with the repos.

    washadjeffmad · HN #49404999

52. The punchline nobody wants to hear: your local model isn't dumber, it's just finally talking to you the way you actually sound.

    luciana1u · HN #49407801