The Pac-Man our 8-bit configuration produced, running in a browser. Three rendering faults were reported back to the model and fixed by it, not by us. Screenshot, 18 August 2026.

The fastest local model we tested was also the broken one

Someone posted that they had Qwen3.8-27B running at 36 tokens per second on the same AMD APU we use. We get 22.49. A 60% gain on hardware already under the desk is worth an evening, so we spent one reproducing it.

The number was real. The text was not, and it was fast because it was broken. If you are evaluating local inference on throughput, that sentence is the whole article.

We run Strix Halo as the inference backbone for our own systems, so this is not an academic question for us. The claim came from a community repackaging of Qwen3.8-27B: a custom 4-bit weight format plus MTP speculative decoding, built against a modified llama.cpp engine. Our own build is Unsloth's Qwen3.8-27B-UD-Q4_K_XL on upstream llama.cpp with Vulkan.

The work behind that claim is not trivial and it is worth describing, because the failure is easier to understand once you know what was changed. Three things, all of them reasonable ideas:

• A custom 4-bit weight format. Rather than a standard k-quant, weights are packed in blocks of 32 with a shared 16-bit scale, at 4.26 bits per weight, so the block size lines up with the GPU's vector register stride and dequantisation happens in the same pass as the matrix multiply. 13.55GB of weights instead of 15.92GB, and fewer arithmetic steps to decode them.
• MTP speculative decoding. Qwen3.8 ships multi-token-prediction heads, so the model drafts several tokens ahead of itself and verifies them in one memory sweep. That is where the headline speedup comes from: a 27B model is memory-bandwidth-bound, so producing 4 tokens per pass instead of 1 breaks the usual ceiling.
• An asymmetric KV cache. Attention keys are kept at 8-bit and values dropped to 4-bit, on the reasoning that value precision matters less, which frees bandwidth for streaming the weights.

Packaged with prebuilt weights on Hugging Face, a serving script with tuned flags, and a benchmark to prove the numbers. That last piece is where this goes wrong.

The number reproduced. The text did not.

Their benchmark script, unmodified, on our machine: 35.95 tok/s at 81.8% draft acceptance. Faster than ours on every one of the 4 prompts. Then we read what it had written.

prompt: "In one sentence, what does Altronis do?"

temp 0.0 -> _JOJOJOJOJOJOJOJOJOJOJOJOJOJOJOJOJOJO
temp 0.2 -> nix I'm not sure what you're asking. Allyx is a...
temp 0.7 -> iklklklklkllklllllllllllllllllllklkllklkl

leading tokens seen across runs: 壁 ritz uhh 说说 粉丝

That was not an edge case at an exotic setting. That was the documented default, --spec-draft-n-max 6 --spec-draft-p-min 0.60, at the speed the benchmark reported as best in the matrix.

Why a speed benchmark cannot see this

The benchmark script records completion tokens, latency and draft acceptance. It never reads the generated string. Nothing in the harness can fail a run for being gibberish.

That would be a minor gap if the failure were neutral. It is not, because the failure makes the number better. Speculative decoding drafts several tokens ahead and accepts them when a cheaper head agrees. Set the acceptance threshold too loose and wrong tokens get through, and a repeated wrong token decodes very fast. The corrupted run is quick precisely because it is corrupted.

Two changes fixed the text. Tightening acceptance to n-max 2 / p-min 0.90 gave clean output at 20.98 tok/s. Rebuilding the same engine from current source, 83 commits past the commit its build script pins, gave clean output at 33.63 tok/s with no other change. The pinned commit predates a fix that corrects the position passed to the draft head. A draft head handed the wrong position is exactly how you get plausible tokens that are wrong in context. We reported all of this upstream before writing any of it down.

The failure that survives the fix

With the newer engine, short answers were clean on every configuration we tried. So we gave it something long: the Pac-Man task described below, with a 16,000 token budget.

The first 4-bit variant ran the full 16,000 tokens emitting ,1,1,1,1,1, a maze array with no end, and never closed the file. The second variant also hit 16,000, ending in a MAP[y][x]=='*'||MAP[y][x]=='g'|| chain repeating forever after declaring isPelletOrPowerOrGhostOrWallOrPel..., with one maze row repeated 16 times. Our Q4_K_XL build wrote the same game in 6,643 tokens, closed the file, and the game runs.

Short prompts were fine on all of them. The failure only appears when the correct answer is long, which describes most real engineering work and all agentic coding.

What we measure now

1. Read a long generation, not a short one

Give the model a task whose correct answer runs to thousands of tokens, then check whether it terminated: did it hit the ceiling, did it close the artifact, did any line or phrase repeat past what the content needs. Two of our own rules were wrong on first contact, and both corrections are worth stealing. A row of 21 identical characters is an ASCII maze wall, not degeneration. ctx.beginPath(); repeated 13 times is ordinary canvas code. Repeated data is the tell. Repeated code is not.

2. Run the artifact

We load the generated page in headless Chromium, press arrow keys, and read the canvas and score back. This is the check that mattered. Three candidates generated at identical settings gave us a working game scoring 150, a weaker one scoring 120, and one that threw Assignment to constant variable on load and never animated. Every text-based check passed the broken one.

What actually raised quality on a 27B

Having built the instrument, we used it on a more useful question: how far can Qwen3.8-27B on one machine be pushed, and what does each step cost.

The test is the same one throughout this article, and it is worth stating plainly because a benchmark you cannot picture is a benchmark you cannot argue with. One prompt asks for a playable Pac-Man as a single self-contained HTML file, with 7 numbered requirements: a tile maze with walls the player cannot pass, pellets worth 10 points, 4 power pellets worth 50 that make ghosts edible for 8 seconds, 4 ghosts that chase, 3 lives, a score and lives display, and win and lose states with a restart key. A correct answer is roughly 6,000 to 12,000 tokens of HTML, CSS and JavaScript in one file, which is long enough to expose a model that cannot hold a structure together.

Scoring is not a rubric. We open the generated file in headless Chromium, wait for it to load, press the arrow keys, then read the canvas pixels and the score element back. A game that draws nothing, throws on load, or never increments the score fails, whatever the code looks like.

Every row below is that same task at 131,072 context, MTP on at n-max 2, one model resident at a time.

Configuration	Decode	Result when run
Q4_K_XL, thinking 512, chat sampling	22.67 tok/s	Runs, never scores
Q8_0, thinking 512, chat sampling	15.97 tok/s	Runs, scores 100
Q8_0, thinking 8192, code sampling	17.37 tok/s	Runs
Same settings, attempt 2 of 3	16.69 tok/s	Best artifact: scores 150, 4 ghosts, restart
Same settings, attempt 1 of 3	15.69 tok/s	Throws on load

8-bit weights cost about 30% of throughput and clearly improved the work. Raising the thinking budget from 512 to 8192 and switching from chat sampling (temp 0.7, presence penalty 1.5) to code sampling (temp 0.3, presence penalty 0) improved it again. The largest single gain came last, and it was not a model setting: generate 3 candidates and keep the one that runs.

If you are evaluating a local model

1. Make the benchmark read its own output. If the harness cannot fail a run that produced gibberish, its ranking is not about quality.
2. Test at the length you will actually use. Parity on a 200-token answer says nothing about a model that stops terminating at 6,000.
3. Check which commit you built. The corruption we hit was fixed upstream months ago and is still shipping, because a build script pins an old one.
4. Budget for repeated attempts. Where quality beats latency, 3 generations and an automatic check will beat 1 generation every time.

None of this makes the packaging work bad. The format is clever, the speed is real, and the author shipped reproducible artefacts and a benchmark anyone can run, which is more than most. The gap is that the benchmark measures the one thing that cannot tell you whether the model is working.

Measurements above are from one machine, one night, and one class of task: a Ryzen AI Max+ 395 with 128GB unified memory, Radeon 8060S (gfx1151), Vulkan, llama.cpp. Throughput figures come from the packaged build's own benchmark script, unmodified, with one model resident at a time.