What actually runs on 128GB of unified memory
Almost every local-LLM guide is written about a 24GB graphics card. This one is not. These are our own measurements on the machine we run production work on, including the results that went badly. If a number is here, we produced it.
The machine
- Machine
- AMD Ryzen AI Max 395 (Strix Halo, gfx1151)
- Unified memory
- 123 GiB usable
- GPU live cap
- 96 GiB (ttm.pages_limit)
- Memory bandwidth
- ~256 GB/s
- Runtimes
- llama.cpp (Vulkan), vLLM (ROCm)
Single-stream serving
| Model | Build | Active | tok/s | Notes |
|---|---|---|---|---|
| Qwen3.6-35B-A3B | UD-Q4_K_XL + native MTP | 3B active | 78.1 | Our production model. MTP on, exclusive GPU. |
| Qwen3.6-35B-A3B | UD-Q4_K_XL, no MTP | 3B active | ~48 | Same weights, speculative decoding off. |
| Gemma-4-26B-A4B | UD-Q8_K_XL | 4B active | 41.7 | Q8 rather than Q4, and it barely matters. See below. |
| Qwen3.6-35B dense distill | i1 quants | dense | 12 to 15 | Dense at this size is bench-worthy, not serving-worthy. |
Under load: 32 people at once
Qwen3.6-27B AWQ INT4 on vLLM, 19 GiB of weights. This is the table that decides whether a box can serve a team.
| Concurrent users | Aggregate tok/s | Per user | Slowest response |
|---|---|---|---|
| 1 | 5.66 | 5.66 | 22.6s |
| 8 | 38.6 | 4.84 | 26.5s |
| 16 | 71.3 | 4.46 | 28.7s |
| 32 | 111.2 | 3.48 | 36.8s |
What we learned, including the parts that cost us
For a sparse MoE, quantisation is not the speed lever
Qwen3.6-35B-A3B at Q4 ran 48 tok/s against 41.7 at Q8. Only about 15% for half the precision, because just 3B parameters are read per token and the idle experts' quantisation barely enters the arithmetic. Speculative decoding was the real lever: the same model went to 78.1 tok/s with MTP on. Teams routinely quantise harder and lose quality for a gain that was never going to be there.
One user is the wrong way to judge a box
A single stream on our vLLM setup managed 5.66 tok/s and felt slow enough to write the machine off. At 32 concurrent users the same hardware did 111.2 tok/s in aggregate, 19.6 times the throughput. A shared on-prem box should be judged on a full room, not on one person typing.
We roofline-checked our own numbers
19.05 GiB of weights over 256 GB/s of bandwidth puts a hard ceiling of 12.5 tok/s single-stream and about 400 tok/s at batch 32. We measured 45% and 28% of those ceilings. That consistent efficiency band is what tells us the numbers are real rather than too good to be true.
Distilled models lost to the stock model, every time
We ran a series of fine-tuned and distilled variants against the stock model on a fixed 14-point task. Stock Qwen3.6-35B scored 8/14 at 77.7 tok/s. The best distill managed 6/14 at 68.25. Slower and worse. Style transplants erode instruction-following, and the marketing around them rarely mentions it.
The ceiling is memory pressure, not model size
Swap on this class of machine is 8 GiB of zram, RAM-backed, and largely full. There is no real headroom, so a model that nearly fits does not run slowly, it takes the desktop down. We have done it twice. Leave room; the last few gigabytes are not usable capacity.
How big a model actually fits
| Tier | Model | Footprint |
|---|---|---|
| Safe, fast | gpt-oss-120B (~5B active) or GLM-4.5-Air 106B-A12B | ~65 GiB at 4-bit, 32k context |
| Comfortable dense | ~120B at Q4, or 70B at Q8 | ~70 to 75 GiB |
| At the edge, risky | Qwen3-235B-A22B at Q3 | ~100 GiB, against a 96 GiB GPU cap |
| Fits, but batch only | DeepSeek-V4-Flash IQ2_XXS | ~91 GiB, overnight work not chat |
Want these numbers for your own workload?
The figures above are ours, on our tasks. Yours will differ, and the only honest way to find out is to measure on the hardware you would actually buy. That is a thing we do.
Talk to us about an on-prem build →Measurements taken between June and August 2026 on the machine described above. Throughput varies with context length, concurrency and what else is resident on the GPU, so treat these as a floor for the same hardware rather than a promise.