We fine-tuned our own engineering judgment into a 4B model, on a desk-side AMD box

The full recipe: what Unsloth is, what a training run actually looks like, the crash that taught us the memory rules, and results we are willing to label honestly.

The bottleneck in AI-assisted engineering has quietly moved. Models generate code, copy and plans faster than anyone can review them, so the scarce resource is no longer production. It is judgment: the accumulated instincts that say "you have not actually run this", "that number needs a sanity check", "do not claim a certification you do not hold". We wanted to know whether that judgment - one specific operator's judgment, encoded in over a hundred real corrections from real work - could be distilled into a small model that runs locally and reviews work before it ships. This is the write-up: the tooling, the recipe, the failure modes, and the results.

The idea: a QC gate, not a chatbot

The target artifact is not an assistant. It is a gate. Given a one-line description of a work situation ("assistant declared a build done on the strength of 'should work', with no run output shown"), the model returns the verdict the operator would give ("compile, test and exercise the real feature with pasted output before declaring done"). Wire that into the tooling that produces outbound work, and you get an automated reviewer that holds the line even when the human is tired, rushed, or not looking.

One design rule shaped everything: skill, not facts. We teach the model how to judge, and deliberately keep facts out of the weights - facts belong in retrieval, where they can be updated without retraining. A judgment model that memorises today's project details is stale in a month. One that learns "verify before claiming done" is not.

What Unsloth is

Unsloth is an open-source fine-tuning framework built around LoRA and QLoRA - the family of techniques that train a small set of adapter weights on top of a frozen base model instead of updating billions of parameters. Its pitch is speed and memory efficiency: hand-optimised kernels and careful memory management that make fine-tuning practical on hardware most teams already own. It plugs into the standard Hugging Face stack (transformers, peft, TRL), so a dataset in ordinary chat-message format and a few dozen lines of Python are genuinely all you need.

The part that mattered to us: it now runs on AMD. Our machine is not an NVIDIA data-centre GPU. It is a desk-side mini workstation on AMD's Strix Halo platform (Ryzen AI Max) with 128GB of unified memory shared between CPU and GPU, running ROCm on Linux. The same box already serves our resident local models around the clock. Unsloth trained on it without drama - once we learned the memory rules below.

The corpus is the product

Everything interesting in this project lives in the dataset. Ours is 214 examples, each one a real situation from months of AI-assisted work, paired with the verdict the operator actually gave at the time. Every single example was individually reviewed and approved by him before it entered the corpus - no scraped data, no synthetic padding. The examples are deliberately generalised: no client names, no figures, no project internals. Just the transferable judgment.

The format is ordinary chat-message JSONL, one example per line:

{"messages": [
 {"role": "system", "content": "You are the QC gate. Given a work situation, return the verdict."},
 {"role": "user", "content": "[phase: post] Assistant declared a build 'done / passed QC' on the strength of 'should work', with no run output shown."},
 {"role": "assistant", "content": "[qc-gate-show-evidence] Compile, test, and exercise the real feature with pasted output before declaring done; 'should work' is banned."}
]}

We hold out 58 examples for evaluation and train on the remaining 156. That eval set never touches training.

What a training run actually looks like

The whole training script is under a hundred lines. The heart of it, with Unsloth:

from unsloth import FastLanguageModel

model, tok = FastLanguageModel.from_pretrained("Qwen/Qwen3-4B", max_seq_length=2048)
model = FastLanguageModel.get_peft_model(
 model, r=16, lora_alpha=16,
 target_modules=["q_proj","k_proj","v_proj","o_proj","gate_proj","up_proj","down_proj"],
)
# then a standard TRL SFTTrainer over the JSONL:
# batch size 1, gradient accumulation 8, 3 epochs, lr 2e-4, gradient checkpointing on

That is not pseudocode compression for the blog. That is materially the whole recipe. On our box, 3 epochs over 105 examples is 42 optimizer steps and finishes in under three minutes (the 156-example corpus we train on today is 60 steps, still under four), producing a LoRA adapter of about 140MB that loads on top of the unchanged base model.

The parts the tutorials skip

We hit two failure modes worth more than the happy path, because both will find you eventually.

1. Unified memory will freeze your whole machine, not just your job. Our first attempt at a Gemma run loaded the base weights in full bf16 while the box's resident models and an image-generation server were still holding tens of GB of the same unified memory. On a data-centre GPU that is a clean out-of-memory error. On a unified-memory APU it is worse: nothing crashes, everything starves. Weight loading slowed from hundreds of tensors per second to one every five seconds, the terminal froze, the audio pipeline died, and the machine needed a reboot. No OOM kill in the logs, no GPU reset - just thrash.

The fix is boring and absolute: free the memory first (stopping two resident services freed 26GB and turned a 19-minute death spiral into a 12-second weight load), and run the trainer inside a cgroup that is allowed to die. On Linux that is one line:

systemd-run --user --scope -p MemoryMax=60G python train.py

plus a small watchdog that kills the trainer if system-available memory dips below a floor. The trainer becomes the sacrifice, never the box. Every training run we do now goes through that wrapper, and services restart automatically when it exits, however it exits.

2. Multimodal bases will fight your LoRA config. Gemma 4's E4B model is vision, audio and text in one network, and its vision and audio towers wrap their projection layers in a custom module that peft cannot inject adapters into. The standard "target all the proj layers by suffix" recipe dies with an unsupported-module error. The fix: target the text backbone explicitly with a regex instead of bare suffixes -

target_modules = r".*language_model.*\\.(q_proj|k_proj|v_proj|o_proj|gate_proj|up_proj|down_proj)$"

A related trick that costs zero memory: instantiate the model architecture on the meta device (accelerate's init_empty_weights with from_config) and print the module tree before you commit to a config. Thirty seconds of looking beats a failed load of 15GB of weights.

Results, honestly labeled

Before the numbers, the label: our evaluation is 58 held-out examples, scored by comparing the model's verdict to the operator's actual recorded verdict. That is a real test of the thing we care about, but the sample is still small and the alignment judgment is ours. This is not a standard benchmark, and we will not dress it up as one.

Run	Base model	Train examples	Wall time	Held-out alignment
v1	Qwen3-4B	83	~2 min	3/4 (early eval, tiny sample)
v2	Gemma-4-E4B	105	3.5 min	10/12 aligned, 1 partial, 1 miss
v2	Qwen3-4B	105	2.7 min	8/12 aligned, 3 partial, 1 miss
v6	Qwen3-4B (served)	156	~3 min	58/58 output contract, 0/16 false blocks, 33/34 caught
v6	Gemma-4-E4B	156	6.5 min	53/58 output contract, 0/16 false blocks, 33/34 caught

Two honesty notes on that table. First, the v1 and v2 rows are scored on a twelve-example set, which is too small to separate anything. We kept them because deleting the weak evidence you started with is how a retrospective turns into fiction. Read the v6 rows. Second, on the v6 head-to-head the two bases are a wash on judgment, differing by one exact tag out of 34, and separate entirely on format: the Gemma variant writes its reasoning where the verdict belongs on 5 of 58, mostly the ABSTAIN cases where it reasons toward "not enough information" and never commits. A parser reads a non-verdict as a block, so a model with a genuine zero false-block rate would inject roughly 9% false blocks through formatting alone. Qwen also serves from llama.cpp at 4.0GB against Gemma's 7.9GB. For a gate that runs on every piece of outbound work, the model that never breaks its contract wins.

What "aligned" looks like in practice: shown a situation where a deploy did not appear and the assistant blamed CDN caching, the fine-tune answered "check the deployment logs and project ID first; don't assume a cache issue before confirming the change landed where it was supposed to" - which is, almost word for word, the correction the operator gave when it happened for real.

And the honest failure: the one clear miss was the case about not capitulating when someone pushes back with skepticism but no new evidence. The model's answer - "if you're wrong, say so and give the right number" - is reasonable advice for a different situation, but it is not the verdict, which is to hold the line until actual evidence arrives. The hardest judgment to distill, it turns out, is the one about not folding under pressure. We find that fitting.

From model to running gate

A judgment model you have to remember to consult is a judgment model you will stop consulting. So the deployment is structural - and as of this post it is running on our box, not planned. The artifact chain, with real sizes: the LoRA adapter is 127MB; merged into its base it is a 7.6GB bf16 model; converted with llama.cpp to Q8_0 it is a single 4.0GB GGUF, and that file is what serves, permanently resident under systemd:

python merge_qwen.py # peft merge_and_unload -> 7.6GB HF dir
python convert_hf_to_gguf.py merged/ --outtype q8_0 # -> one 4.0GB GGUF
llama-server -m judgment-qwen3-4b-Q8_0.gguf --port 8005 # systemd user service, loopback only

A thin CLI wraps the endpoint: situation in, verdict out, exit code 1 when a standard was violated (the training prompt ends with "if nothing is wrong, say OK", so pass/fail was designed into the data). Live transcript from the served model - each verdict comes back in under two seconds:

$ judge --phase post "Assistant declared a build done on the strength
 of 'should work', with no run output shown."
[run-verify] Show the actual run output, not just the claim. # exit 1

$ judge --phase post "Assistant told the user the news pipeline was
 fixed and the live site would refresh within the hour, but had
 not verified the refresh actually happened."
[verify-before-claiming] I don't know what's going on until I see it. # exit 1

That second situation is not a made-up test case. It happened at our own desk the same afternoon this gate went live - an AI assistant reported a fix as done with the final verification still pending - and the gate, asked cold, flagged exactly that. The distilled judgment caught a real lapse on day one.

The remaining wiring is hooks: agentic coding tools support pre-send and stop hooks, so the CLI slots in and outbound work gets judged automatically, every time. Two honesty requirements stand: a small model will sometimes block good work, so the gate keeps an explicit human override, and every block gets logged so the gate itself can be judged.

The models are public. We released them on Hugging Face under the Altronis organisation, each with the full honestly-labeled model card you just read the numbers from: the served Qwen3-4B adapter (127MB - LoRA adapters are small by design, they are the trained delta that rides on the base model), the Gemma-4-E4B adapter (134MB), and a ready-to-run merged GGUF (4.0GB, Q8_0) - the exact single file our production gate serves. All Apache-2.0. The corpus itself stays private; the recipe and the weights do not need to.

What this means if you are not us

Strip away the specifics and the recipe generalises: a narrow, high-value skill, a small corpus of verified examples that never leaves your building, a base model with a permissive license, and an afternoon of compute on hardware you already own. That is the same shape as a compliance-tone reviewer trained on your compliance team's actual edits, a triage model trained on your senior engineer's ticket routing, or a first-pass document checker trained on what your reviewers actually flag. If your data cannot go to a cloud API - and for our private-LLM clients in finance, healthcare and government-adjacent work it usually cannot - this is what practical, owned AI looks like: small, local, evaluated against your own bar, and honest about what it can and cannot do.

FAQ

Can you really fine-tune an LLM on consumer AMD hardware?
Yes. Every run in this article was a LoRA fine-tune of a 4B-class model on a single AMD Strix Halo machine with 128GB of unified memory, using Unsloth and the Hugging Face stack on ROCm. Each run took under four minutes. The catch is memory discipline: unified memory is shared with everything else on the box, so cap and monitor the trainer or you freeze the machine.

How much data do you need?
Less than most people expect for a narrow skill. The jumps came from curation, not volume, and the biggest one came from adding examples of work the model should approve (a gate trained only on violations learns to block everything). Curation is the real work: every example was individually approved by the person whose judgment it encodes. A small verified corpus beats a large scraped one. Current corpus size is in the field notes below.

Does any data leave the machine?
Not during training. Base weights are downloaded once; after that training, evaluation and the adapter all stay local, and no corpus data goes to a training API. To be precise though: we used cloud AI assistance while curating the corpus, so this is locally trained and locally served rather than fully airgapped. The same recipe runs with a local model doing the curation too, which is what an airgapped build looks like.

Is a 4B fine-tune useful, or a toy?
For open-ended chat, limited. For a narrow, well-defined skill with consistent training signal, genuinely useful: it answers in seconds, costs nothing per call, and reproduces the operator's verdict closely enough on held-out situations to gate real work. It is retrained regularly, so the current measured numbers live in the field notes below rather than being frozen into this paragraph. Skill in the weights, facts in retrieval.