How much data do you need to fine-tune a judgment model?

Less than most people expect for a narrow skill. We saw a measurable jump going from 83 to 105 curated training examples. The catch is that curation is the real work: every example in our corpus was individually reviewed and approved by the person whose judgment it encodes. A small corpus of verified examples beats a large scraped one for this kind of task.

Does any data leave the machine during fine-tuning?

No. The base weights are downloaded once from Hugging Face, and after that the corpus, the training, the evaluation and the resulting adapter all stay on the local machine. That is the point: this is the workflow we recommend to organisations whose data cannot leave their walls.

By Z. Aw | Published 24 Jul 2026

Fine-tuning a local LLM with Unsloth on an AMD Strix Halo workstation

We fine-tuned our own engineering judgment into a 4B model, on a desk-side AMD box

The full recipe: what Unsloth is, what a training run actually looks like, the crash that taught us the memory rules, and results we are willing to label honestly.

The bottleneck in AI-assisted engineering has quietly moved. Models generate code, copy and plans faster than anyone can review them, so the scarce resource is no longer production. It is judgment: the accumulated instincts that say "you have not actually run this", "that number needs a sanity check", "do not claim a certification you do not hold". We wanted to know whether that judgment - one specific operator's judgment, encoded in over a hundred real corrections from real work - could be distilled into a small model that runs locally and reviews work before it ships. This is the write-up: the tooling, the recipe, the failure modes, and the results.

The idea: a QC gate, not a chatbot

The target artifact is not an assistant. It is a gate. Given a one-line description of a work situation ("assistant declared a build done on the strength of 'should work', with no run output shown"), the model returns the verdict the operator would give ("compile, test and exercise the real feature with pasted output before declaring done"). Wire that into the tooling that produces outbound work, and you get an automated reviewer that holds the line even when the human is tired, rushed, or not looking.

One design rule shaped everything: skill, not facts. We teach the model how to judge, and deliberately keep facts out of the weights - facts belong in retrieval, where they can be updated without retraining. A judgment model that memorises today's project details is stale in a month. One that learns "verify before claiming done" is not.

What Unsloth is

Unsloth is an open-source fine-tuning framework built around LoRA and QLoRA - the family of techniques that train a small set of adapter weights on top of a frozen base model instead of updating billions of parameters. Its pitch is speed and memory efficiency: hand-optimised kernels and careful memory management that make fine-tuning practical on hardware most teams already own. It plugs into the standard Hugging Face stack (transformers, peft, TRL), so a dataset in ordinary chat-message format and a few dozen lines of Python are genuinely all you need.

The part that mattered to us: it now runs on AMD. Our machine is not an NVIDIA data-centre GPU. It is a desk-side mini workstation on AMD's Strix Halo platform (Ryzen AI Max) with 128GB of unified memory shared between CPU and GPU, running ROCm on Linux. The same box already serves our resident local models around the clock. Unsloth trained on it without drama - once we learned the memory rules below.

The corpus is the product

Everything interesting in this project lives in the dataset. Ours is 117 examples, each one a real situation from months of AI-assisted work, paired with the verdict the operator actually gave at the time. Every single example was individually reviewed and approved by him before it entered the corpus - no scraped data, no synthetic padding. The examples are deliberately generalised: no client names, no figures, no project internals. Just the transferable judgment.

The format is ordinary chat-message JSONL, one example per line:

{"messages": [
  {"role": "system", "content": "You are the QC gate. Given a work situation, return the verdict."},
  {"role": "user", "content": "[phase: post] Assistant declared a build 'done / passed QC' on the strength of 'should work', with no run output shown."},
  {"role": "assistant", "content": "[qc-gate-show-evidence] Compile, test, and exercise the real feature with pasted output before declaring done; 'should work' is banned."}
]}

We hold out 12 examples for evaluation and train on the remaining 105. That eval set never touches training.

What a training run actually looks like

The whole training script is under a hundred lines. The heart of it, with Unsloth:

from unsloth import FastLanguageModel

model, tok = FastLanguageModel.from_pretrained("Qwen/Qwen3-4B", max_seq_length=2048)
model = FastLanguageModel.get_peft_model(
    model, r=16, lora_alpha=16,
    target_modules=["q_proj","k_proj","v_proj","o_proj","gate_proj","up_proj","down_proj"],
)
# then a standard TRL SFTTrainer over the JSONL:
# batch size 1, gradient accumulation 8, 3 epochs, lr 2e-4, gradient checkpointing on

That is not pseudocode compression for the blog. That is materially the whole recipe. On our box, 3 epochs over 105 examples is 42 optimizer steps and finishes in under three minutes, producing a LoRA adapter of about 140MB that loads on top of the unchanged base model.

The parts the tutorials skip

We hit two failure modes worth more than the happy path, because both will find you eventually.

1. Unified memory will freeze your whole machine, not just your job. Our first attempt at a Gemma run loaded the base weights in full bf16 while the box's resident models and an image-generation server were still holding tens of GB of the same unified memory. On a data-centre GPU that is a clean out-of-memory error. On a unified-memory APU it is worse: nothing crashes, everything starves. Weight loading slowed from hundreds of tensors per second to one every five seconds, the terminal froze, the audio pipeline died, and the machine needed a reboot. No OOM kill in the logs, no GPU reset - just thrash.

The fix is boring and absolute: free the memory first (stopping two resident services freed 26GB and turned a 19-minute death spiral into a 12-second weight load), and run the trainer inside a cgroup that is allowed to die. On Linux that is one line:

systemd-run --user --scope -p MemoryMax=60G python train.py

plus a small watchdog that kills the trainer if system-available memory dips below a floor. The trainer becomes the sacrifice, never the box. Every training run we do now goes through that wrapper, and services restart automatically when it exits, however it exits.

2. Multimodal bases will fight your LoRA config. Gemma 4's E4B model is vision, audio and text in one network, and its vision and audio towers wrap their projection layers in a custom module that peft cannot inject adapters into. The standard "target all the proj layers by suffix" recipe dies with an unsupported-module error. The fix: target the text backbone explicitly with a regex instead of bare suffixes -

target_modules = r".*language_model.*\.(q_proj|k_proj|v_proj|o_proj|gate_proj|up_proj|down_proj)$"

A related trick that costs zero memory: instantiate the model architecture on the meta device (accelerate's init_empty_weights with from_config) and print the module tree before you commit to a config. Thirty seconds of looking beats a failed load of 15GB of weights.

Results, honestly labeled

Before the numbers, the label: our evaluation is 12 held-out examples, scored by comparing the model's verdict to the operator's actual recorded verdict. That is a real test of the thing we care about, but it is a small sample and the alignment judgment is ours. This is not a standard benchmark, and we will not dress it up as one.

Run	Base model	Train examples	Wall time	Held-out alignment
v1	Qwen3-4B	83	~2 min	3/4 (early eval, tiny sample)
v2	Gemma-4-E4B	105	3.5 min	10/12 aligned, 1 partial, 1 miss
v2	Qwen3-4B	105	2.7 min	8/12 aligned, 3 partial, 1 miss

Two more honesty notes on that table. First, a two-example gap on a twelve-example set is inside the noise; we treat the two v2 runs as roughly tied on quality, not as a Gemma win. Second, quality is not the only axis that matters for an always-on gate: the Qwen adapter converts to GGUF and serves from llama.cpp in about 2.5GB, while the Gemma base is a multimodal architecture that today means a much heavier serving path. For a gate that runs on every piece of outbound work, the smaller, natively-servable model is the practical choice.

What "aligned" looks like in practice: shown a situation where a deploy did not appear and the assistant blamed CDN caching, the fine-tune answered "check the deployment logs and project ID first; don't assume a cache issue before confirming the change landed where it was supposed to" - which is, almost word for word, the correction the operator gave when it happened for real.

And the honest failure: the one clear miss was the case about not capitulating when someone pushes back with skepticism but no new evidence. The model's answer - "if you're wrong, say so and give the right number" - is reasonable advice for a different situation, but it is not the verdict, which is to hold the line until actual evidence arrives. The hardest judgment to distill, it turns out, is the one about not folding under pressure. We find that fitting.

From model to running gate

A judgment model you have to remember to consult is a judgment model you will stop consulting. So the deployment is structural - and as of this post it is running on our box, not planned. The artifact chain, with real sizes: the LoRA adapter is 127MB; merged into its base it is a 7.6GB bf16 model; converted with llama.cpp to Q8_0 it is a single 4.0GB GGUF, and that file is what serves, permanently resident under systemd:

python merge_qwen.py                                    # peft merge_and_unload -> 7.6GB HF dir
python convert_hf_to_gguf.py merged/ --outtype q8_0     # -> one 4.0GB GGUF
llama-server -m judgment-qwen3-4b-Q8_0.gguf --port 8005 # systemd user service, loopback only

A thin CLI wraps the endpoint: situation in, verdict out, exit code 1 when a standard was violated (the training prompt ends with "if nothing is wrong, say OK", so pass/fail was designed into the data). Live transcript from the served model - each verdict comes back in under two seconds:

$ judge --phase post "Assistant declared a build done on the strength
    of 'should work', with no run output shown."
[run-verify] Show the actual run output, not just the claim.        # exit 1

$ judge --phase post "Assistant told the user the news pipeline was
    fixed and the live site would refresh within the hour, but had
    not verified the refresh actually happened."
[verify-before-claiming] I don't know what's going on until I see it.  # exit 1

That second situation is not a made-up test case. It happened at our own desk the same afternoon this gate went live - an AI assistant reported a fix as done with the final verification still pending - and the gate, asked cold, flagged exactly that. The distilled judgment caught a real lapse on day one.

The remaining wiring is hooks: agentic coding tools support pre-send and stop hooks, so the CLI slots in and outbound work gets judged automatically, every time. Two honesty requirements stand: a small model will sometimes block good work, so the gate keeps an explicit human override, and every block gets logged so the gate itself can be judged.

The models are public. We released them on Hugging Face under the Altronis organisation, each with the full honestly-labeled model card you just read the numbers from: the served Qwen3-4B adapter (127MB - LoRA adapters are small by design, they are the trained delta that rides on the base model), the Gemma-4-E4B adapter (134MB), and a ready-to-run merged GGUF (4.0GB, Q8_0) - the exact single file our production gate serves. All Apache-2.0. The corpus itself stays private; the recipe and the weights do not need to.

This is a living write-up. The gate now judges real outbound work every day and logs every verdict, so the numbers above will grow from a 12-example eval into field results: true catches, false blocks, override rates. We will update this post as they accumulate - stay tuned.

Field notes, day 1 (updated 24 Jul 2026, same evening)

We promised honest field results, so here is day one, unvarnished. Across roughly 25 live judgments of real outbound work: one genuinely useful catch (the gate forced an unverified claim to be downgraded from stated fact to "likely"), one false block (a style opinion that contradicted this site's own conventions), and a majority of degenerate verdicts, echoes of the judged text and recursive word-salad, traceable to the corpus's open-ended verdict format. The day's one real mistake happened in tool-space, where a prose-judging gate structurally cannot see it.

Three fixes enter training tomorrow, each bought with a day-one bruise: a closed verdict schema (fixed tag vocabulary, the same discipline that keeps production safety classifiers coherent), deliberately-flawed training examples, and machine-checkable evidence obligations, because a worker's own prose, the agent's or the judge's, is not evidence. A judgment model whose own evaluation isn't judged honestly would be a contradiction in terms. Day two's numbers will replace these when they exist.

Same evening: the schema fix shipped, and the word salad is dead

The first fix could not wait for tomorrow, and it paid off the same night. We traced the degenerate-output class to a concrete data bug: our 117 examples carried 85 distinct free-form verdict tags, which trained the model that inventing a new tag is part of the job. v3 migrated every example onto a closed vocabulary of 14 canonical tags with a strict one-line output format, retrained in under three minutes, and swapped into the live gate.

First results, honestly labeled as a same-author demo battery rather than an independent eval: replaying the day's six worst failure inputs produced zero degenerate outputs, every verdict in valid format with a legal tag, and, the part v2 was structurally incapable of, a genuinely clean situation now passes with "VERDICT: OK". The known residual: ambiguous inputs still draw over-eager verdicts, because the corpus contains no explicit pass examples yet. That is the next batch's job. The updated model and GGUF are live on the Hugging Face repos, and the gate judging our outbound work as we type this is already v3, in proper format, including when it is wrong.

What this means if you are not us

Strip away the specifics and the recipe generalises: a narrow, high-value skill, a small corpus of verified examples that never leaves your building, a base model with a permissive license, and an afternoon of compute on hardware you already own. That is the same shape as a compliance-tone reviewer trained on your compliance team's actual edits, a triage model trained on your senior engineer's ticket routing, or a first-pass document checker trained on what your reviewers actually flag. If your data cannot go to a cloud API - and for our private-LLM clients in finance, healthcare and government-adjacent work it usually cannot - this is what practical, owned AI looks like: small, local, evaluated against your own bar, and honest about what it can and cannot do.

FAQ

Can you really fine-tune an LLM on consumer AMD hardware?
Yes. Every run in this article was a LoRA fine-tune of a 4B-class model on a single AMD Strix Halo machine with 128GB of unified memory, using Unsloth and the Hugging Face stack on ROCm. Each run took under four minutes. The catch is memory discipline: unified memory is shared with everything else on the box, so cap and monitor the trainer or you freeze the machine.

How much data do you need?
Less than most people expect for a narrow skill - we saw a measurable jump going from 83 to 105 curated examples. Curation is the real work: every example was individually approved by the person whose judgment it encodes. A small verified corpus beats a large scraped one.

Does any data leave the machine?
No. Base weights are downloaded once; after that the corpus, training, evaluation and the adapter all stay local. That is the point.

Is a 4B fine-tune useful, or a toy?
For open-ended chat, limited. For a narrow, well-defined skill with consistent training signal, genuinely useful: ours reproduces the operator's verdict on roughly ten of twelve held-out situations, in seconds, at zero cost per call. Skill in the weights, facts in retrieval.