Private, on-prem LLM deployment in Singapore — real benchmarks from our own box, honest hardware guidance, and PDPA / MAS / MOH-ready delivery. We measure; we don't quote vendor slides.
Live decode from our own Strix Halo box — a 35B model, fully offline.
In one paragraph
Private (on-prem) LLM deployment means running open-weight models — Llama, Qwen, DeepSeek, Gemma — entirely inside your own network, so sensitive data never leaves your perimeter and there's no per-token API bill. On a single 128 GB unified-memory box you can serve a 100B-class model at interactive speed. In Singapore this is the practical path to AI for PDPA-, MAS- and MOH-regulated data. Altronis sizes the hardware, deploys the stack, and builds the governance — inside your control, not ours.
Why on-prem
Three reasons teams move AI in-house
On-prem isn't always the answer — but when one of these is true, it's the only answer. We'll tell you honestly which side of the line you're on.
Data sovereignty
Regulated or sensitive data (PDPA, MAS, MOH, client IP) that legally or contractually can't leave your environment. On-prem removes cross-border transfer and third-party exposure at the root.
Cost at scale
Once you sustain millions of tokens a day, per-token API pricing outruns owning the hardware — often within 12–18 months. Predictable capex replaces an unbounded metered bill.
Latency & control
No network round-trip, no rate limits, no silent model swaps under you. Fixed latency and a model you pin, version and audit — essential for agents and real-time workloads.
Measured, not marketed
Real on-prem LLM benchmarks
Numbers from our own hardware — read from the server's own timings on real requests, not synthetic benchmarks or vendor slides. The Qwen3.6-MTP figure is measured live on this box.
Generation speed — tokens / second
higher is better
Qwen3.6-35B-A3B · native MTP live~90
Qwen3.6-35B-A3B ~44
Gemma 4 26B-A4B ~41
Qwen3.5-122B-A10B ~22
Model
Quant
Active params
Gen t/s
Prompt t/s
Context
Qwen3.6-35B-A3B · native MTPmeasured live · peak 102 t/s
Q4_K_XL
3B active / 35.5B
~90
~900
256K
Qwen3.6-35B-A3Bhigher-fidelity quant
Q8_K_XL
3B active / 35.5B
~44
~839
128K
Gemma 4 26B-A4Bextraction / structured output
Q8_K_XL
4B active
~41
~720
128K
Qwen3.5-122B-A10Blegacy 122B unit
Q4_K_XL
10B active
~22
~393
64K
The Qwen3.6-MTP row is measured live on this box (2026-07-10, llama.cpp fb30ba9, kernel 7.1-rc7, thinking off, short-to-mid context — it eases toward ~75–80 t/s at the full 256K). Other rows are documented runs on earlier builds. Hardware: AMD Ryzen AI Max+ 395 / Radeon 8060S (gfx1151), 128 GB unified LPDDR5X, Vulkan/RADV. Configs: github.com/sypherin/strix-halo-setup.
The three 128 GB boxes, head to head
Gemma-4 26B (Q8, 4K context), same 15-page document batch. The A100 row is a documented cloud reference.
AMD Strix Halo
Vulkan / RADV · 128 GB unified
~30
gen t/s
~130
prompt t/s
~22 min
15-pg batch
~US$1.5–2k
NVIDIA DGX Spark
CUDA · 128 GB unified
~50
gen t/s
~250
prompt t/s
~13 min
15-pg batch
~US$4–5.5k
A100 cloud
reference · 40–80 GB HBM
~85
gen t/s
~700
prompt t/s
~7 min
15-pg batch
recurring cloud spend
Pick the box
Which hardware for a private LLM?
For a single team it comes down to four options. The right one depends on budget, tooling preference, and how many people hit it at once.
Specs & pricing verified 10 July 2026
Best value
AMD Strix Halo (Ryzen AI Max+ 395)
The most memory-per-dollar on a desk: 16 Zen 5 cores, 40 RDNA 3.5 CUs, XDNA2 NPU, and 128 GB unified LPDDR5X (up to 96 GB usable as VRAM). Holds a 100B-class MoE via Vulkan/RADV; ~256 GB/s bandwidth is the ceiling. A ~US$1,500–2,000 mini-PC — best cost-per-token for a single-team private stack.
Best tooling
NVIDIA DGX Spark (GB10 Grace Blackwell)
GB10 Grace Blackwell Superchip, 128 GB unified, ~1 petaFLOP (FP4), up to ~200B params per unit. ~25% faster and steadier per-token than Strix with mature CUDA tooling (vLLM, TensorRT-LLM). ~US$4,000 (Founders) to ~$5,500 (OEM). The pick for NVIDIA's ecosystem or mixed training + inference.
Quiet / Mac
Apple Silicon (M3 Ultra Mac Studio)
Up to 256 GB unified memory (Apple discontinued the 512 GB tier in early 2026 amid the memory shortage), with 546–819 GB/s bandwidth — well above Strix — plus MLX / llama.cpp Metal. Silent, and a fit for Mac-first teams and quiet offices.
Max throughput
Data-centre GPU (NVIDIA H100 / H200 / B200)
80–192 GB HBM3e (H100 80 GB, H200 141 GB, B200 192 GB) at up to 8 TB/s — 3–5× the throughput for many concurrent users, but capex-heavy or recurring cloud cost. For high-QPS multi-tenant production, not a single team's box.
Under the hood
What actually makes local LLMs fast
Running a big model on a small box is a stack problem, not a hardware problem. These are the levers that decide whether it flies or crawls.
llama.cpp + GGUF
Fresh upstream llama.cpp with Vulkan or CUDA. GGUF quantization (Q4_K_XL for speed, Q8 for fidelity) shrinks a 35B model to ~23 GB. Native multi-token prediction roughly doubles decode speed vs a plain run.
MoE beats dense
Mixture-of-Experts models (Qwen3.6-35B-A3B, Gemma-4-26B-A4B) activate only 3–4B parameters per token, so a 35B model decodes at 3B-model speed while keeping large-model quality — ideal for a memory-rich, bandwidth-bound box.
Backend matters
On AMD gfx1151 we measured Vulkan/RADV ~5% faster generation and ~47% faster prompt-processing than ROCm for LLM inference, and far more stable. FP8 is broken on RDNA 3.5 — use BF16. Backend choice is not cosmetic.
Context & KV cache
256K-token native context on a single box, with a q8_0 KV cache to keep memory tiny even at long context. Unified memory (a ~124 GiB GTT window) means weights + KV live in host RAM without a hard GPU/CPU split.
How we deploy
One private endpoint, many models
We don't hand you a research script. We deploy a governed, single-endpoint stack your apps point at — models behind an authenticated gateway, everything inside your perimeter.
The stack we stand up
A path-routing gateway: one authenticated URL → chat, vision/OCR, embeddings — model ports never exposed directly.
GGUF-quantized open models sized to your hardware and task, pinned and versioned.
Governance: access control, full prompt/response audit logging, human-in-the-loop checkpoints.
Optional private tunnel for remote access; runs on your kit or in your VPC.
What you get on handover
A documented, self-serve runbook — your team can operate and update it, not just us.
A proof of concept in 1–2 weeks; daily production use in 6–8 weeks.
Model + hardware advisory so you don't over-buy or over-size.
Optional managed-ops retainer if you'd rather we keep it running.
Try Deneb — our on-prem setup assistant
Ask it, in plain English, how to stand up a private LLM on your own hardware — models, quantization, the gateway, the exact commands. Fittingly, Deneb runs on a local model itself.
On-prem is the architecture regulated Singapore teams use to keep data they can't send to a hosted API inside their own walls. To be clear — we're not a compliance certifier: on-prem, plus the governance and audit we build, supports your obligations, but the regulatory relationship stays yours.
Data residency (PDPA)
Because data never leaves your perimeter, cross-border transfer and third-party-processor exposure largely fall away — the hardest part of your PDPA obligations, handled by the architecture rather than a policy promise.
Finance (MAS)
For financial institutions, on-prem inference keeps prompts, outputs and customer data inside the environment you already control and audit — the posture you need before putting AI near client data. Your MAS relationship stays yours.
Healthcare (MOH / HSA)
Patient data stays in-tenant; no identifiable data enters a hosted model. Where AI touches triage or diagnosis we flag that HSA may treat it as a medical device (AI-SaMD) — so you assess it early, not after a build.
Straight answers
Private LLM deployment: FAQ
Can you run a private LLM fully on-premise in Singapore?
Yes. Open-weight models (Llama, Qwen, DeepSeek, Gemma, GPT-OSS) run entirely inside your own network on a single 128 GB unified-memory box such as an AMD Strix Halo or NVIDIA DGX Spark — no data leaves your perimeter, no per-token API bill.
Is an on-prem LLM PDPA compliant?
On-prem removes the hardest PDPA risks — cross-border transfer and third-party processing — because data never leaves your environment. It is not automatically compliant on its own; you still need access control, audit trails and governance, which we build in. No serious vendor should claim 'fully PDPA-compliant' out of the box.
How fast is local LLM inference, really?
Measured live on our own AMD Strix Halo box, a 35B Qwen3.6 MoE model decodes at around 90 tokens/second (peak 102) with native multi-token prediction, and processes prompts at roughly 900 tokens/second. A NVIDIA DGX Spark is about 25% faster again. That is comfortably interactive for chat, extraction and agent workloads.
What hardware do I need to run a local LLM?
For a single team, a 128 GB unified-memory box: AMD Strix Halo (Ryzen AI Max+ 395) for best value, or NVIDIA DGX Spark for the CUDA ecosystem. Both hold a 100B-class MoE model. For many concurrent users you move to data-centre GPUs (A100/H100), which trade cost for throughput.
AMD or NVIDIA for on-prem LLM?
AMD Strix Halo gives the most memory per dollar and runs the largest models on a desk via Vulkan/RADV; NVIDIA DGX Spark is ~25% faster, steadier per-token, and has the mature CUDA/vLLM/TensorRT tooling. Pick AMD for cost-per-token, NVIDIA for tooling and mixed workloads.
When does on-prem beat cloud AI?
When data can't leave your perimeter (regulated data), when you sustain enough volume that per-token API costs outrun hardware (typically millions of tokens/day), or when you need predictable low latency. Below that, cloud APIs are usually cheaper and faster to start — we'll tell you honestly which side you're on.
Can I run a 70B+ model on a single box?
Yes. A 128 GB unified-memory box holds a 100B-class model in 4-bit — we run a 122B MoE unit locally. Mixture-of-Experts models are the sweet spot: large quality, only 3–10B parameters active per token, so they stay fast on memory-bandwidth-bound hardware.
How long does a private LLM deployment take?
A working proof of concept in 1–2 weeks and daily production use in 6–8 weeks is typical — hardware sizing, model selection and quantization, the inference gateway, governance and audit, then integration with your systems. We hand over a documented, self-serve stack, not a black box.
Which open models can we self-host?
Llama, Qwen (incl. Qwen3.6 MoE), DeepSeek, Google Gemma, GPT-OSS and Mistral families — plus OCR/vision models (Qwen-VL, Surya) for document workloads. We match the model to your task, hardware and quality bar rather than defaulting to the biggest one.
Do you deploy on our hardware or supply it?
Either. We advise on and size the hardware (Strix Halo, DGX Spark, Apple Silicon, data-centre GPUs), or deploy onto kit you already own or run in your own VPC. The whole stack — models, gateway, governance — runs inside your control; day-to-day ops stay with your team unless you want a managed retainer.
Go deeper
Field notes from running this for real
We build in the open. These are the real write-ups behind the benchmarks above — the trade-offs, the failures, and the fixes.
Best value
Best tooling
Quiet / Mac
Max throughput