Self-hosting
Running a model on your own hardware is the only way to be certain your data never leaves it. It is also more work than most people expect, and the honest answer for many is that they do not need it. This page covers both halves.
Why self-host
Your data never leaves
This is the only reason on the list that is categorically true rather than a matter of degree. Every other argument for self-hosting has a cloud workaround; this one does not.
No model gets retired under you
Underrated, and increasingly the strongest practical argument. OpenAI commits to at least six months notice for generally available models and as little as two weeks for previews. A file on your own disk is never deprecated. If you have validated a prompt suite against a specific model, that matters.
No per-token cost at volume
Real, but do the arithmetic honestly. A graphics card pays for itself only against a sustained workload — small-model API pricing is low enough that intermittent use never breaks even, and the electricity is not free either.
Offline, and no lock-in
Both genuine, both usually secondary to the two above. Worth naming if you work somewhere without reliable connectivity, or air-gapped by policy.
When not to
The quality gap is not where you think
The open models that genuinely rival the frontier are enormous — hundreds of billions to trillions of parameters. You are not running those on a workstation. The gap that matters to you is between the frontier and what fits in your memory, and that gap is still wide.
Mixture-of-experts does not save you memory
A common and expensive misunderstanding. Google states it plainly for its own 26B model with 4B active: it still requires loading all 26 billion parameters, so memory demand is comparable to a dense model of that size. Sparse models are fast, not small.
You become the operations team
Runtimes ship updates constantly. You are now the one who upgrades, and the one who discovers that a chat-template change quietly degraded your output last Tuesday.
Most people overestimate their privacy need
A useful test: if the data would be fine in Microsoft 365 or Google Workspace, it is probably fine on an EU-region API endpoint under a processor agreement. The genuine cases are narrower — legally privileged material, health data, source code under NDA, data you are contractually barred from disclosing to sub-processors.
The right call when
Confidential or special-category data. High sustained volume. Offline or air-gapped work. A model you must freeze for reproducibility. Or you already own the hardware.
The wrong call when
You want frontier quality. The workload is intermittent. Nobody wants to own the operations. Or "privacy" is a feeling rather than a documented requirement.
The middle path
There is a middle path people forget: a managed endpoint in an EU region under a processor agreement gets you most of the compliance story with none of the operational burden.
Runtimes
Check that local really is local
Check that "local" really is local. Ollama now offers a cloud mode where a command that looks exactly like a local one transparently runs the model on someone else’s servers. If your reason for self-hosting is that data must not leave your infrastructure, verify which mode you are in before you send anything confidential.
| Runtime | Licence | API | Suits | Main limitation |
|---|---|---|---|---|
| Ollama | MIT | localhost:11434/v1 | The fastest route from nothing to a working model. | It hides everything behind defaults — context size, template, sampling — which is exactly what makes it a poor choice for prompt-quality work until you have overridden them. |
| llama.cpp | MIT | configurable | Control, and unusual hardware — it runs on almost anything, including CPU-only machines. | You are exposed to every flag, and its defaults are not the model vendor’s defaults. |
| vLLM | Apache 2.0 | localhost:8000 | Serving several people at once. Its advantage is concurrency, not single-user speed. | It wants a real GPU and real operations. At one user the advantage over Ollama is negligible and the cost is not. |
| LM Studio | free for work use; app licence not stated | localhost:1234/v1 | People who would rather click than type. A graphical app with a command-line tool when you outgrow it. | The desktop application’s own licence is not stated in its documentation, so do not assume it is open source. |
| Jan | Apache 2.0 | localhost:1337 | The licence-clean alternative to LM Studio, built on llama.cpp and fully offline. | Smaller community than Ollama or LM Studio, so fewer answers when something breaks. |
Hardware
The rule of thumb
Weights need roughly 2 GB per billion parameters at 16-bit, 1 GB at 8-bit, and 0.5 GB at 4-bit. Then add 20–50% for the key-value cache, more if you want long context. So: halve your VRAM in gigabytes and that is roughly the largest model in billions of parameters you can run at 4-bit.
| Precision | VRAM per B params | An 8B model | A 27B model |
|---|---|---|---|
| 16-bit (as trained) | 2 × B | 16 GB | 54 GB |
| 8-bit | 1 × B | 8 GB | 27 GB |
| 4-bit | 0.5 × B | 4 GB | 14 GB |
A modern laptop
16–32 GB of memory runs 2B–8B models at 4-bit. Genuinely useful for summarising, extraction and drafting. Apple Silicon punches above its weight here because the memory is shared with the graphics processor.
A single 24 GB graphics card
The sweet spot. Models up to roughly 30B at 4-bit fit with room for context. This is where most serious self-hosting actually happens.
A workstation
32 GB on one card, or two 24 GB cards. Larger models at better quantisation. Note that a high-end card can draw well over 500 watts and wants a 1000 watt power supply — noise, heat and electricity all become real considerations.
A small server
One or more 80 GB accelerators. This is where vLLM starts to earn its complexity, because you are now serving concurrent users rather than yourself.
What quantisation actually is
A model’s weights are billions of numbers, stored during training at 16 bits each. Quantisation rewrites them with fewer bits — 8, 5 or 4 — so the file shrinks and more of it fits in fast memory. It is lossy, like a JPEG. Down to about 4 bits the loss is usually invisible for ordinary work; below that it degrades noticeably, showing up first in long-form coherence and structured output.
- Prefer a quantisation-aware build where the vendor ships one. Google reports roughly halving the quality drop at 4-bit compared with quantising after the fact.
- Aggressive quantisation breaks tool calling before it breaks prose. If your model has started producing malformed function calls, suspect the quantisation before the prompt.
Prompting local models
On a frontier API the platform absorbs your mistakes. Locally, nothing does — and the failures are silent. Nothing errors; the output just gets quietly worse. These are the five that catch people.
The chat template
A chat model is still a next-token predictor: the roles you send get flattened into one string using control tokens specific to that model. Two models fine-tuned from the same base can use entirely different markup, and Hugging Face is blunt that with the wrong control tokens performance is drastically worse. Three ways it goes wrong: a missing generation prompt, so the model continues your message instead of answering it; doubled special tokens from hand-assembling a string that already contained them; and a silent fallback to a generic template when the runtime does not recognise the model.
What to do
Use the OpenAI-compatible chat endpoint and pass a message array. Never hand-build the raw string. If you must, check the runtime’s startup log for which template it actually chose.
System messages are not universal
Gemma’s instruction-tuned models support only user and model roles — there is no system role at all, and Google’s guidance is to put system-level instruction into the first user turn. At the other extreme, OpenAI’s open models use a format that splits what you would call a system prompt across two roles with a strict precedence order, and OpenAI states they will not work correctly without it. A system prompt that works on one open model may be silently ignored on another.
What to do
Read the model card for its role convention before you port a prompt. Test that your system instruction actually took effect rather than assuming it did.
Chain of thought, revisited
This is where local models genuinely differ from the advice elsewhere on this site. OpenAI says that prompting its reasoning models to think step by step is unnecessary and can hinder them. But the original 2022 research found chain-of-thought only helps at around 100B parameters and above, and that smaller models produce fluent but illogical chains — which is precisely the failure you see on a laptop model. The 2026 answer is three-way, not two-way.
What to do
Frontier reasoning model: do not ask for step-by-step, it is redundant. Small open model with a built-in thinking mode: use the mode, do not bolt chain-of-thought on top of it. Small open model without one: explicit step-by-step still helps — but verify on your own task, because a fluent-but-wrong chain is the characteristic failure.
The context window you actually get
A model advertising 262,144 tokens may be running at 4,096 on your machine, because the runtime sizes context by available memory rather than by what the model supports. The front of your conversation is then silently truncated — which is exactly the experience that makes people conclude local models are stupid. Two further traps: an extended context window is a trade rather than a free upgrade, and vendors note that performance on shorter texts can degrade when extension is switched on. And attention is uneven across a long input, so the middle gets less of it than the edges.
What to do
Check what you are actually running at and raise it deliberately. Then put your instruction and your most important document at the edges of a long prompt, not buried in the middle.
Sampling defaults are not the vendor’s
On a hosted API you rarely touch sampling. Locally the runtime picks defaults — commonly a temperature of 0.8 — that are not what the model’s makers recommend, and the mismatch degrades output quietly. Recommended settings differ not just between vendors but between thinking and non-thinking modes of the same model, and they have changed between generations of the same family. There is no universal good default. One counter-intuitive warning worth heeding: Qwen states in capitals that you should not use greedy decoding, because it causes degradation and endless repetition. Many people reflexively set temperature to zero for determinism; on a reasoning model that is actively harmful.
What to do
Open the model card and copy its recommended temperature, top-p and top-k. The exception is structured extraction against a schema, where a temperature of zero is genuinely advised.
Getting clean structured output
Local models chatter. The fix is not to nag them in the prompt but to constrain decoding, which is better supported locally than on most hosted APIs: llama.cpp can force output to match a grammar or a JSON schema, and Ollama accepts a schema directly. There is one caveat that almost everyone misses, and both projects state it: the schema constrains the output format only — the model never sees the schema, so you must still describe the expected structure in the prompt.
What to do
Schema and prompt, not schema instead of prompt. Set temperature to zero for this one case.
GDPR and the AI Act
Self-hosting changes your legal position in ways that are easy to overstate in both directions. This is orientation, not legal advice — see the disclaimer on the legal page, and take proper advice for anything consequential.
What self-hosting genuinely helps with
- The processor relationship disappears. Using someone else’s API makes them a processor, requiring a contract and a chain of sub-processor authorisations behind it. Running the model yourself removes that chain entirely.
- International transfer questions become moot if the hardware sits in the EU.
- Security measures become easier to evidence, because you control encryption, pseudonymisation and testing directly rather than inheriting someone’s attestations.
What it does not solve
- You are still the controller. Lawful basis, purpose limitation, data minimisation, transparency and data-subject rights are all unchanged. Running the hardware changes who processes, not whether the processing is lawful.
- A data protection impact assessment may still be required, and must happen before the processing starts, not after.
- The model itself may carry personal data. The European Data Protection Board treats model anonymity as something to assess case by case, not assume. Downloading an open-weights model does not import a clean compliance record with it.
- Your infrastructure is now your security problem. An unauthenticated model server exposed to the internet is worse than a reputable API.
Where it changes your position under the AI Act
Under the AI Act the distinction that matters is between a deployer, who uses a system under its own authority, and a provider, who develops one and puts it into service — including, in the Act’s own words, for own use. That phrase is why building an internal system around a self-hosted model sits closer to the provider role than calling someone else’s API does, where you are typically plainly a deployer. Treat this as a genuine risk of role change rather than a certainty; much depends on whether you are developing a system at all or merely operating one, and that is a legal judgement. Two obligations already apply regardless: the AI literacy duty on providers and deployers alike, and the transparency rules in force since 2 August 2026. Note also that the Act was amended in July 2026, which moved several deadlines — check current dates rather than an older summary.
In the Netherlands
In the Netherlands the Autoriteit Persoonsgegevens coordinates algorithm supervision, alongside the Rijksinspectie Digitale Infrastructuur in a central AI supervision role, with a regulatory sandbox being established where organisations can test systems with supervisor guidance.
Getting started
Be honest about why
Write one sentence naming the data that must not leave. If you cannot, stop here and use an EU-region API under a processor agreement — you will save yourself the hardware and the maintenance.
Check your ceiling before downloading anything
Find your VRAM, or unified memory on a Mac. Halve it: that is roughly the largest model in billions of parameters you can run at 4-bit, before context. 8 GB gets you around 7–8B; 24 GB gets you around 27–31B quantised.
Install exactly one thing
Ollama if you live in a terminal. LM Studio if you would rather click. Do not evaluate five runtimes before you have run one model.
Fix the two defaults that quietly ruin output
This is the step every other guide omits. Raise the context length deliberately rather than accepting what the runtime guessed. Then open the model card and copy its recommended sampling settings over the runtime’s.
Point a real client at it
Every runtime here speaks an OpenAI-compatible interface, so changing one base URL in an existing script is usually the whole integration. This is also where you discover which features are missing.
Re-test your prompts, assuming nothing carries over
Read the model card for its system-message convention. Use a schema and describe the structure in the prompt. Then compare against the API you were using, on your own real tasks, before you commit to anything.
Sources and grounding
- 1Chat templatesHugging Face (Transformers docs)Source for the chat-template failure modes, including the generation prompt and doubled special tokens.
- 2Optimizing LLMs for speed and memoryHugging Face (Transformers docs)Source for the VRAM rule of thumb and the size of the key-value cache.
- 3Gemma prompt structureGoogleConfirms Gemma supports only user and model roles, with no system role.
- 4Gemma model overviewGoogleSource for the mixture-of-experts memory point and per-variant memory figures.
- 5Reasoning best practicesOpenAIStates that asking a reasoning model to think step by step is unnecessary and can hinder it.
- 6Chain-of-Thought Prompting Elicits Reasoning in Large Language ModelsWei et al., 2022 (arXiv)The finding that chain-of-thought helps only at large scale, and that small models produce fluent but illogical chains.
- 7Lost in the Middle: How Language Models Use Long ContextsLiu et al., 2023 (arXiv)The uneven-attention finding behind putting instructions at the edges of a long prompt.
- 8Structured outputsOllamaSource for schema-constrained output and the advice to describe the structure in the prompt as well.
- 9Context lengthOllamaSource for context being sized by available memory rather than by the model.
- 10Ollama CloudOllamaThe cloud mode that makes a local-looking command run on remote servers.
- 11GBNF grammarsllama.cppGrammar and JSON-schema constrained decoding, and the note that the model never sees the schema.
- 12Model deprecationsOpenAIThe notice periods behind the argument that a local model is never retired under you.
- 13Article 3: DefinitionsEU Artificial Intelligence ActProvider and deployer definitions, including putting into service for own use.
- 14Article 4: AI literacyEU Artificial Intelligence ActThe literacy duty that applies to deployers as well as providers.
- 15Opinion 28/2024 on AI models and GDPREuropean Data Protection BoardModel anonymity assessed case by case, not assumed.