Glossary

The words that show up on every model page, in plain language.

Agentic	A model working in a loop — planning, calling tools, reading results, continuing — rather than answering once. Agentic prompts need clear boundaries on what the model may do without asking.
Benchmark	A standard test set used to compare models. Useful for rough ranking, weak evidence for your specific task — models are often tuned against popular benchmarks, and your work is not the benchmark.
Chat template	The exact text format a model was trained to receive, including which special tokens mark the system, user and assistant turns. Hosted APIs apply it for you; when you self-host you must get it exactly right, and getting it wrong degrades output quietly rather than raising an error.
Context window	The total amount of text a model can hold at once, prompt and answer together, measured in tokens. Exceeding it silently drops the oldest material.
Distillation	Training a smaller model on the outputs of a larger one, to get much of the behaviour at a fraction of the size and cost. Most small open models you can run locally are distilled from something bigger.
Few-shot	Including a small number of worked examples in the prompt. Zero-shot means including none.
Fine-tuning	Further training of an existing model on your own examples, to bake in a behaviour rather than describing it in every prompt. Worth it only once prompting has genuinely stopped being enough — it costs data, money and an ongoing maintenance commitment.
GGUF	The file format most local model runtimes expect. A single file holding the weights, usually quantised, plus the metadata a runtime needs — including the chat template.
Grounding	Giving a model source material to answer from, rather than relying on what it absorbed during training.
Guardrails	Checks placed around a model rather than inside it: input filters, output validation, blocked topics, human approval before an action. A prompt instruction is a request; a guardrail is enforcement.
Hallucination	Fluent output that is factually wrong — an invented citation, statute, figure or quotation. It reads exactly like a correct answer, which is what makes it dangerous.
Inference	Running a trained model to get an answer, as opposed to training it. All the cost and waiting you experience day to day is inference.
Knowledge cutoff	The date beyond which a model has no reliable knowledge. Ask the vendor documentation, not the model — models are unreliable narrators about themselves.
KV cache	Working memory a model keeps while generating, so it need not re-read the whole conversation for every token. It grows with context length and consumes VRAM — which is why a long context costs more memory than the model weights alone suggest.
Lost in the middle	The tendency of models to attend well to the start and end of a long input and less well to the middle. A large context window is not the same as equal attention across it — which is why the instruction usually belongs at the end.
Mixture of experts (MoE)	An architecture where only a fraction of the model runs for any given token. Inference is as fast as a much smaller model, but you still need memory for all the weights — a 400B MoE with 17B active still has to fit somewhere.
Model card	The publisher’s own description of a model: what it was trained for, its licence, its prompt format, its known limits. The first thing to read before self-hosting anything, and the authority on the chat template.
Multimodal	A model that accepts more than text — images, audio or video — in the same prompt. Support varies sharply between models and even between one vendor’s own tiers, so check rather than assume.
Open weights	A model whose trained parameters you can download and run yourself. Not the same as open source: many open-weight licences carry conditions on commercial use, user numbers or naming. Read the licence, not the announcement.
Prompt injection	An attack where instructions hidden in content the model reads — a web page, a document, an email — are followed as if they came from the user.
Quantisation	Storing a model’s numbers at lower precision so it fits in less memory — an 8-bit or 4-bit version of a 16-bit model. Roughly halves or quarters the memory needed, at a quality cost that is small at 8-bit and noticeable below 4-bit.
RAG	Retrieval-augmented generation: searching a document store for relevant passages and putting them in the prompt before the model answers.
Reasoning model	A model that generates internal deliberation before its visible answer. Most frontier models now do this by default, with a control for how much.
Runtime	The software that actually loads a model file and serves answers — Ollama, llama.cpp, vLLM and others. It handles the chat template, the sampling settings and, usually, an API your own code can call.
Special tokens	Reserved markers in a model’s vocabulary that delimit turns and roles rather than carrying meaning as words. Part of the chat template, and one of the easiest things to get subtly wrong when self-hosting.
Structured output	Constraining a model to return data in a fixed shape, usually JSON matching a schema, so a program can read it.
System prompt	Durable instructions that apply to every turn of a conversation, held separately from the user’s message.
Temperature	A sampling setting controlling randomness. Lower is more predictable and repetitive; higher is more varied and more error-prone.
Thinking budget / effort	A provider setting for how much internal reasoning to spend before answering. Called effort on Claude and thinking_level on Gemini.
Throughput and latency	Latency is how long until you see an answer; throughput is how many requests per second the setup sustains. Local setups often have acceptable latency for one person and poor throughput for a team.
Token	The unit a model reads and bills in — roughly a short word or word fragment. Context windows and prices are quoted in tokens, not words.
Top-p and top-k	Sampling settings that limit which candidate next words a model may choose from. Alongside temperature they trade predictability against variety; leave them alone unless you have a measured reason.
Verbosity	An OpenAI response setting for default answer length, applied outside the prompt text.
VRAM	Memory on a graphics card. The practical ceiling on which models you can self-host: the weights plus the KV cache must fit, or the model falls back to ordinary system memory and runs far slower.