Temperature does not stop hallucinations
Contents
- What does temperature actually do?
- Does a lower temperature reduce hallucination?
- What actually fixed it
- The case where temperature 0 was the worst setting
- Which temperature for which job
- The knobs that actually shape output
- FAQ
- Does temperature 0 guarantee identical output?
- How do I force valid JSON instead of hoping?
- Is a hallucination at temperature 0 always the same hallucination?
- Should I just use a bigger model instead?
- Put the fact in the prompt first
Turning the temperature down does not make a model more truthful. It makes it more repetitive. I asked a local 1.2B a question it had no way to answer, at five temperatures, five times each: at temperature 0 it gave the same wrong answer five times out of five, and at 1.5 it gave four different wrong answers. Not one correct answer at any setting. If you are reaching for temperature to stop a model inventing things, you are turning the wrong knob, and this is what the right one looks like.
What does temperature actually do?
It rescales the model’s scores before they become probabilities, and nothing else. The model produces one raw score (a logit) per possible next token; temperature T divides every logit by T before the softmax turns them into probabilities:
p_i = softmax(z_i / T)
Divide by a small number and the gaps between scores get bigger, so the leading token takes almost all the probability mass. Divide by a large one and the gaps shrink toward each other, so unlikely tokens get a real chance.
At T = 0 the division is undefined, and llama.cpp’s temperature sampler handles it as an explicit branch rather than an extreme case: for any temp <= 0 it finds the highest-scoring token and sets every other logit to negative infinity. Greedy decoding, implemented inside the temperature sampler itself.
Read that again with hallucination in mind. Temperature only redistributes probability across the ranking the model already produced. If the model’s top-ranked next token is wrong, temperature 0 guarantees you get that wrong token, every single time. Nothing in the formula knows what is true.
Trap: A confident wrong answer and a confident right answer look identical to the sampler. Low temperature makes both more reliable.
Does a lower temperature reduce hallucination?
No. Measured: 0 correct answers out of 5, at every temperature tested.
The setup is LFM2.5-1.2B-Instruct (Q4_K_M) on a llama.cpp server, CPU only, asked in German which country hosted the 2026 football World Cup - a fact well past its training data. Five samples per temperature, one plain HTTP call each, no system prompt, no retrieval.
| Temperature | Correct | Distinct answers | What it settled on |
|---|---|---|---|
| 0.0 | 0/5 | 1 | ”in Katar” - five times, identical |
| 0.3 | 0/5 | 2 | ”in Katar” - four times |
| 0.7 | 0/5 | 3 | ”in Katar” - three times |
| 1.0 | 0/5 | 4 | ”in den Vereinigten Arabischen Emiraten” |
| 1.5 | 0/5 | 4 | ”in den Vereinigten Arabischen Emiraten” |
The only column that moves is variety. At 0 the model is a broken clock: reliably, reproducibly wrong. Turn the temperature up and it becomes creatively wrong instead. At no point does it say it does not know, because “I don’t know” is not a high-probability continuation of a confident question.
That last part is the mechanism worth internalising. A 1.2B has no world knowledge to fall back on and no signal that it is missing something. It is completing a sentence, and every sampler setting is just a policy for choosing among completions.
What actually fixed it
Putting the fact in the prompt. Same model, same question, same five temperatures - with one sentence of context added and an instruction to answer only from it.
| Temperature | Correct |
|---|---|
| 0.0 | 5/5 |
| 0.3 | 5/5 |
| 0.7 | 5/5 |
| 1.0 | 5/5 |
| 1.5 | 1/5 |
Perfect from 0 through 1.0, and it only falls apart at 1.5 where the sampler starts picking genuinely improbable tokens. That is a far wider usable band than the folklore suggests, and it points at the real division of labour: context decides whether the answer can be right, temperature decides how much the wording wobbles.
The prompt was three parts, and all three do work:
- The fact itself, stated plainly.
- “Answer only from the fact above.”
- “If the answer is not in it, answer UNKNOWN.”
Part 3 is the one people skip. Without an explicit escape hatch, a model has no licence to refuse, and refusing is exactly what you want when the context does not cover the question.
Rule: Every grounded prompt needs a named way out. Give the model a literal token to emit when the context is insufficient, or it will invent instead.
The case where temperature 0 was the worst setting
Here is the run I did not expect. I first wrote the grounded prompt with the question “in which country did the 2026 World Cup take place” while the supplied fact named three countries. Same fact, same instruction, only the question’s presupposition changed.
| Temperature | Correct |
|---|---|
| 0.0 | 0/5 |
| 0.3 | 1/5 |
| 0.7 | 3/5 |
| 1.0 | 3/5 |
| 1.5 | 5/5 |
At temperature 0 it answered “the answer is not in there” five times out of five. The fact was right in front of it. It had locked onto the over-literal reading - one country asked, three countries given, therefore no match - and greedy decoding cannot escape a first token it has committed to. Randomness is what let it out.
I am not going to dress this up as a general law from one prompt on one small model. But it kills the idea that 0 is a safe default you can apply without thinking. Greedy decoding does not find the best answer, it finds the highest-scoring first token and then lives with the consequences.
It also carries a plainer lesson that has nothing to do with sampling: my question presupposed something the context contradicted, and the model was not wrong to notice. Before blaming a model for failing on grounded input, check whether the question and the ground truth actually fit each other.
Which temperature for which job
| Job | Setting | Why |
|---|---|---|
| Extraction, classification, routing, tool choice | 0 | You want the same input to give the same output, forever. Reproducibility is the whole point, and there is no creative content to lose. |
| Structured generation into a schema | 0 to 0.2 | The format is constrained anyway; wobble buys nothing. |
| Summaries, rewriting, anything a human reads | 0.1 to 0.3 | Enough variation to avoid stilted phrasing, not enough to drift off the source. |
| Brainstorming, alternative phrasings | 0.7 to 1.0 | Variety is the deliverable. Assume you will discard most of it. |
Above 1.0 | avoid unless you know why | Past this, quality falls off faster than variety rises. In the grounded run above, 1.5 was the only setting that broke a task the model had otherwise nailed. |
Where a vendor publishes a recommended temperature, that wins over this table - it was tuned on the actual checkpoint. Treat the rows above as the starting point when no recommendation exists, and as the reason to deviate when you need strict reproducibility for testing.
One practical note for anyone building on a local server: temperature 0 gives you reproducibility on a single-sequence CPU run, which is genuinely useful for regression-testing prompts. It does not survive continuous batching. Users on llama.cpp issue 7052 report 5 to 8 distinct completions from identical input at temperature 0 once multiple slots are in play, because float addition is not associative and the reduction order changes with whatever else is in the batch. That is enough to flip which token holds the top logit.
The knobs that actually shape output
Temperature is one sampler in a chain, and it runs last. The default chain in llama-server, straight from its README, is:
penalties → dry → top_n_sigma → top_k → typ_p → top_p → min_p → xtc → temperature
Everything that removes candidates has already happened by the time temperature gets to reshape the distribution. Which is the practical point: temperature cannot rescue a token that an earlier filter cut, and it cannot exclude a bad one that survived. The two filters worth knowing:
top_kkeeps the k highest-scoring tokens and drops the rest, flat. Blunt, predictable, and hard to get wrong.min_pkeeps only tokens whose probability is at leastmin_ptimes the top token’s probability. It adapts to confidence: a peaked distribution gets cut hard, a flat one stays permissive - which is the failure mode oftop_pthat it was proposed to fix.repetition_penaltydiscounts tokens that already appeared. Useful against loops on small models, harmful when set high on structured output, where repetition is correct - JSON keys, XML tags, repeated field names.
Then check the model card, because vendor recommendations are per-model and they change between generations. Liquid’s own numbers make the point:
| Model | temperature | top_k | min_p | repetition_penalty |
|---|---|---|---|---|
| LFM2-1.2B (previous generation) | 0.3 | - | 0.15 | 1.05 |
| LFM2.5-1.2B-Instruct | 0.1 | 50 | - | 1.05 |
| LFM2.5-2.6B | 0.1 | 50 | - | 1.1 |
| LFM2.5-1.2B-Base | 0.3 | - | 0.15 | 1.05 |
Two things fall out of that table. The recommended temperature for the instruct models is 0.1, not the 0.7 that most tooling defaults to - if you left a chat UI on its default you have been running these models three to seven times hotter than their authors suggest. And the instruct line dropped min_p in favour of top_k=50 between generations, while the base models kept the old min_p recipe. Copying a sampling config from the previous generation, or from a base model to an instruct model, gets you settings nobody tuned for that checkpoint.
FAQ
Does temperature 0 guarantee identical output?
On a single-slot CPU llama.cpp server with the same build, prompt and settings: yes, that is what I measured - 5 of 5 identical responses across three different prompts. It is not a promise. Turn on continuous batching and it stops being true, for the reasons in the section above. Treat it as reproducible within one deployment configuration, not as a hash.
How do I force valid JSON instead of hoping?
Constrain the output rather than the sampler. llama-server’s OpenAI-compatible endpoint accepts response_format with {"type": "json_schema", "schema": {...}}, and the native /completion endpoint takes a raw GBNF grammar or a json_schema it converts for you. That makes malformed output structurally impossible, which is a stronger guarantee than any temperature setting gives you. It still cannot make the values true - that is the grounding problem, not the format problem.
Is a hallucination at temperature 0 always the same hallucination?
In this test, yes - “Katar”, five times, word for word. That is worth knowing when you evaluate a prompt: testing once at temperature 0 tells you nothing about how often the model is wrong, only that it is wrong in a stable way. Sample at a non-zero temperature if you want to see the spread of failure modes.
Should I just use a bigger model instead?
For factual recall, a bigger model has more to recall and will do better on questions like this one. But it does not change the shape of the problem: it moves the boundary of what the model knows without giving it any awareness of where that boundary is. Retrieval plus an explicit “answer UNKNOWN” escape works on every size, and it is the only thing that worked on this one.
Put the fact in the prompt first
If you take one thing from the numbers above: fix grounding before you touch sampling. Temperature has a real job, and it is a good one - controlling how much the wording varies between runs - but it was never a truth dial. Every table here came from the same small loop, and it is worth running against your own model: one question the model cannot know, five temperatures, five samples each, no system prompt and no retrieval, then the same question again with the fact pasted into the prompt. The gap between those two runs is your grounding budget.