You Set Temperature to Zero and the Output Still Changes
You set temperature to 0 so the same prompt returns the same tokens, then your eval flips red on a rerun with nothing changed. The cause is not your sampler.
A green eval that would not stay green
A team I was helping had a classifier eval wired into CI. One hundred cases, exact string match against a gold label, temperature set to 0 so the whole thing would be deterministic. It was green for weeks. Then one afternoon it went red on a pull request that only touched a README. They reran it. Green. They reran it again. Red, a different case this time.
Nobody had changed the prompt, the model string, or the eval cases. The obvious suspects were all innocent. The author of the failing pull request spent an hour reading a diff that could not possibly have caused it, which is the most expensive kind of hour because at the end of it you have learned nothing and you trust your pipeline less than when you started.
The reflex explanation is that the model is random and you should just accept some noise. That is half true and completely useless. The useful version is specific: temperature 0 fixes the sampler, not the numbers the sampler reads, and on a shared server those numbers move. Once you see why, the fix stops being a superstition and becomes an engineering decision with a known cost.
Why temperature zero is not the switch you think it is
Generating a token is two steps. The model does a forward pass that produces a vector of logits, one score per token in the vocabulary. Then a sampler turns those logits into a choice. Temperature lives entirely in the second step. At temperature 0 the sampler becomes argmax: pick the token with the highest logit, every time, no dice involved.
So people reason that argmax of a fixed vector is deterministic, therefore temperature 0 is deterministic. The hidden assumption is that the logit vector is fixed. It is not guaranteed to be. If the forward pass hands argmax a vector where the top two tokens are separated by a hair, and the numbers wobble by one part in a million between runs, argmax can flip. One flipped token early in a decode changes everything after it, because the model conditions each new token on the ones before. A single bit at position three becomes a completely different sentence by position thirty.
This is why "set temperature to 0 and move on" quietly fails in production. You removed the loud source of randomness and assumed you removed all of them. The quiet source is upstream, in the arithmetic that builds the logits, and it does not care what your sampler is doing. If you have ever shipped a structured output gateway and watched a field occasionally come back in a different shape at temperature 0, this is the mechanism underneath it.
The real cause is batch invariance
Horace He and collaborators at Thinking Machines laid this out carefully in Defeating Nondeterminism in LLM Inference, and the diagnosis is sharper than the usual hand waving about "floating point and concurrency." Their point: a matrix multiply run twice on the same inputs on the same GPU returns bitwise identical results. The forward pass is not randomly corrupting itself run to run. So where does the variation come from?
It comes from batch size. High performance kernels change how they split and accumulate work depending on how many sequences are in the batch. A reduction summed in one order for a batch of 1 is summed in a different order for a batch of 48, and because floating point addition is not associative, (a + b) + c is not always a + (b + c) at the bit level. The kernel is not batch invariant: its output for your request depends on who else was in the batch.
On a real server you do not control the batch. Requests arrive, the scheduler groups whatever is in flight, and the batch your prompt rides in at 2pm under load is not the batch it rode in at 2am when you were alone. Same prompt, same weights, same temperature 0, different batch, different bits, occasionally a different token. The Thinking Machines experiment makes the size of this concrete: a single prompt sent to Qwen3-235B at temperature 0, one thousand times, produced eighty distinct completions. With batch invariant kernels enabled, all one thousand were identical.
# The experiment that exposes it. Not a bug in your code.
# Same prompt, temperature 0, many times, against a shared endpoint.
import collections, openai
client = openai.OpenAI()
prompt = "List the first 20 primes as a JSON array."
outputs = collections.Counter()
for _ in range(1000):
r = client.chat.completions.create(
model="your-served-model",
temperature=0,
seed=42, # seed does nothing at temp 0
messages=[{"role": "user", "content": prompt}],
)
outputs[r.choices[0].message.content] += 1
# If len(outputs) > 1, your "deterministic" setting is not.
print(len(outputs), "distinct outputs")
If that counter comes back greater than 1, you have measured your own noise floor. That number is the real reason your eval gate in CI flickers, and it has nothing to do with the code in the pull request that happened to be running when it flipped.
Where this actually bites you
The flaky eval is the loudest symptom, but it is not the only one.
Evals that assert on exact strings inherit the full noise floor. Every rerun is a fresh roll, so a case near a decision boundary flips on its own schedule and you learn to rerun until green, which is the exact habit that lets a real regression slip through. This is also why single sample scoring lies to you and pass^k style consistency checks tell you more: they measure the distribution instead of one draw from it.
Prompt caching assumes a stable prefix produces a stable continuation. When the same cached prefix can yield a slightly different logit vector depending on batch, your cache hit and your fresh computation can disagree, and a prefix cache that was supposed to be a pure speedup becomes a subtle source of divergence between the cached and uncached path.
LLM as judge pipelines compound it twice. The thing being graded is nondeterministic and the grader is nondeterministic, so a borderline answer gets a borderline score that wobbles, and you end up debating a 0.02 move in a metric that was never that precise. If you lean on a judge for your evals, you want to know how much of its variance is the model under test and how much is the judge's own jitter.
Reinforcement learning setups get the worst of it. If the sampler that generates rollouts and the trainer that computes gradients disagree at the bit level, your training signal is quietly off policy. This is exactly the gap the November 2025 paper Deterministic Inference across Tensor Parallel Sizes targets, with kernels that stay invariant even when the tensor parallel layout changes between training and inference.
Making inference reproducible when you control the server
If you host the model yourself, you can buy back determinism. vLLM ships a batch invariant mode, documented as beta, that pins the reduction order so a request's output depends only on its input and not on batch composition. You turn it on with an environment variable.
# vLLM batch invariant serving. One flag, real throughput cost.
# Reduction order is fixed, so output no longer depends on batch size.
VLLM_BATCH_INVARIANT=1 \
vllm serve your-model \
--tensor-parallel-size 2
# Verify it actually took: send the same prompt under varying load
# and confirm the distinct-output count collapses to 1.
The vLLM batch invariance docs are explicit that this "may impact performance compared to the default non-deterministic mode," and that the trade is intentional. That is the honest version of the decision: you are spending throughput to buy reproducibility. You almost never want this on for your whole production fleet. You want it on for the lane where reproducibility earns its cost: the eval runner, a debugging reproduction, a training rollout worker, a golden regression suite. Route those to a batch invariant endpoint and let the rest of your traffic run fast and loose.
Pinning the kernels is necessary but not sufficient. Reproducibility also needs a pinned model build, not just a model name. "gpt-ish-4" or "your-model:latest" can point at a different set of weights next week, and a weights change is a legitimate output change that you want to see as a deliberate event, not discover through a mystery eval flip. Record the exact build identifier with every score, the same way you would pin and record it for a living eval set so a serving change can never masquerade as a quality change.
What to do when you do not control the server
Most teams call a hosted API and cannot set VLLM_BATCH_INVARIANT on someone else's fleet. You get no batch control, no kernel control, and a seed parameter the provider documents as best effort. So you engineer around the noise instead of eliminating it.
Stop asserting on exact strings. Assert on the property you actually care about. If you want valid JSON, validate the schema and the field types, not the byte sequence. If you want a number, check it against a tolerance. If you want a judgment, grade it with a rubric and accept a band, not a point. An exact match assertion on a probabilistic system is a test that measures the wrong thing precisely.
# Stable eval: sample N times, score a property, judge the distribution.
def eval_case(call, case, n=5):
results = [score(call(case.prompt), case.expected) for _ in range(n)]
pass_rate = sum(results) / n
# Gate on the rate with a margin, not on a single green run.
# Record the model build so a serving change is visible, not silent.
return {
"case": case.id,
"pass_rate": pass_rate,
"passed": pass_rate >= case.threshold, # e.g. 0.8, not 1.0
"model_build": call.model_build,
}
Run each case several times and gate on the rate with a margin. A case that passes four times in five is not the same as one that passes zero in five, and a single run cannot tell them apart. Treat this as the same discipline you would apply to debugging a nondeterministic agent: you cannot replay the exact tokens, so you capture enough structure to reason about the run instead of chasing one trace. And keep the model build in every record, because the one change you genuinely need to catch, a silent model update underneath a stable alias, hides perfectly inside noise you have decided to ignore.
The parts that bite
Batch invariant mode is not free and the docs say so. Measure the throughput hit on your own model and hardware before you route real volume through it. For an eval lane the cost is usually irrelevant because the volume is small, but do not quietly flip it on for production and then wonder where your tokens per second went.
Determinism does not mean correct. A batch invariant endpoint will reproduce a wrong answer exactly as faithfully as a right one. Reproducibility is a debugging and measurement tool, not a quality improvement. It makes your failures legible, which is worth a great deal, but it does not make them fewer.
Pinning one layer and forgetting the others gives false confidence. Kernels, batching, model build, tokenizer version, and sampler settings all have to be nailed down together, because any one of them drifting reintroduces exactly the variation you thought you had removed. Reproducibility is a property of the whole stack, and it is only as strong as the layer you forgot to pin.
Finally, do not over correct into forcing determinism everywhere. Your users are served by the fast, batch variant path, and that is the right call for them. The goal is a reproducible lane for the work that needs it, not a slower product for everyone. Keep the two separate on purpose, and know which requests are in which lane.
The takeaway
Temperature 0 was never a determinism switch. It fixes the sampler and leaves the arithmetic that feeds the sampler free to wobble, and on a shared server that arithmetic wobbles with batch size because the kernels are not batch invariant and floating point addition does not forgive a change in order. That is the whole mystery behind the eval that flips on a README change. It is not your prompt, your code, or a flaky model being flaky. It is the server, and it is explainable, measurable, and fixable.
If you own the server, pin the kernels for the lane that needs reproducibility and pay the throughput for it knowingly. If you rent the model, stop testing exact strings, sample enough to see the distribution, and record the build with every score so a real change cannot hide in the noise. If your evals flicker and no one can say why, the fastest way to stop arguing about it is to measure your own noise floor first. I am happy to look at it with you, thirty minutes, no pitch.
Questions this essay answers
01Why is temperature 0 not deterministic?
Temperature 0 removes the sampler as a source of randomness, so it fixes which token the model would pick given a fixed set of logits. It does not fix the logits. On a shared inference server the GPU kernels that compute those logits often change their reduction strategy with batch size, and batch size changes with how many other requests arrive at the same moment. Floating point addition is not associative, so a different summation order produces slightly different numbers, which can flip the top token. Your request is identical, but the batch it rode in was not.
02Does setting a seed make LLM output reproducible?
A seed only controls the random number generator the sampler uses. At temperature 0 there is nothing to sample, so a seed changes nothing. Even with sampling, a seed cannot undo batch variance in the forward pass, which is why provider seed parameters are documented as best effort rather than a guarantee. A seed helps you reproduce the sampling decision given fixed logits. It does not give you fixed logits. If you need bit for bit reproducibility you have to pin the kernels and the batching, not just the seed, and verify it with a repeated run.
03What is batch invariance in LLM inference?
Batch invariance means a request produces the same output regardless of how many other requests are processed alongside it. Most high performance kernels are not batch invariant by default: they split and accumulate work differently for a batch of 1 than for a batch of 64, and the differing floating point order yields different bits. Batch invariant kernels fix the reduction order so the result depends only on the input. They cost some throughput, which is why serving stacks leave them off unless you ask for reproducibility.
04How do I make my evals stable without reproducible inference?
Stop asserting on exact strings. Score the property you care about: a schema check, a numeric tolerance, a rubric graded by a judge, or a pass rate over several samples. Run each eval case more than once and treat the distribution, not a single run, as the result. Pin the model build and record it with every score so a change in the serving stack cannot masquerade as a quality regression. This does not remove nondeterminism, but it stops nondeterminism from reading as a red build.
