April 28, 20263 min read

LLM Temperature: The One Parameter That Changes Everything

TL;DR

LLM temperature sets how randomly a model picks its next word. In a healthcare pipeline a wrong value does not just give mediocre text, it gives unpredictably wrong output. Choose a low, documented default for clinical tasks and treat any increase as an explicit design decision that you test.

LLMAI EngineeringHealthcare AIMachine Learning

Diego Muñoz Casinos

Healthcare AI Engineer

The demo below answers a question that matters in healthcare AI: at what exact point does a "stable" model start generating dangerously variable responses?

What Does LLM Temperature Actually Do?

When a model generates text, it doesn't pick a word at random: it assigns scores to all possible continuations and chooses. Temperature controls how much it respects those scores — at T=0 it always picks the highest, at T=1 it treats them almost as equals.

P(i) = exp(logit_i / T) / Σ exp(logit_j / T)

Low T amplifies the gaps between scores. The top token captures most of the probability mass and the output becomes nearly deterministic. High T compresses those gaps. The distribution flattens and tokens with lower scores gain real probability. The formula has four steps and each one does exactly one thing — click the i button in the demo to walk through them.

Interactive Demo

LLM Temperature Parameter

How randomness shapes next-token selection

Temperature0.70

0 — Greedy1 — Random

SoftmaxP(i) = exp(logit_i / T) / Σ exp(logit_j / T)

"The discussion was primarily ___"

40.1%

about

22.6%

would

14.7%

9.6%

6.3%

when

4.1%

makes

2.7%

T=0.70: probabilities spread across tokens — high variation across runs.

about40.1%

would22.6%

of14.7%

is9.6%

when6.3%

makes4.1%

we2.7%

Loading demo...

Temperature in Healthcare AI

The medical tab swaps "The discussion was primarily ___" for "Based on vitals and labs, the patient appears ___ ." Same math. Different tokens: stable, improving, mild, moderate, critical, deteriorating, uncertain.

At T=0.1, "stable" holds above 95% of probability mass. Appropriate for a triage tool where consistency is the whole point. At T=0.5, "critical" and "deteriorating" carry real probability — even when the logits strongly favor "stable." That is not an edge case. That is the parameter working as designed.

Worth flagging: T=0 is greedy decoding, not true determinism. GPU floating-point operations, INT4/INT8 quantization, and tensor parallelism can all shift which token wins. For FDA SaMD or EU MDR contexts, treat all LLM outputs as probabilistic regardless of the temperature you set.¹

Interactive Demo

Same Prompt — Three Temperatures

Hit run to re-execute — T=0 never changes

Prompt

"The patient is a 67-year-old male. HR 89 bpm, BP 134/86 mmHg, SpO₂ 96%, temp 37.5°C. Mild fatigue for 48 h. Summarize clinical status."

T = 0.0Always identical

Hemodynamically stable. Mild hypertension noted. SpO₂ within acceptable range. Fatigue is non-specific; monitoring recommended. No immediate intervention required.

T = 0.4Controlled variation

Patient is clinically stable with borderline elevated blood pressure. Oxygenation is adequate. Persistent fatigue warrants reassessment in 4–6 hours to rule out early decompensation.

T = 1.03 different outputs

Run 1Vital signs indicate a compensated state. Mild hypertension and borderline SpO₂ noted. Fatigue may be functional. Reassessment in 24 h recommended.

Run 2Elevated BP combined with reduced SpO₂ and reported fatigue may indicate early cardiovascular or pulmonary compromise. Closer monitoring advised.

Run 3Clinical picture is concerning. Persistent hypertension, borderline oxygen saturation, and fatigue could represent early deterioration. Urgent evaluation recommended.

The three T=1 outputs change on every run — from "no intervention" to "urgent evaluation" on the same input.

Loading demo...

A Safe Default

For production healthcare AI:

•T = 0.0–0.2: extraction, classification, triage scoring, adverse event coding. Same answer every time.
•T = 0.3–0.5: second-opinion tools. Human review mandatory.
•T = 0.6+: drafts, summaries, brainstorming. Not for clinical decisions.

Not a creativity dial. A precision dial.

References

1.Anthropic API Documentation — Temperature parameter. Available at: docs.anthropic.com
2.OpenAI API Documentation — Temperature parameter. Available at: platform.openai.com/docs/api-reference
3.U.S. FDA — Artificial Intelligence and Machine Learning (AI/ML)-Based Software as a Medical Device (SaMD) Action Plan (2021). Available at: fda.gov
4.Holtzman A, Buys J, Du L, Forbes M, Choi Y. The Curious Case of Neural Text Degeneration. ICLR 2020.