Temperature

Background

Temperature is a sampling parameter that controls how random a model's next-token choice is. A language model outputs a vector of logits — one raw score per possible token — which are turned into a probability distribution with the softmax function. Temperature divides those logits before softmax is applied. Dividing by a value below 1 sharpens the distribution, pushing probability mass toward the already-likely tokens, so the model becomes more deterministic and repetitive. Dividing by a value above 1 flattens the distribution, raising the odds of less-likely tokens and producing more varied, sometimes more surprising output. At a temperature of 0 the model effectively always takes the single highest-probability token (greedy decoding), giving reproducible output for the same input. Concretely, generating a factual answer or structured data benefits from a low temperature so the model stays on the most probable, consistent path, while brainstorming names, writing creative copy, or wanting diverse completions benefits from a higher one. When building software with AI, temperature is your main knob for the tradeoff between reliability and diversity, and it interacts with truncation samplers like top-k and top-p that first restrict the candidate set. A caveat worth remembering: even at temperature 0, output is only reproducible up to hardware and implementation details, and low temperature reduces variety but does not make a wrong answer correct — it only makes the model commit harder to whatever it already favours.