Top-p
Nucleus sampling — an alternative to temperature that restricts sampling to the smallest set of tokens whose cumulative probability reaches p. Top-p=0.9 keeps only the top 90 % probability mass and resamples from that subset.
Background
Top-p, also called nucleus sampling, is a decoding strategy that controls the randomness of a language model's output by restricting each token choice to a dynamically sized set. At every step the model produces a probability distribution over the whole vocabulary. Top-p sorts those candidate tokens from most to least probable and accumulates their probabilities until the running total first reaches the threshold p; that smallest qualifying set is the "nucleus." The model then renormalizes the probabilities within the nucleus and samples the next token only from it, discarding the long tail of unlikely tokens. Because the cutoff is by cumulative probability rather than a fixed count, the nucleus grows and shrinks with the model's confidence: when one token dominates, the set may hold just a few options; when the distribution is flat, it may include many. For example, p = 0.9 keeps just enough of the top tokens to cover 90% of the probability mass and drops the rest, so a wildly improbable token can never be sampled. This differs from top-k, which always keeps a fixed number of tokens regardless of the distribution's shape. When building with AI, top-p is a core knob for tuning behaviour: lower values (say 0.1) make output tight and near-deterministic, useful for extraction or code, while higher values loosen it for more varied, creative text. It is usually set alongside — and interacts with — temperature.