Top-p (nucleus sampling)

Top-p sampling, a.k.a. nucleus sampling, sorts the next-token candidates by probability and keeps only those whose cumulative probability sums to p. At p=0.9, you sample from the smallest set of tokens that together cover 90% of the probability mass. Everything in the long tail is discarded entirely. The advantage over top-k (which always keeps the top K tokens regardless of distribution) is that top-p adapts to context. When the model is highly confident — say, the next token after "The capital of France is" — top-p might only consider 1-2 tokens. When the model is uncertain — say, the start of a creative story — top-p might keep dozens. This produces more natural variation than top-k's fixed cutoff. Top-p and temperature interact, and most APIs let you set both. A common production setting is temperature 0.7, top-p 0.95: temperature gently flattens the distribution, top-p truncates the bottom. Setting both aggressively (high temp + low p) is usually not what you want; pick one as your main lever. For deterministic-ish output, set temperature to 0 and top-p becomes irrelevant. For creative work, lean on temperature first and use top-p as a safety net to prevent the occasional very-low-probability token from making output incoherent.