If you ask a language model the same question twice, you may get two different answers. That is not a bug; it is by design. The model assigns a probability to every possible next token, and the system then samples from those probabilities. Settings like temperature and top-p control how adventurous that sampling is.
From scores to probabilities
At each step the model produces a score for every token in its vocabulary. These scores are converted into probabilities that add up to 1. For example, after "The capital of Nepal is", the model might assign:
- " Kathmandu": 0.97
- " the": 0.01
- " Lalitpur": 0.005
- other tokens: 0.015 combined
The decoding strategy decides which token to choose.
Greedy decoding
Always choosing the highest-probability token is called greedy decoding. It is deterministic and consistent, but can be repetitive and may lead to dull or looping text on open-ended tasks.
Temperature
Temperature reshapes the probability distribution before sampling.
- Low temperature (near 0): makes the distribution sharper. The most likely tokens become even more likely, so output is focused, predictable and more repeatable.
- Medium temperature (around 0.7 to 1): balanced variety.
- High temperature (above 1): flattens the distribution, so unlikely tokens get a real chance. Output becomes more varied, creative and also more error-prone or incoherent.
Mathematically, scores (logits) are divided by the temperature before the softmax. A small temperature magnifies differences between scores; a large one shrinks them.
import numpy as np
def softmax(x):
e = np.exp(x - np.max(x))
return e / e.sum()
tokens = [" Kathmandu", " the", " Lalitpur", " Pokhara"]
logits = np.array([5.0, 1.0, 2.0, 1.5])
for t in (0.2, 1.0, 2.0):
probs = softmax(logits / t)
print(f"temperature {t}:", {tok: round(float(p), 3) for tok, p in zip(tokens, probs)})Run it and watch the top token dominate at low temperature and the others gain probability at high temperature.
Top-p (nucleus sampling)
With top-p, the model considers only the smallest set of tokens whose probabilities add up to p, then samples within that set.
- Top-p = 0.9: consider tokens until their cumulative probability reaches 90%, ignoring the long tail of unlikely ones.
- Top-p = 0.5: only the most likely few tokens are allowed.
- Top-p = 1.0: no restriction.
Top-p adapts to context: when the model is confident, the set is tiny; when many continuations are plausible, the set is larger.
Top-k
Some systems offer top-k, which keeps only the k most likely tokens (for example 40). It is a fixed-size cut-off, unlike top-p, which adjusts dynamically.
Other related settings
- Max tokens: the length limit for the reply.
- Stop sequences: strings that end the generation.
- Frequency and presence penalties: discourage repeating the same words or topics.
- Seed: some APIs let you fix the random seed for more reproducible output, though identical results are not always guaranteed across systems.
Which settings for which task?
| Task | Suggested approach |
|---|---|
| Data extraction, classification, JSON output | Temperature at or near 0 |
| Code generation, maths, factual answers | Low (0 to 0.3) |
| Summaries, emails, general chat | Medium (0.3 to 0.7) |
| Brainstorming, stories, slogans | Higher (0.8 to 1.0 or more) |
These are starting points. Defaults and valid ranges differ between providers, and some newer models fix or restrict sampling parameters, so check the documentation for your model.
A common recommendation is to change either temperature or top-p, not both at once, so you can understand the effect.
Using them in code
import anthropic
client = anthropic.Anthropic()
def ask(prompt: str, temperature: float):
reply = client.messages.create(
model="claude-sonnet-5-5",
max_tokens=120,
temperature=temperature,
messages=[{"role": "user", "content": prompt}],
)
return reply.content[0].text
prompt = "Suggest a name for a small tea shop in Pokhara."
for t in (0.0, 1.0):
print(f"--- temperature {t}")
for _ in range(3):
print(ask(prompt, t))At temperature 0 the three answers will be identical or nearly so; at higher values they will differ. Check your provider's documentation, since parameter names and limits vary.
Low temperature does not mean true
A common misunderstanding: reducing randomness does not make the model more accurate about facts. It just makes it choose the most probable continuation consistently. If the most probable answer is wrong, a low temperature will repeat the same wrong answer every time. Accuracy depends on the model's knowledge, grounding in documents and verification.
Reproducibility
For tests and automated pipelines, use low temperature (or 0), fixed prompts and, if available, a seed. Still expect slight variations between model versions and sometimes between runs, because of numerical differences on hardware.
How to experiment
- Pick 10 sample prompts for your task.
- Run each at several temperatures, three times per setting.
- Rate the outputs for correctness, style and consistency.
- Choose the lowest temperature that gives enough variety for your use.
- Re-check after changing models or prompts.
Frequently Asked Questions
What is the best temperature?
There is no single best value. Use low values for precision and higher values for creativity, and test on your own task.
Is temperature 0 completely deterministic?
Usually close to it, but small differences can still appear due to implementation details.
What is the difference between temperature and top-p?
Temperature reshapes the whole distribution. Top-p cuts off the unlikely tail. Both control randomness in different ways.
Do I need to set these in a chat app?
Most chat apps hide them. They are mainly options for developers using APIs or advanced tools.
Conclusion
Temperature and top-p tune the balance between predictable and creative text. Lower them for accuracy and consistency, raise them for variety, change one at a time and always verify facts regardless of settings.