Predicting the next token

At heart, an LLM is a very well-read autocomplete.

Here is the whole trick: given the tokens so far, the model works out a probability for every possible next token, picks one, adds it, and repeats. Your whole reply is built one token at a time.

How it picks is controlled by temperature. Low temperature means it nearly always takes the most likely token (steady, repeatable). High temperature gives unlikely tokens a better chance (varied, creative, sometimes weird).

Tip: Models are first trained on enormous amounts of text to predict the next token, then tuned further to follow instructions and be helpful. The "autocomplete" at the core is why they can write, summarise and code so fluently.

Quick check

You want the same, predictable answer every time. Which temperature setting fits best?

1. Low
2. High
3. It makes no difference
4. Exactly 100

Answer: Low. Low temperature concentrates probability on the likeliest tokens, so outputs are more consistent.