How Large Language Models Work: Tokens, Attention and Prediction

Large language models, or LLMs, power modern chatbots, writing assistants and coding helpers. They can look almost magical, but the core mechanism is surprisingly simple to state: an LLM predicts the next piece of text, over and over. This article walks through the main steps so that the technology feels less mysterious.

Step 1: text becomes tokens

Computers cannot work with letters directly, so the first step is tokenisation. The text is cut into small pieces called tokens. A token can be a whole word, part of a word or punctuation. For example, a sentence like "Unbelievable results" might become pieces such as "Un", "believ", "able" and " results".

Each token maps to an integer ID from a fixed vocabulary, usually tens of thousands of entries. The model never sees the original letters, only these IDs.

Tokens matter in practice because:

• Pricing and limits for AI services are usually counted in tokens.
• Different languages split into different numbers of tokens. Text in some scripts needs more tokens per word, so the same content can cost more or fit less in the context window.

Step 2: tokens become vectors

Each token ID is converted into a list of numbers called an embedding. Words with related meanings end up with similar embeddings. These vectors are what the neural network actually computes with.

Because order matters ("dog bites man" is not "man bites dog"), the model also adds information about each token's position.

Step 3: attention connects the words

The key invention behind modern LLMs is the transformer architecture and its attention mechanism. Attention lets each token look at other tokens in the text and decide how much each one matters for understanding it.

Take the sentence: "The trophy did not fit in the suitcase because it was too big." What does "it" refer to? Attention helps the model weigh "trophy" and "suitcase" when processing "it", using clues from the rest of the sentence.

A transformer stacks many layers of attention and small neural networks. Each layer refines the representation of every token using information from the others.

Step 4: predicting the next token

At the end, the model produces a score for every token in its vocabulary, saying how likely each one is to come next. These scores are turned into probabilities.

The system then picks one token, appends it to the text, and repeats the whole process. A paragraph is simply the result of hundreds of these small steps.

Here is the idea in pseudo-Python:

def generate(model, tokens, max_new_tokens=50):
    for _ in range(max_new_tokens):
        probabilities = model.next_token_probabilities(tokens)
        next_token = pick(probabilities)   # e.g. sample or take the highest
        tokens.append(next_token)
        if next_token == END_OF_TEXT:
            break
    return tokens

How the pick step works affects the style. Always choosing the highest probability gives predictable text. Sampling with some randomness (controlled by settings like temperature) gives more varied text.

How the model learns

LLMs are trained in stages.

1. Pretraining: the model reads a huge amount of text and learns to predict masked or next tokens. This teaches grammar, facts, writing styles and some reasoning patterns.
2. Instruction tuning: the model is trained on examples of questions and good answers so that it follows instructions rather than just continuing text.
3. Preference tuning: human or automated feedback teaches the model which answers are more helpful and safer.

Learning means adjusting billions of parameters so that predictions improve. The knowledge is stored in those numbers, not in a database of sentences.

Why LLMs make mistakes

Understanding the mechanism explains common problems.

• Hallucinations: the model is trained to produce plausible text, not to check truth. If it lacks the fact, it may still generate a convincing but wrong answer.
• Limited context: the model only "sees" the text inside its context window. Anything outside is forgotten.
• Sensitivity to wording: small changes in a prompt can change the answer because the prediction depends on the exact tokens.

These limits are why important outputs should be verified, and why techniques like retrieval, where the model is given trusted documents, are popular.

Frequently Asked Questions

Do LLMs understand language?

They model language statistics extremely well and can perform many tasks that look like understanding. Whether this equals human understanding is debated, but they have no experiences or beliefs of their own.

Where does the model store its knowledge?

In its parameters, the numerical weights adjusted during training. It does not look up facts in a table unless it is connected to external tools or documents.

Why do different answers appear for the same question?

Most systems sample from probabilities, so the chosen tokens can differ between runs. Lowering the randomness setting makes answers more consistent.

What is a context window?

It is the maximum amount of text, measured in tokens, the model can consider at once, including your prompt and its reply.

Conclusion

A large language model turns text into tokens, converts them to vectors, uses attention to relate them, and predicts the next token repeatedly. Knowing this simple loop explains both why LLMs are so capable and why they need careful checking.