If you read about modern AI, you will see the word "transformer" constantly. It is the architecture behind most large language models and many systems for images, audio and protein structure. Introduced in a 2017 research paper titled "Attention Is All You Need", it changed AI because it handles sequences faster and more effectively than earlier designs.
The problem transformers solved
Before transformers, text was usually processed by recurrent neural networks (RNNs) and their improved versions, LSTMs. These read a sentence one word at a time, carrying a memory forward. That had two weaknesses.
- Slow training: each step depends on the previous one, so computation cannot easily be run in parallel.
- Forgetting: information from the start of a long sentence tended to fade by the end.
Transformers removed the step-by-step dependency. They look at all positions in the input at the same time and let every word directly "attend" to every other word.
The key ingredient: self-attention
Self-attention lets the model decide, for each word, which other words are important for understanding it.
Take: "The animal did not cross the street because it was too tired." To interpret "it", the model needs to link it to "animal". Self-attention computes a score between "it" and every other word, and combines information from the high-scoring words.
Technically, each token produces three vectors:
- Query: what this token is looking for
- Key: what this token offers
- Value: the information it passes on
Scores come from comparing a token's query with all keys. The scores are normalised into weights, and the output is a weighted sum of the values. All tokens are processed in one set of matrix operations, which GPUs execute very efficiently.
Multi-head attention
Instead of one attention calculation, transformers run several in parallel, called heads. Different heads can specialise: one may track grammar, another may follow references, another may look at nearby words. Their results are combined.
Position information
Because attention itself ignores order, transformers add positional encodings to the token vectors so that the model knows which word came first. Newer designs use methods such as rotary position embeddings.
The building block
A transformer layer typically contains:
- A multi-head self-attention step
- A small feed-forward neural network applied to each token
- Residual connections and normalisation to keep training stable
Many layers are stacked, sometimes dozens, so representations become richer layer by layer.
Encoder, decoder and both
- Encoder-only models (BERT-style) read text in both directions and are strong at understanding tasks like classification and search.
- Decoder-only models (GPT-style, and most chat models) generate text left to right, predicting the next token with a mask that hides future words.
- Encoder-decoder models (T5-style) read an input and generate an output, suited to translation and summarisation.
Why transformers changed AI
- Parallel training: much faster use of modern hardware.
- Long-range connections: direct links between distant words.
- Scalability: performance kept improving as models and data grew, which led to large language models.
- Transferability: pretrain once on huge data, then adapt to many tasks.
- Versatility: the same idea works for images (vision transformers), audio, video and biology.
A tiny attention example in code
This simplified NumPy snippet shows the central computation, scaled dot-product attention:
import numpy as np
def softmax(x):
e = np.exp(x - x.max(axis=-1, keepdims=True))
return e / e.sum(axis=-1, keepdims=True)
def attention(Q, K, V):
d = Q.shape[-1]
scores = Q @ K.T / np.sqrt(d) # how much each token matches every other
weights = softmax(scores) # each row sums to 1
return weights @ V, weights # weighted mix of values
np.random.seed(0)
tokens = 4
Q = np.random.randn(tokens, 8)
K = np.random.randn(tokens, 8)
V = np.random.randn(tokens, 8)
out, w = attention(Q, K, V)
print("Attention weights:\n", w.round(2))Each row of the weights shows how much one token pays attention to all the tokens. Real transformers add learned projections, several heads and many layers.
Limits of transformers
- Cost grows with sequence length: standard attention compares every token with every other, so long inputs become expensive. Many optimisations exist.
- Huge data and compute needs for the largest models.
- No built-in reliability: they can still hallucinate and reflect biases in data.
- Fixed context window: they only see a limited amount of text at once.
Frequently Asked Questions
Is a transformer the same as ChatGPT?
No. A transformer is the architecture. Chat systems are large transformer models trained and tuned to hold conversations.
What is the difference between BERT and GPT?
BERT is an encoder trained to understand text with context from both sides. GPT is a decoder trained to predict the next token, which makes it natural for generation.
Why is it called a transformer?
The paper described a model that transforms one sequence into another using attention, without recurrence.
Are transformers only for language?
No. They are used in image models, speech recognition, music generation, protein structure prediction and more.
Conclusion
Transformers replaced step-by-step reading with self-attention across the whole sequence. That made training parallel, captured long-range meaning and scaled remarkably well, which is why they sit at the centre of modern AI.