A neural network learns by making guesses, measuring how wrong they are, and nudging its internal numbers to be less wrong next time. Repeat that millions of times and the network becomes useful. This article explains the pieces of that loop: weights, loss, gradient descent and backpropagation.
The building block: a neuron
An artificial neuron is a tiny function. It takes inputs, multiplies each by a weight, adds them up with a bias, and passes the result through an activation function:
output = activation(w1*x1 + w2*x2 + ... + b)
The weights say how important each input is. The bias shifts the result. The activation function adds non-linearity, which lets the network model curved and complex relationships instead of only straight lines. Common choices include ReLU, sigmoid and tanh.
Layers make a network
Neurons are arranged in layers: an input layer that receives the data, one or more hidden layers, and an output layer that gives the prediction. Data flows forward through the layers. This forward pass produces the network's current guess.
At the start, weights are random, so guesses are essentially noise.
Measuring error: the loss function
To improve, the network needs a number telling it how bad its guess was. That is the loss. For predicting numbers, a common loss is mean squared error, the average of the squared differences between prediction and truth. For classification, cross-entropy loss is typical.
A lower loss means better predictions. Training is the process of finding weights that minimise the loss on the training data.
Gradient descent: walking downhill
Imagine the loss as a hilly landscape where each position corresponds to a set of weights. We want the lowest valley. Gradient descent moves the weights a small step in the direction that reduces the loss most steeply.
The step size is the learning rate. Too large, and training overshoots or becomes unstable. Too small, and learning is very slow.
The update rule for each weight is:
w = w - learning_rate * (d loss / d w)
The term d loss / d w is the gradient: how much the loss changes if that weight changes slightly.
Backpropagation: computing gradients efficiently
A network can have millions of weights. How do we find the gradient for each one? Backpropagation applies the chain rule of calculus to work backward from the loss through each layer, computing every gradient in a single efficient sweep.
In words: the output layer sees the error first and works out how it should change. It passes the blame back to the previous layer, which does the same, and so on to the start. Each weight gets a signal showing whether to go up or down.
You rarely implement this by hand, because libraries such as PyTorch and TensorFlow compute gradients automatically (automatic differentiation).
A small example in Python
This minimal NumPy example trains a single neuron to learn the rule y = 2x + 1. It shows the full loop: forward pass, loss, gradients, update.
import numpy as np
# training data for y = 2x + 1
x = np.array([0.0, 1.0, 2.0, 3.0, 4.0])
y = 2 * x + 1
w, b = 0.0, 0.0 # start with arbitrary weights
lr = 0.02 # learning rate
for epoch in range(500):
pred = w * x + b # forward pass
error = pred - y
loss = np.mean(error ** 2) # mean squared error
dw = 2 * np.mean(error * x) # gradients
db = 2 * np.mean(error)
w -= lr * dw # gradient descent step
b -= lr * db
if epoch % 100 == 0:
print(f"epoch {epoch:3d} loss {loss:.4f} w={w:.3f} b={b:.3f}")
print("Learned:", round(w, 2), round(b, 2))After training, w is close to 2 and b close to 1. A deep network does the same thing at much larger scale, with automatic gradients.
Epochs, batches and overfitting
- An epoch is one full pass through the training data.
- A batch is a small group of examples used for one update. Using batches makes training faster and smoother.
- Overfitting happens when the network memorises training examples instead of learning general patterns. It is reduced with more data, regularisation, dropout and by monitoring performance on separate validation data.
Frequently Asked Questions
What is the difference between a weight and a bias?
A weight scales an input's importance. A bias is an extra constant added to the sum, letting the neuron shift its output.
Why do we need activation functions?
Without them, stacked layers would collapse into one linear function. Activations let networks learn complex, non-linear patterns.
Is backpropagation the same as gradient descent?
No. Backpropagation computes the gradients. Gradient descent uses those gradients to update the weights.
How long does training take?
It depends on model size and data. Small examples take seconds; large models can take days or weeks on specialised hardware.
Conclusion
Neural networks learn through a simple cycle: predict, measure loss, compute gradients with backpropagation, and update weights with gradient descent. Everything else, from image recognition to language models, is built on that loop.