Training a deep neural network from scratch needs enormous data and computing power. Transfer learning avoids most of that cost. You start with a model already trained on a large dataset and adapt it to your own problem. It is the reason a small team with a few hundred photos can build a useful image classifier in an afternoon.
The idea
A network trained on millions of images learns general visual skills in its early and middle layers: edges, textures, shapes, parts of objects. Those skills are useful for almost any image task. Only the final layers are specific to the original classes. So instead of learning everything again, you reuse the general layers and train only what is needed for your task.
The same principle applies to language: models pretrained on huge text collections learn grammar, facts and reasoning patterns that transfer to classification, translation or question answering.
Two main strategies
1. Feature extraction
Freeze the pretrained network (do not update its weights), remove its final classification layer, and use its outputs as features. Then train a small new classifier on top using your data.
- Very fast and needs little data
- Low risk of overfitting
- Good when your task is similar to the original
2. Fine-tuning
Start from the pretrained weights and continue training some or all layers on your data, with a small learning rate.
- Can reach higher accuracy, especially when your data differs from the original
- Needs more data and care
- Common practice: first train only the new head, then unfreeze later layers gradually
A simple rule: small dataset and similar domain means feature extraction; larger dataset or different domain means fine-tune more layers.
A PyTorch example
This example adapts a pretrained ResNet-18 to a two-class problem (for example, healthy versus diseased leaves). The dataset is stored in folders data/train/healthy, data/train/diseased, and similarly for validation.
import torch
from torch import nn, optim
from torchvision import datasets, models, transforms
weights = models.ResNet18_Weights.DEFAULT
preprocess = weights.transforms()
train_ds = datasets.ImageFolder("data/train", transform=preprocess)
val_ds = datasets.ImageFolder("data/val", transform=preprocess)
train_dl = torch.utils.data.DataLoader(train_ds, batch_size=32, shuffle=True)
val_dl = torch.utils.data.DataLoader(val_ds, batch_size=32)
model = models.resnet18(weights=weights)
for p in model.parameters(): # freeze the pretrained layers
p.requires_grad = False
model.fc = nn.Linear(model.fc.in_features, len(train_ds.classes)) # new trainable head
device = "cuda" if torch.cuda.is_available() else "cpu"
model.to(device)
optimizer = optim.Adam(model.fc.parameters(), lr=1e-3)
loss_fn = nn.CrossEntropyLoss()
for epoch in range(5):
model.train()
for x, y in train_dl:
x, y = x.to(device), y.to(device)
optimizer.zero_grad()
loss_fn(model(x), y).backward()
optimizer.step()
model.eval()
correct = total = 0
with torch.no_grad():
for x, y in val_dl:
pred = model(x.to(device)).argmax(1).cpu()
correct += (pred == y).sum().item()
total += len(y)
print(f"epoch {epoch + 1}: validation accuracy {correct / total:.2%}")To fine-tune further, unfreeze the last block of the network and train it with a much lower learning rate such as 1e-4 or 1e-5.
Transfer learning in language
With the Hugging Face transformers library you can load a pretrained language model and fine-tune it for classification with relatively little labelled text. Another light option is to use embeddings from a pretrained model as input to a simple classifier like logistic regression.
For modern large language models, there are also parameter-efficient techniques such as LoRA, which train a small number of added parameters instead of the whole model, greatly reducing memory and cost.
When transfer learning works well
- Your data is limited
- The pretrained model's domain is related to yours (everyday photos for general image tasks, general web text for general language tasks)
- You need results quickly
- You want strong baselines
When it works less well
- Domain mismatch: features from natural photos may transfer imperfectly to medical scans, satellite images or microscope slides, though fine-tuning often still helps.
- Different input types: a model for colour photos might need adjustment for spectrograms or multispectral data.
- Licence and data restrictions: check the licence of the pretrained model and the data it was trained on.
- Hidden biases: the pretrained model carries biases from its original data.
- Size and speed: large models may be too heavy for phones or edge devices. Consider smaller architectures or distillation.
Practical tips
- Use the same preprocessing the pretrained model expects (image size, normalisation, tokeniser).
- Start with a small learning rate for pretrained layers.
- Use data augmentation to stretch small datasets.
- Monitor validation metrics and stop early if they worsen.
- Compare with a simple baseline to see if the added complexity helps.
- Check results by group and condition for fairness and robustness.
- Save the preprocessing steps and class names alongside the model for deployment.
Frequently Asked Questions
What is the difference between transfer learning and fine-tuning?
Transfer learning is the general idea of reusing knowledge. Fine-tuning is one method: continuing training on the new task.
How much data do I need?
With strong pretrained models, even a few hundred labelled images per class can be enough for simple tasks. More data helps for harder ones.
Can I use transfer learning on a laptop?
Yes, especially feature extraction or small models. Free cloud notebooks with GPUs also help.
Is a pretrained model always better than training from scratch?
On small and medium datasets, almost always. For very large datasets in a unique domain, training from scratch or from domain-specific pretraining may win.
Conclusion
Transfer learning lets you stand on the shoulders of models trained on massive datasets. Freeze and extract features for small data, fine-tune for larger or different tasks, and always validate on your own data.