Open a video app and the home page is full of things you might like. Open a shopping app and "recommended for you" appears. Behind these suggestions are recommendation systems, which sort millions of items down to a handful chosen for you. Understanding them helps you build one and also understand why your feed looks the way it does.
The goal
A recommender predicts how likely a given user is to engage with a given item (click, watch, buy, rate) and shows the items with the highest expected value. Engagement might be measured by watch time, purchases or ratings, depending on the business.
Types of recommendation methods
Popularity-based
The simplest approach shows what is popular overall or trending. It works for new users with no history but is not personal.
Content-based filtering
The system recommends items similar to ones you liked, based on their attributes: genre, actors, product category, price, text description. If you watch many cooking videos, it suggests more cooking videos. It does not need other users' data, but may be repetitive.
Collaborative filtering
This method uses the behaviour of many users. If people with tastes similar to yours liked an item you have not seen, it is a good candidate.
- User-based: find users similar to you and recommend what they liked.
- Item-based: find items that are often liked by the same people.
- Matrix factorisation: learn hidden "taste" factors for users and items so that their product predicts ratings or clicks. A user vector and item vector that point in a similar direction produce a high score.
Deep learning and embeddings
Modern systems learn embeddings for users, items and context with neural networks. Sequence models consider what you did recently, such as the last ten videos, to predict what comes next.
Hybrid systems
Real products combine many signals: collaborative data, content features, context (time, device, location) and business rules.
The two-stage pipeline
With millions of items you cannot score every item for every user in real time. Large systems usually work in stages.
- Candidate generation: quickly retrieve a few hundred plausible items using simple models, embeddings and nearest-neighbour search.
- Ranking: a more precise model scores these candidates using many features and orders them.
- Re-ranking and rules: adjust for diversity, freshness, safety policies, ads and business goals, and remove items you already saw.
Implicit and explicit feedback
- Explicit: ratings, likes, thumbs up or down. Valuable but scarce.
- Implicit: clicks, watch time, purchases, skips, searches. Plentiful but noisy; a click does not always mean you liked it.
Most systems rely mainly on implicit signals.
A small example in Python
The code below builds a basic item-based recommender from a tiny ratings table using cosine similarity:
import numpy as np
import pandas as pd
from sklearn.metrics.pairwise import cosine_similarity
ratings = pd.DataFrame(
{"Movie A": [5, 4, 0, 1],
"Movie B": [4, 5, 1, 0],
"Movie C": [0, 1, 5, 4],
"Movie D": [1, 0, 4, 5]},
index=["Asha", "Bikash", "Chandra", "Dipa"],
)
item_sim = pd.DataFrame(cosine_similarity(ratings.T), index=ratings.columns, columns=ratings.columns)
def recommend(user: str, top_n: int = 2):
seen = ratings.loc[user]
scores = {}
for item in ratings.columns:
if seen[item] == 0: # not yet rated
liked = seen[seen > 0].index
scores[item] = sum(item_sim.loc[item, l] * seen[l] for l in liked) / len(liked)
return sorted(scores.items(), key=lambda kv: kv[1], reverse=True)[:top_n]
print(recommend("Asha"))A zero means "not rated". Real systems handle millions of users with sparse data and efficient libraries.
Evaluating recommenders
- Offline metrics: precision at k, recall at k, NDCG and RMSE on held-out data.
- Online experiments: A/B tests compare real user behaviour with a new algorithm versus the old one.
- Beyond accuracy: diversity, novelty, fairness to creators and long-term satisfaction.
Offline results do not always match real behaviour, so online testing is crucial.
Common challenges
- Cold start: new users and items lack data. Solutions include asking preferences, using content features and showing popular items.
- Filter bubbles: showing only similar content narrows exposure. Adding diversity helps.
- Popularity bias: popular items get more exposure and become more popular.
- Feedback loops: the system's choices shape the data it learns from next.
- Manipulation and spam: fake reviews and click farms distort signals.
- Privacy: personalisation relies on tracking behaviour, so transparency and controls are important.
Taking control as a user
Most apps let you adjust what you see. Remove items from your history, mark content as "not interested", clear watch history, use separate profiles and review privacy settings. If you notice the feed is pushing you toward content you do not want, these signals do change future suggestions.
Frequently Asked Questions
Do apps listen to my conversations to recommend things?
There is no public evidence that this is how mainstream recommenders work. Suggestions are usually explained by your activity, searches, location and what similar people do, which can look uncanny.
Why do I see the same type of videos repeatedly?
Systems optimise for engagement, so they show what you have already engaged with. Searching for different topics and giving feedback broadens the mix.
Can I build a recommender without much data?
Yes. Start with popular and content-based recommendations, then add collaborative methods as data grows.
Is a recommendation system the same as search?
No. Search responds to a query. Recommendation proactively suggests items without a query.
Conclusion
Recommendation systems predict what you will engage with using item content, behaviour of similar users and context, in a pipeline that retrieves candidates and ranks them. Knowing how they work helps you build better ones and manage your own feeds.