Computers can only calculate with numbers, yet we want them to understand words, documents, images and songs. Embeddings bridge that gap. An embedding is a list of numbers (a vector) that represents an item so that similar items have similar vectors. They are the quiet foundation of semantic search, recommendations and retrieval-augmented generation.
From words to vectors
The simplest idea is one-hot encoding: give each word its own position in a huge vector filled with zeros and a single one. "Cat" and "dog" would be as different from each other as "cat" and "tax", because the vectors share nothing.
Embeddings fix this by placing items in a smaller continuous space, typically a few hundred to a few thousand dimensions, where distance reflects similarity. Words used in similar contexts, like "doctor" and "nurse", end up near each other, while unrelated words sit far apart.
A famous early demonstration showed that vector arithmetic could capture relationships, such as "king" minus "man" plus "woman" being close to "queen". Real behaviour is messier, but the example conveys the idea: directions in the space can encode meaning.
How embeddings are learned
Embedding models learn from the way items appear in data.
- Word2Vec and GloVe learned one vector per word by predicting words from their neighbours.
- Contextual models (transformers) produce a different vector for a word depending on the sentence, so "bank" in "river bank" differs from "bank account".
- Sentence and document embeddings summarise a whole passage in a single vector, trained so that texts with similar meaning are close.
- Multimodal embeddings map text and images into the same space, allowing search for photos with a text description.
The model is trained with examples of what should be close (a question and its answer, a caption and its image) and what should be far.
Measuring similarity
The most common measure is cosine similarity, the cosine of the angle between two vectors. It ranges from -1 to 1, and higher means more similar. Other options are dot product and Euclidean distance. For vectors normalised to length 1, cosine similarity and dot product give the same ranking.
Example: semantic search in Python
The sentence-transformers library provides ready-to-use embedding models.
from sentence_transformers import SentenceTransformer, util
model = SentenceTransformer("all-MiniLM-L6-v2")
documents = [
"How do I reset my account password?",
"Our shop opens at 9 am on weekdays.",
"Steps to change your login credentials",
"Delivery is free for orders above 2000 rupees.",
]
query = "I forgot my password"
doc_vectors = model.encode(documents, convert_to_tensor=True)
query_vector = model.encode(query, convert_to_tensor=True)
scores = util.cos_sim(query_vector, doc_vectors)[0]
for score, doc in sorted(zip(scores.tolist(), documents), reverse=True):
print(f"{score:.2f} {doc}")Even though the query shares few words with "Steps to change your login credentials", the embedding model rates it as highly relevant. That is the advantage over plain keyword search.
Where embeddings are used
- Semantic search: find documents by meaning, not exact words.
- Recommendations: items close to what you liked.
- Clustering and topic discovery: group similar texts.
- Duplicate detection: spot repeated questions or products.
- Classification: use embeddings as inputs to a simple classifier.
- RAG systems: retrieve relevant passages to give a language model.
- Anomaly detection: find items far from everything else.
Practical tips
- Use the same embedding model for documents and queries; vectors from different models are not comparable.
- Split long documents into chunks of a few hundred words so that each vector represents a focused idea.
- Store vectors in a vector database or index when you have many items, to search quickly.
- Evaluate on your own data. Try real queries and check whether the top results make sense.
- Mind the language. Choose a multilingual model if you have Nepali or mixed-language content, and test it.
- Re-embed when the model changes, since old vectors will not match new ones.
Limits and cautions
- Embeddings compress meaning, so nuance can be lost.
- They inherit biases from training data. Similarity may reflect stereotypes.
- Negation and fine details can be missed, so "the room is clean" and "the room is not clean" may look similar.
- Embeddings of sensitive text can sometimes leak information, so protect them like the original data.
Frequently Asked Questions
How many dimensions should an embedding have?
It depends on the model. Common sizes range from 384 to several thousand. More dimensions can hold more detail but cost more to store and search.
Are embeddings the same as tokens?
No. Tokens are pieces of text. Embeddings are numeric vectors that represent tokens, sentences or whole documents.
Can I use embeddings without a vector database?
Yes, for small collections you can keep vectors in memory or a file and compare directly. Databases help when you have many thousands or millions of items.
Do embeddings work for images and audio?
Yes. Many models produce embeddings for images, audio and video so you can search and compare them just like text.
Conclusion
Embeddings convert items into vectors where closeness means similarity. They power semantic search, recommendations and retrieval for language models. Choose a suitable model, test with your own data and keep text and queries in the same embedding space.