Vector Databases: What They Are and When You Need One

A vector database is a system built to store embeddings, the numeric representations of text, images and other data, and to find the ones most similar to a query vector very quickly. Vector databases became popular with the rise of semantic search and retrieval-augmented generation (RAG), but you may not need one for every project. This article explains what they do and how to decide.

The problem they solve

Suppose you have 500,000 document chunks, each represented by a vector with 768 numbers. A user asks a question, and you turn it into a vector too. To find the best matches, the straightforward method compares the query with every stored vector and keeps the highest scores. That is called exact or brute-force search. It is perfectly accurate, but it slows down as the collection grows.

Vector databases use special data structures to find the nearest neighbours without checking everything. This is called approximate nearest neighbour (ANN) search. It trades a tiny amount of accuracy for a very large gain in speed.

How they work

Storage

Each record usually has three parts: an ID, the vector, and metadata such as the title, source URL, language, date or category.

Indexes

Common ANN index types include:

HNSW (Hierarchical Navigable Small World): a graph of connections where the search hops from point to point toward the query. Fast and accurate, but uses more memory.
IVF (inverted file): divides the space into clusters and searches only the closest clusters.
Product quantisation: compresses vectors to save memory.

Querying

You supply a query vector and a number k. The database returns the k closest records, often with a filter such as "only documents in English from 2025".

Distance metrics

Cosine similarity, dot product and Euclidean (L2) distance are standard. Choose the one that matches how your embedding model was trained.

Options you will meet

Dedicated vector databases: Qdrant, Milvus, Weaviate, Pinecone and similar systems.
Extensions to existing databases: PostgreSQL with the pgvector extension, and vector features in search engines such as Elasticsearch and OpenSearch.
Libraries: FAISS and Annoy, which run in your own program rather than as a separate server.
Lightweight embedded stores: Chroma and similar tools, convenient for prototypes.

Do you actually need one?

Use this rough guide.

Situation	Suggested approach
Fewer than about 10,000 chunks, prototype	NumPy arrays or a simple in-memory index
Hundreds of thousands of items, one app	FAISS or `pgvector` inside your existing stack
Millions of items, many users, filters, high availability	A dedicated vector database
You already run PostgreSQL	Try `pgvector` first to keep one system

Adding another database brings operational work: backups, upgrades, security and cost. Start simple and move when measurements show you need more.

Example 1: a tiny in-memory search with FAISS

import faiss
import numpy as np

dim = 384
rng = np.random.default_rng(0)
vectors = rng.random((1000, dim), dtype="float32")   # pretend these are embeddings
faiss.normalize_L2(vectors)                          # cosine similarity via inner product

index = faiss.IndexFlatIP(dim)   # exact search
index.add(vectors)

query = vectors[42:43]
scores, ids = index.search(query, 3)
print(ids[0], scores[0].round(3))   # the first hit should be item 42 itself

Example 2: PostgreSQL with pgvector

If your project already uses PostgreSQL, a vector column keeps everything in one place:

CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE chunks (
    id SERIAL PRIMARY KEY,
    content TEXT NOT NULL,
    language TEXT,
    embedding VECTOR(384)
);

CREATE INDEX ON chunks USING hnsw (embedding vector_cosine_ops);

-- find the 5 closest chunks to a query embedding (passed as a parameter)
SELECT id, content
FROM chunks
WHERE language = 'en'
ORDER BY embedding <=> %s
LIMIT 5;

The <=> operator measures cosine distance, so the smallest values are the best matches.

Design tips

Choose chunk size carefully. Too large and vectors blur several topics; too small and context is lost. A few hundred words with slight overlap is a common start.
Store useful metadata and filter on it, for example by document, date or user permission.
Respect access control. Make sure a user cannot retrieve chunks from documents they should not see.
Measure recall. Compare ANN results with exact search on a sample to check accuracy.
Plan updates. Decide how to add, delete and re-embed documents when content changes.
Watch memory use. Large indexes may need compression or sharding.

Frequently Asked Questions

Is a vector database a replacement for SQL?

No. It complements regular databases. Many applications use both, or a relational database with a vector extension.

What is the difference between a vector index and a vector database?

An index is a data structure for fast similarity search. A database adds storage, filtering, persistence, scaling, security and an API around one or more indexes.

Do I need a GPU?

Not usually for searching. GPUs help when creating embeddings or searching extremely large collections.

How do I keep it up to date?

Re-embed changed documents, delete outdated records and version your embedding model so old and new vectors are not mixed.

Conclusion

Vector databases make similarity search fast at scale and are a key part of semantic search and RAG. For small projects, simple tools are enough. Choose a dedicated database only when data size, filtering needs or traffic justify the extra system.