Embeddings & Vector Search

Embeddings turn text into vectors whose distances track meaning. Similarity math finds the nearest ones. A vector database makes that search fast at scale. RAG puts the results into a prompt so a language model can answer from your data instead of from its memory.

That is the whole process. Every section below expands one of those four sentences, in that order.

The Core Idea

An embedding model turns a piece of text into a fixed-length list of numbers called a vector. Text with similar meaning lands at nearby points in that space.

"dog"    ->  [0.21, 0.78, 0.11, ...]
"puppy"  ->  [0.28, 0.71, 0.19, ...]   near "dog"
"car"    ->  [0.90, 0.09, 0.05, ...]   far from "dog"

A program has no way to tell that "dog" and "puppy" are related words. It can measure the distance between two lists of numbers. Embeddings convert the first problem into the second, which is what makes these possible:

• finding documents that answer a question without sharing its keywords
• recommending items similar to one someone already liked
• grouping related content without predefined categories
• pulling relevant context into an LLM prompt (RAG, later in this pack)

Getting a Vector

You use a pre-trained model rather than training one. Every major AI provider hosts embedding models behind an API, and open models you can run yourself are on Hugging Face. Vector lengths across them run from a few hundred numbers to a few thousand — bigger is not automatically better for your data.

The examples in this pack use OpenAI's API and the open all-MiniLM-L6-v2 model, because they are widely available and the call shapes are typical. Whatever you use, the request looks about like this:

from openai import OpenAI

client = OpenAI()  # reads OPENAI_API_KEY from the environment

response = client.embeddings.create(
    model="text-embedding-3-small",
    input="The quick brown fox",
)

vector = response.data[0].embedding
len(vector)  # a fixed number, set by the model

Three Things to Know About the Vectors

• The length is fixed by the model, not the input. A one-word input and a full page both come back as the same number of values. Some models let you request a shorter vector — a smaller, faster index in exchange for some accuracy.
• The same text always produces the same vector — for the same model. Vectors from two different models are not comparable at all. Switching models means re-embedding everything you have stored.
• Some models return normalized vectors (length exactly 1). Check your provider's docs rather than assuming — it changes which similarity math is worth using, which is the next section.

External Resources

The Question

You have two vectors. How close are they?

Cosine Similarity

Cosine similarity measures the angle between two vectors and ignores their length. It runs from -1 (opposite) through 0 (unrelated) to 1 (identical direction).

import numpy as np

def cosine_similarity(a, b):
    a, b = np.asarray(a), np.asarray(b)
    return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)))

dog   = [0.21, 0.78, 0.11]
puppy = [0.28, 0.71, 0.19]
car   = [0.90, 0.09, 0.05]

cosine_similarity(dog, puppy)  # 0.99
cosine_similarity(dog, car)    # 0.36

Real vectors have hundreds or thousands of components instead of three, but the arithmetic is exactly this.

The Other Metrics

When vectors are normalized to length 1, these collapse into each other: dot product equals cosine similarity, and Euclidean distance ranks results identically. That is why vector databases let you pick a metric per index and why the choice usually does not change your results — only your compute cost.

Scores Are Relative

A cosine similarity of 0.82 does not mean "82% relevant." Score distributions differ by model and by corpus, so a cutoff that filters noise on one dataset admits garbage on another. Rank by score, then decide a threshold by looking at your own labelled examples.

Searching a Small Corpus

With a few thousand documents you can compare against everything:

query_vector = embed("best pizza in NYC")

ranked = sorted(
    ((doc, cosine_similarity(query_vector, vec)) for doc, vec in stored),
    key=lambda pair: pair[1],
    reverse=True,
)
top_5 = ranked[:5]

That loop is linear in the number of documents. The next section is about what to do when it stops being fast enough.

External Resources

The Scale Problem

Comparing a query against 1,000 stored vectors is a loop. Comparing it against 10 million is a data-structures problem — and that is what a vector database solves.

What They Actually Do

1. Store vectors alongside metadata — the original text, an ID, fields you want to filter on.
2. Build an approximate-nearest-neighbor index so a query does not touch every vector.
3. Return the top matches with scores, usually in milliseconds.

The word to notice is approximate. These indexes trade a small amount of recall for a very large speedup, which is almost always the right trade at scale.

The Options

If your data already lives in Postgres, start with pgvector. Adding a second datastore is a real operational cost, and pgvector handles millions of vectors.

The Usage Pattern

Every one of these has the same three-step shape. Pinecone, for example:

from pinecone import Pinecone

pc = Pinecone(api_key="your-key")
index = pc.Index("my-embeddings")

# 1. Write vectors, with metadata you may want to filter or display
index.upsert(
    vectors=[
        {"id": "doc-1", "values": pizza_vector, "metadata": {"text": "Pizza recipe"}},
        {"id": "doc-2", "values": pasta_vector, "metadata": {"text": "Pasta recipe"}},
    ],
)

# 2. Query with a vector you produced the same way
results = index.query(
    vector=embed("italian food"),
    top_k=5,
    include_metadata=True,
)

# 3. Read the matches
for match in results.matches:
    print(match.score, match.metadata["text"])

Chroma, Qdrant, and pgvector differ in the method names, not the shape: write vectors with metadata, query with a vector, read back scored matches.

Metadata is worth populating carefully. Besides giving you the original text to display, it lets you filter before the vector search runs — restricting to one customer, one language, or one document type narrows the candidate set and speeds up the query.

When You Don't Need One

• Under roughly 10,000 embeddings — NumPy and a sorted list are fine, and simpler to debug.
• Batch jobs where latency does not matter — brute force finishes while you get coffee.
• Data that changes on every run, so an index would be rebuilt each time anyway.

External Resources

Keyword vs Semantic

Keyword search matches the words the user typed. A search for "best budget laptop" finds documents containing budget and laptop, and misses "affordable notebook computer" entirely.

Semantic search matches the meaning. "Affordable notebook computer" embeds near "best budget laptop", so it comes back — no shared words required.

The flip side: semantic search will happily return something thematically close when the user wanted an exact string, like a product SKU or an error code.

The Four Steps

1. Index time — embed every document and store the vectors.
2. Query time — embed the incoming query with the same model.
3. Search — find the nearest stored vectors.
4. Rank — return the top K, optionally re-scored by something else.

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("all-MiniLM-L6-v2")

documents = [
    "Python is a programming language",
    "Machine learning uses data",
    "JavaScript runs in browsers",
]

doc_embeddings = model.encode_document(documents)
query_embedding = model.encode_query("coding languages")

scores = model.similarity(query_embedding, doc_embeddings)[0].tolist()

for doc, score in sorted(zip(documents, scores), key=lambda pair: pair[1], reverse=True):
    print(f"{score:.2f}  {doc}")

Queries and Documents Are Not the Same Thing

Note encode_query and encode_document above — they are not stylistic. A short question and a long passage that answers it look different to an embedding model even when they mean the same thing, so retrieval models are trained to embed the two roles differently.

Libraries expose this differently — separate encode methods here, a parameter on the embed call elsewhere — but the idea is the same wherever you meet it. Check whether the model you picked draws the distinction, because ignoring it is a quiet accuracy loss rather than an error.

Hybrid Search

Production search usually runs both kinds and merges the results:

• Keyword catches exact matches — identifiers, names, error strings, rare jargon.
• Semantic catches paraphrases and related concepts.
• Merge and re-rank the two result sets into one list.

Weaviate, Qdrant, and Pinecone all offer hybrid search directly. A reranker model — a second, slower model that scores each candidate against the query — is the usual next accuracy upgrade once hybrid search is in place.

External Resources

What RAG Fixes

An LLM only knows what was in its training data. It has never seen your company handbook, last week's incident report, or the customer's account history. RAG (Retrieval Augmented Generation) closes that gap by retrieving relevant text and putting it in the prompt before the model answers.

The Pipeline

question -> embed -> vector search -> top chunks -> prompt with context -> answer

Every piece of that comes from the previous sections. RAG is semantic search with a generation step bolted on the end.

A Worked Example

The user asks: What's our return policy?

1. Embed the question with the same model used to index the handbook.
2. Search the vector index, take the top 3 chunks — "Returns accepted within 30 days", "Original receipt required", "Refunds processed in 5-7 business days".
3. Build a prompt containing those chunks plus the question.
4. Send it to the model, which answers from the supplied text.

The prompt you assemble in step 3 looks like this:

Answer the question using only the context below. If the context
does not cover it, say so.

Context:
Returns accepted within 30 days
Original receipt required
Refunds processed in 5-7 business days

Question: What's our return policy?

The Code

from openai import OpenAI
from pinecone import Pinecone

client = OpenAI()
index = Pinecone(api_key="your-key").Index("handbook")

question = "What's our return policy?"

query_vector = client.embeddings.create(
    model="text-embedding-3-small",
    input=question,
).data[0].embedding

results = index.query(vector=query_vector, top_k=3, include_metadata=True)
context = "\n\n".join(match.metadata["text"] for match in results.matches)

prompt = (
    "Answer the question using only the context below. "
    "If the context does not cover it, say so.\n\n"
    f"Context:\n{context}\n\nQuestion: {question}"
)

Then send prompt to whichever chat model you use — that call is one line of an SDK and is the same call you would make without retrieval. Everything above it is the part RAG adds.

Where RAG Goes Wrong

• Bad chunking. Split a document mid-argument and neither half retrieves well. See the next section.
• Wrong chunks retrieved. The most common cause of a bad answer is a good model reading the wrong text. Log what was retrieved, not just what was said.
• Too much context. Stuffing 50 chunks in costs tokens and buries the relevant one.
• The model ignores the context anyway. Instruct it to answer only from the supplied text and to say when the text does not cover the question — and check that it complies.

Related Concepts

External Resources

Why Split At All

A single vector for a fifty-page PDF represents the average of everything in it, which is to say nothing in particular. Retrieval happens over the pieces, so documents get split before they are embedded — and how you split them decides what can be found.

Fixed-Size Chunks

The simple approach: a set number of words per chunk, with a little overlap so a sentence that straddles a boundary still appears whole somewhere.

def chunk_text(text, chunk_size=500, overlap=50):
    words = text.split()
    step = chunk_size - overlap
    return [
        " ".join(words[i:i + chunk_size])
        for i in range(0, len(words), step)
    ]

It ignores meaning entirely, which is the drawback — a definition and the sentence that qualifies it can land in different chunks.

Structure-Aware Chunks

Splitting on the document's own boundaries — headings, sections, paragraphs — usually retrieves better, because each chunk is then about one thing. Keep the source title and document ID in each chunk's metadata so a retrieved chunk can be traced back and shown with its source.

The Tradeoff

There is no correct answer here, only an answer for your documents and your questions. It is worth testing directly — chunking changes retrieval quality more than swapping embedding models does, and it is far cheaper to change.

What Embedding Costs

Hosted embedding models are priced per input token, and in a retrieval system they are almost never the expensive part — generation costs more. Two habits keep it that way:

• Embed in batches. Every API takes a list of inputs. One request with a hundred texts beats a hundred requests.
• Store the vector; never recompute it. Text you have already embedded should be embedded once, with the vector saved alongside the record.

Running an open model yourself trades the per-call price for a machine to run it on. all-MiniLM-L6-v2 is 22.7M parameters — about 90 MB — and works on a CPU, though bulk indexing is much faster with a GPU.

External Resources

Which Piece Do I Reach For?

Choosing an Embedding Model

• Starting out: any provider's general-purpose model. They are cheap, and the differences between them will not be your bottleneck.
• Data cannot leave your infrastructure: an open model you host, such as all-MiniLM-L6-v2.
• Specialized content — code, legal, finance, medicine: domain-tuned models exist and usually beat general ones inside their domain.

Public benchmarks like MTEB give you a shortlist, not an answer. Write down twenty real questions and the document that should come back for each, then rank your candidates on that; the ordering often differs from the leaderboard. That same set is what you use to compare chunk sizes and similarity metrics, so it is worth building early.

External Resources