RAG: Giving the AI Your Own Facts

Look it up first, then answer

RAG means the AI searches your documents before it answers. It is how you stop it guessing about your business, your prices and your clients.

The problem

The AI knows a lot about the world. It knows nothing about your world. It has never seen your rate card, your client list, your service agreement or last month's numbers.

Ask it anyway and it will still answer, because it always answers. That answer is a guess wearing a confident voice.

What RAG actually does

A question going into a search step that pulls three matching chunks out of a document store, then the question plus those chunks going into the AI, which answers with sources — Search first, answer second. The AI only sees the pieces that matched.Open full size

You ask a question.
The system searches your documents for the parts that look related.
It takes the best few pieces, maybe three or five.
It puts those pieces plus your question in front of the AI.
The AI answers using them, and can point at which file it used.

That is the whole idea. The clever part is step 2: the search finds text by meaning, so a question about "how much for a website" can still find a document that says "landing page pricing".

But how can a search understand meaning?

This is the part that sounds like magic, and it is not. It comes down to one idea: turn text into numbers, in a way where similar meanings end up with similar numbers.

A small program called an embedding model reads a piece of text and writes out a long list of numbers. Not a summary, not keywords. Just numbers. The same text always gives the same numbers, and text that means something similar gives numbers that are close.

Three panels: four pieces of text, then each one turned into a list of numbers, then those numbers plotted as dots in a space where money-related text clusters together and food-related text clusters somewhere else — Two things about money land near each other. Two things about food land somewhere else.Open full size

Think of those numbers as coordinates. Two numbers would put something on a map. Three would put it in a room. These lists are hundreds of numbers long, so the "space" is impossible to picture, but the rule is the same one you already understand: things near each other are alike.

A vector database is just the storage for all those number lists, built to answer one question very fast: given this point, what are the nearest ones? "Vector" is only a fancy word for that list of numbers.

Why this beats searching for words

Your question is turned into numbers by the same model.
That drops your question at some point in the same space.
The database finds the nearest pieces of text.
Those get handed to the AI as the answer material.

So asking "how much for a website" can pull up a file that only ever says "landing page pricing". Not one word matches. They just sit in the same neighbourhood. An old-fashioned keyword search would have found nothing at all.

Why not just paste everything in?

Because there is a limit to how much the AI can read at once, and because quality drops as you fill it up. Pasting a 200-page handbook to ask one question buries the useful paragraph in noise, and you pay for all of it on every message.

RAG sends the three paragraphs that matter instead of the two hundred pages that do not.

Where you already use this

Project knowledge in Claude: you upload your documents once and every chat in that project can use them.
Connectors: the AI reaches into your Drive or your notes and pulls only what it needs.
A support bot that answers from your own help articles instead of making things up.

A freelancer example

You put your rate card, your service agreement and your last ten proposals into a project. Now "write a proposal for a bakery that wants three posts a week" comes back priced from your real rate card, in your real format, with your real terms. You did not paste any of it. It was retrieved.

Where RAG goes wrong

One document cut three ways: into one oversized piece, into three right-sized pieces, and into six fragments, with what each does to the answer — The search hands back pieces, not files. How you cut them is most of the quality.Open full size

Problem	What you see	The fix
Chunks too big	The right file is found but the answer misses the detail	Cut documents into smaller pieces
Chunks too small	Answers lose the context around the fact	Give each piece enough surrounding text
Bringing back too few	It misses a document that mattered	Raise the number of results
Bringing back too many	Noise, slower, worse answers	Lower it again
Old documents never removed	Confidently quotes last year's price	Delete or replace what is out of date

Always ask for the source

A good RAG setup can tell you which file each part of the answer came from. Use that. An answer with a source you can click is checkable. An answer without one is just a claim.

More words

Prompt Engineering
Writing the instruction so the answer comes back right
Context Engineering
Choosing what the AI gets to see while it answers
Loop Engineering
Teaching an AI to check on its own, no prompt needed

See all 11 explanations

Reading time: 10 min
Group: Running them for real

Words inside this word

RAG: Retrieval Augmented Generation. Fancy words for: search first, then answer using what you found.
Vector store: A special database that finds text by meaning, not by exact words.
Chunk: A small piece of a document. Long files are cut into chunks so the search can return only the useful part.
Top-k: How many chunks the search brings back. k is just a number, like 5.
Embedding model: A small program that turns a piece of text into a long list of numbers.
Vector: Just the name for that list of numbers.
Embedding: The list of numbers for one piece of text. Similar meanings get similar numbers.