How do you stop an AI from making things up? RAG Basics

Imagine you have a friend who has read almost everything.

Every book, every website, every manual — they went through the lot, once, about two years ago. And then they threw all of it away. No notes, no highlighting, nothing kept. What is left is a feel for how things go together.

Ask them anything and they will answer immediately and confidently. Quite often they are right. Sometimes they are confidently, fluently wrong, and they cannot tell the difference — because they are not remembering, they are doing the thing they have got very good at: guessing what comes next.

That friend is a large language model. And RAG is the very simple idea of handing them the book before you ask the question.

That is the whole thing. Everything below is just how.

Why does it make things up at all?

It helps to know what the model is actually doing, because "it lies" is the wrong picture and leads people to the wrong fixes.

A language model predicts the next word. That is genuinely all it does. It has seen so much text that its predictions are astonishingly good — good enough to write code, explain photosynthesis, and argue with you about football.

But notice what is missing. There is no shelf inside it with the facts on. There is no moment where it checks anything. When you ask "who is the CTO of a small company in Accra?", nothing in the machine goes and looks. It produces the kind of sentence that usually follows that kind of question — a plausible name, a plausible title — because a plausible answer and a true answer look identical from the inside.

People call this hallucination, which makes it sound like a malfunction. It is closer to the opposite. It is the thing working exactly as built, applied to a question it has no business answering.

So you cannot fix it by asking nicely. "Please don't make things up" is advice given to something with no way to tell whether it is making things up.

You fix it by changing the question — from what do you remember? to what does this say?

Closed book, open book

You already know the fix, because school taught it to you.

A closed-book exam tests what is in your head. You will get some of it wrong, and you will be most confident about the bits you half-remember.

An open-book exam is a different activity. The knowledge is on the desk. The skill being tested is finding the right page and using it properly.

A plain chatbot is sitting a closed-book exam on your company, your product, your life — and it has never seen any of them.

RAG turns it into an open-book exam.

Retrieval — go and find the relevant pages.
Augmented — add them to the question.
Generation — now answer, using those.

That is the entire acronym. Retrieval-Augmented Generation. Go and look it up, then answer.

But how does it know which page?

This is the actual engineering, and it is the part worth understanding, because everything interesting happens here.

You have a pile of documents. Someone asks a question. You need to find the two or three bits that help — in a fraction of a second — before the model answers.

The obvious approach is to search for matching words. That works, sometimes, and it breaks in an obvious way. Somebody asks "how much does it cost?" and the document says "pricing starts at £40 per seat."

Not one word in common. A word search finds nothing. But any human can see they are the same question and answer.

So we need to search by meaning, not by spelling.

The map of meaning

Here is the idea that makes the whole field work. It sounds strange for about thirty seconds and then it seems obvious.

Imagine an enormous map. Not of places — of meanings.

Every sentence ever written gets a pin somewhere on this map. And the rule for where a pin goes is: things that mean similar things go near each other.

So "how much does it cost?" gets pinned right next to "pricing starts at £40 per seat", because those two things mean nearly the same thing. Meanwhile "the cat sat on the mat" is pinned miles away, over by the other cats.

Now finding the right page is not a language problem any more. It is a distance problem. Pin the question on the map, look around it, and grab whatever is closest.

Two details, and then you have it.

The map has a lot of directions. A paper map has two — north and east. This one has hundreds, often 1,536 of them. You cannot picture that and you do not need to. All it means is there are many different ways for two things to be similar, and the map has room for all of them at once. Computers are perfectly comfortable with a map like this; we are the ones with the imagination problem.

A pin is just a list of numbers. "Where on the map" is written down as a long list of coordinates. That list is called an embedding, and turning a piece of text into one is a job you hand to a small, cheap model built for exactly that. Feed it a sentence, get back its position on the map of meaning.

That is really all an embedding is. A pin. The address of a meaning.

Tearing the books into pages

One more practical thing.

You do not hand someone an entire 200-page handbook to answer one question. You find the page and photocopy that.

Same here. Before anything gets pinned to the map, documents are cut into pieces — a few paragraphs each. These are chunks, and the cutting is called chunking.

Two reasons, and both matter:

Precision. A whole book is about a hundred things, so its pin lands somewhere vague and unhelpful — the average of everything it says. A single passage about refunds is about refunds, and pins exactly there.

Room. The model can only read so much at once. Three relevant paragraphs will fit. Three books will not.

Chunking is one of those jobs that looks trivial and is not. Cut too small and a passage loses the context that made it meaningful. Cut too large and you are back to vague pins. Cut in the wrong place and you split a sentence down the middle, or divorce a heading from the thing it was heading.

The whole thing, start to finish

That is every piece. Here is how they run together.

Once, up front — you build the library:

Gather your documents.
Cut them into chunks.
Turn each chunk into an embedding — its pin on the map.
Store the pins in a database that is good at "what is near this?".

Then, every time someone asks a question:

Turn the question into an embedding too. Now it is a pin on the same map.
Ask the database for the nearest chunks. Say the closest eight.
Paste those chunks into the prompt, with the question, and roughly: "Answer using only what is below. If it isn't there, say you don't know."
The model answers.

          ONCE                              EVERY QUESTION

     your documents                          the question
           │                                       │
        cut into chunks                      make a pin for it
           │                                       │
        make a pin                          find the nearest chunks
           │                                       │
      store the pins  ─────────────────────►  paste them in
                                                   │
                                              model answers

The model is doing the same next-word prediction it always does. Nothing about it changed. What changed is what is sitting on the desk when it starts.

What this buys you

It is worth being clear about why people go to this trouble, because the benefits are bigger than "fewer wrong answers".

It can cite its sources. You know which chunks you handed over, so you can show them. The answer stops being an oracle and becomes something checkable.

It can be updated on a Tuesday afternoon. Your prices changed? Edit the document, re-chunk it, re-pin it. Done in seconds. The alternative — retraining a model — costs a fortune and takes weeks.

It can know things that are secret. Your internal handbook was never in the training data and never will be. It does not need to be. It just needs to be in your library.

It can decline. This is the underrated one. When the retrieval comes back empty, you have a signal you never had before: there is nothing here about this. Now "I don't know" is something the system can actually determine rather than something you asked it to feel.

The part the tutorials skip

I will be honest with you, because everything above is the happy version.

Every RAG tutorial ends at step 8. That is roughly the first afternoon of work, and it does work. Then you put it in front of real people with a real pile of documents and you find out what the tutorials left out.

A few, in plain terms:

One big document can eat everything. Somebody uploads a 200-page handbook and it becomes 340 of your 450 chunks. Now it has a passage near everything, so every answer comes out of that one handbook and the other forty documents may as well not exist.

Meaning-search misses names. The map is brilliant at "how much does it cost" ↔ "pricing starts at". It is oddly bad at exact things — a product code, a person, a repository. So most serious systems run a plain word search alongside the meaning search and combine them, because each one catches what the other drops.

Nearest is not the same as relevant. The database will always hand you the eight closest chunks, even when the closest thing in your entire library is nowhere near the question. Something has to notice that and say so, or you get a confident answer built out of eight irrelevant paragraphs — which is the original problem again, wearing a better coat.

"Yes" retrieves nothing. Real conversations are full of turns that are not questions. If your system's only move is retrieve, or refuse, then someone saying "yes please" gets told there's no information about that.

Retrieved does not mean allowed. The moment your library contains anything private, "what can be found" and "what this person may see" stop being the same question, and the gap between them is where the leaks live.

None of these are exotic. They all show up within about a fortnight of real use.

Words you will now hear

You can follow most RAG conversations with these.

Word	What it means
LLM	The model that writes the answer. Predicts the next word, very well.
Hallucination	A confident, fluent, wrong answer. Not lying — guessing.
Embedding	A pin on the map of meaning. A list of numbers standing for a meaning.
Vector	That list of numbers. "Vector" and "embedding" get used interchangeably.
Vector database	Where the pins are kept. Its one talent is finding what is near something.
Chunk	One cut-up piece of a document. The unit that gets pinned and retrieved.
Chunking	Deciding where to cut. Harder than it sounds.
Retrieval	Finding the relevant chunks for a question.
Context window	How much the model can read at once. The size of the desk.
Grounding	Making the answer come from the retrieved material rather than from memory.
Citation	Showing which chunk an answer came from.
Semantic search	Searching by meaning. The map.
Keyword search	Searching by words. Old-fashioned, still essential.
Hybrid search	Doing both and combining them. What most good systems do.
Re-ranking	A second, more careful pass over the results to reorder them.

So: is RAG the answer?

For "make the AI know about my stuff", yes, nearly always. It is cheaper than training, faster to update, and it is the only common approach where you can point at an answer and ask where did that come from?

But hold the idea loosely. RAG is not a product and it is not magic. It is a sensible habit — look it up before you answer — with a lot of unglamorous engineering underneath to make the looking-up good.

And that engineering is where all the difficulty actually lives. Not in the model. In the boring parts: how you cut the documents, how you rank the results, what happens when there is nothing to find, and who is allowed to see what.

I built one of these for my own site, and wrote up everything that went wrong — including a safeguard that silently did nothing for weeks. If the section above called the part the tutorials skip was the interesting bit, that is the long version.

Next: what actually breaks in a RAG system — the same subject, with the code and the scars.