RAG for a Codebase: Prompt Patterns That Stay Grounded

Ask a coding agent how authentication works in your repo and, without the right context, it'll describe a plausible auth system that isn't yours. Confident, well-structured, wrong. RAG for a codebase is the fix: fetch the relevant code first, then reason over what's actually there, so the answer is grounded in files that exist instead of patterns the model saw in training.

The indexing and embedding side is heavily documented. The reusable part is thinner on the ground: a prompt that turns a question into a scoped fetch plan and then refuses to answer beyond what it retrieved.

Why agents hallucinate your codebase

A model operating on a large repo hits two walls. The context window can't hold the whole codebase, so something has to choose what to load. And the model's training data is full of how other codebases work, which it'll happily substitute when it doesn't have yours in front of it. The result is invented APIs, wrong internal conventions, and refactors that break silently because the agent guessed at a pattern.

Retrieval cuts both walls. Fetch the files a question actually needs, by relevance, and the model reasons over real code in a small, clean context. The win isn't just accuracy; it's that the answer becomes checkable, because you know which files fed it.

The two-step pattern: plan the fetch, then answer grounded

Codebase RAG works best as two prompted steps, and keeping them separate is what makes the answer trustworthy.

Plan the fetch. Read the question and decide what to retrieve: which symbols, files, or directories, and how much of each. A good fetch plan is specific. "The retryCharge function and its callers" beats "billing code." This step turns a fuzzy question into a precise retrieval the index can serve.

Answer grounded. With the retrieved context in hand, answer the question, but under a contract that forbids going beyond what was fetched. If the context doesn't contain the answer, the right output is "not found in the retrieved files," not a guess. And every claim cites the file it came from, so the answer is auditable.

That citation requirement is the quiet hero. An answer that says "auth is handled in middleware/auth.ts lines 40-72" can be verified in seconds. An answer with no citations is just a more confident hallucination.

What you can do with a codebase RAG prompt

• Turn a vague question into a precise, named fetch plan
• Pull only the symbols a question needs instead of whole files
• Ground every answer in retrieved code and cite the source files
• Return "not in the retrieved context" instead of inventing an API
• Keep long repos usable without overflowing the context window
• Audit an answer by checking the files it claims to have used

Anatomy of the grounded-answer contract

The reusable piece is a prompt that plans the retrieval and then answers under a strict grounding rule.

Variables → {{question}}, {{repo_index}}, {{max_chunks}}

Prompt → Step 1: plan the fetch. Name the symbols/files to retrieve
         and why, up to {{max_chunks}}.
         Step 2: answer {{question}} using ONLY the retrieved context.

Output contract (locked):
  fetch_plan:   [ { target, reason } ]
  answer:       grounded prose, no claims beyond context
  citations:    [ file:line ranges that support the answer ]
  not_found:    [ parts of the question the context didn't cover ]

The not_found block is what separates a grounded answer from a confident one. It forces the model to admit the gaps instead of papering over them. The citations block makes the whole thing checkable. Both are cheap to add and change the trust profile of every answer.

Where models differ

Grounding discipline varies sharply across models, which is exactly why you test it. Claude holds a "use only the retrieved context" instruction well and is comfortable returning not_found rather than guessing. GPT-4o is strong at the fetch-planning step but more prone to filling gaps from training data, so the grounding rule needs to be stated firmly and restated at the end of the prompt. Gemini cites reliably when the citation format is shown explicitly but will sometimes summarize instead of quoting line ranges. Pick your model and verify the grounding contract before you trust the answers.

An opinionated take: most codebase RAG setups over-fetch and under-cite. They pull twenty chunks to feel safe, which buries the relevant three in noise, and then skip citations because they slow the answer down. Flip it. Fetch fewer, cite always. A four-chunk answer with line-level citations is worth more than a twenty-chunk answer you can't verify. The Repo RAG Index Harness Playbook is built around this: a {{question}} variable produces a scoped fetch plan and a cited, grounded-answer contract, so answers stay checkable.

Step-by-step: grounding an agent in your repo

1. Build a retrievable index

Even a simple symbol-and-file search works to start. The fetch plan only needs something to fetch against.

2. Plan the fetch first

Run the planning step before answering. A named, reasoned fetch plan beats a raw similarity query every time.

3. Cap the chunks

Set {{max_chunks}} low. Over-fetching buries the signal. Start small and raise it only if answers are genuinely incomplete.

4. Enforce grounding

The answer step uses only the retrieved context. If it's not there, the output is not_found, not a guess.

5. Require citations

Every claim points to a file and line range. This is what makes the answer auditable and the agent trustworthy.

Variables you'll set

Where codebase RAG breaks

The failure that bites hardest is stale retrieval: the index lags the code, so the agent grounds its answer in a function that was renamed yesterday. The citation is real, the code isn't current, and the answer is wrong in a way that looks right. Re-index on a cadence that matches how fast the repo moves, and treat the index freshness as a first-class concern, not an afterthought. The second failure is the agent quietly ignoring the grounding rule under pressure; the not_found block is your tripwire for that, because an answer with no not_found and no citations on a hard question is the one to distrust. Pin your model version, since grounding discipline shifts across updates.

For mapping a repo so retrieval has something clean to target, see write an AGENTS.md file with AI, and for keeping the retrieved context from overflowing the window, see agent memory design patterns that hold up.

Getting started

1. Build a searchable index over the repo.
2. Run the fetch-planning step before answering.
3. Cap {{max_chunks}} low and raise only if needed.
4. Enforce the grounding rule in the answer step.
5. Require file-and-line citations on every claim.
6. Re-index on a cadence that matches the repo's pace.
7. Reach for the playbook below to get the contract pre-built.

To check that your retrieval is actually returning the right context, pair this with the RAG Quality Audit Playbook and the Repo Context Map Pack.