Evaluate a RAG Pipeline With One Prompt That Scores Both Halves

When a RAG answer is wrong, the useful question isn't "how wrong." It's "which half broke." Retrieval can fetch the wrong chunks, or generation can ignore the right ones. A single quality score blends those two failures into a number that tells you nothing about what to fix. A real rag evaluation prompt scores the halves separately.

The approach here is one prompt, three axes. Given a question, the retrieved context, and the answer, it rates context relevance, faithfulness, and answer relevance on a tight 0-to-2 rubric, then writes one line naming which half failed and why. You can run it on ten triples by hand before you ever touch a framework.

That separation is the point. Retrieval failures and generation failures get fixed in completely different places, so they can't share a score.

Why most RAG evaluation guides leave you with no prompt

The top results teach the metrics and then bury them in framework code. Confident AI's RAG metrics writeup is a clear catalog but couples every metric to DeepEval and never ships one unified scoring prompt. The Towards Data Science retrieval series is strong on precision@k math but assumes labeled relevance judgments you don't have yet. And Deepchecks' pipeline overview surveys frameworks without handing you something to paste.

So the metrics are well-documented and the prompt is missing. That's the gap. A rubric you can run on a query-context-answer triple, today, in any model, beats a metrics catalog you still have to wire into code.

Here's the stance: faithfulness and context relevance are not the same metric, and treating them together is the most common RAG-eval mistake. An answer can be perfectly faithful to context that never contained the answer. That's a retrieval failure wearing a generation costume, and a blended score hides it completely.

What you can do with this prompt

• Score retrieval and generation on separate axes for the same query.
• Rate context relevance: did the fetched chunks contain the answer at all?
• Rate faithfulness: did the answer stick to the context without inventing?
• Rate answer relevance: did it actually address the question asked?
• Get a one-line fix note pointing at retrieval, chunking, or the generation prompt.
• Run it by hand on a sample before adopting any scoring framework.

Anatomy of the RAG evaluation prompt

The prompt takes one triple, the question, the retrieved context, and the generated answer, and emits a three-axis score with a diagnosis.

Variables:
  {{question}}          – the user query
  {{retrieved_context}} – the chunks the retriever returned
  {{answer}}            – the generated answer

Prompt:
  Role: RAG evaluator scoring two halves of one pipeline.
  Score each axis 0 / 1 / 2 with a one-line reason:
    - context_relevance: does the context contain the answer?
    - faithfulness: is the answer grounded in that context?
    - answer_relevance: does it address the question?

Output contract (restate on the final line):
  - three scores (0-2) with reasons
  - failing_half: retrieval | generation | both | none
  - fix_note: one concrete next step

The failing_half field is what makes this diagnostic rather than just descriptive. A score tells you the pipeline is bad; this tells you whether to fix the retriever or the prompt.

Step-by-step usage

1. Capture a real triple

Pull a question your users actually asked, the exact chunks your retriever returned for it, and the answer the model produced. Synthetic triples test a pipeline you don't run. Real ones test yours.

2. Score retrieval before you look at the answer

Read {{retrieved_context}} against the question first. If the answer isn't in there, context relevance is 0 and you've found a retrieval problem. No generation fix will help, so don't waste time on the answer yet.

3. Then score faithfulness and answer relevance

With good context, check whether the answer stuck to it (faithfulness) and whether it answered the question (answer relevance). A faithful answer to an off-topic chunk still scores low on relevance. Both can fail independently.

4. Act on the failing half

A context-relevance failure points at retrieval: chunking, embedding, top-k, or query rewriting. A faithfulness failure points at the generation prompt and its grounding instruction. Fix the half the prompt names, not the one that's easier to touch.

5. Build a scored set, then automate

Once the rubric agrees with your judgment on twenty triples, you've validated it. That's the moment to wire it into RAGAS, DeepEval, or a script, because now you trust what it measures.

Prompt-craft patterns for RAG scoring

Score retrieval against the question, not the answer. The trap is judging context by whether it matches the answer the model gave. Judge it against what the question needed. Otherwise a confidently wrong answer drags the context score down with it, and you misdiagnose a generation bug as a retrieval one.

context_relevance scores the context vs the QUESTION.
Ignore the answer entirely for this axis. The question
asks for X; does the context contain X? Yes=2, partial=1, no=0.

Keep the rubric to three levels. Zero, one, two. A five-point scale invites the model to park everything at three, and you lose the signal. Binary is too coarse for partial-credit cases like "the context had half the answer." Three levels is the sweet spot.

Account for model leniency. GPT-4o tends to score faithfulness generously, rewarding fluent answers that drift slightly from context; Claude is stricter on grounding and will dock an answer that adds a single unsupported clause. When two models disagree on faithfulness, the stricter read is usually the correct one for production. Pin the model version you score with, because a model update can shift the whole scale.

Variables you'll set

Getting started

1. Capture a real question, its retrieved context, and the answer.
2. Score context relevance against the question first.
3. Score faithfulness and answer relevance only if the context holds up.
4. Read the failing_half and fix that half, not the convenient one.
5. Validate the rubric against your judgment on twenty triples.
6. Wire the trusted rubric into a framework for scale.
7. Diagnose systemic issues. The RAG Quality Audit playbook runs this scoring across a set and traces patterns back to chunking and retrieval config.

Scoring tells you the pipeline is broken; fixing retrieval is its own job. When the failure is consistently context relevance, the index itself usually needs work, which a Repo RAG Index Harness playbook addresses at the source.

A RAG eval connects to two neighbors: a detect-hallucinations prompt for checking faithfulness on a single answer, and building an eval dataset from logs so your RAG scoring runs on questions users actually asked.