How to Test Your RAG Pipeline Before It Breaks Your Agent

Bad retrieval is one of the most common root causes of agent failures, and one of the hardest to catch. When your retrieval step pulls the wrong context, the agent does not error out or ask for help. It confidently generates a response based on whatever it received. The answer looks fluent, well-structured, and authoritative. It is also wrong.

The reason this failure mode is so dangerous is that it is invisible at the agent level. By the time you are looking at the final output, the retrieval decision has already been made and buried. You see a confidently wrong answer, but you have no obvious signal that the root cause was a chunking strategy that split a key passage in half, an embedding model that missed the semantic match, or an index returning loosely related documents. The agent worked exactly as designed. It reasoned over the context it was handed. The context was the problem.

This post explains how to test your retrieval system directly, before it reaches your users, using RAG experiments.

What a RAG Experiment Actually Is

A RAG experiment tests that your retrieval system returns the right context for a set of known queries. Instead of running the full agent and trying to reverse-engineer why an answer went sideways, you isolate the retrieval step and measure it on its own.

The setup has three parts:

• A dataset of known queries paired with the context that should be retrieved for each. This expected context is your ground truth.
• A variable you are testing, such as a new embedding model or a different chunking strategy.
• Evals that score whether the retrieved context matches what you expected.

Isolation is the entire point. When retrieval is tangled up inside an end-to-end agent run, a bad result could come from the prompt, the model, the tool logic, or the retrieval step. Pulling retrieval out lets you measure whether changes to your chunking strategy, embedding model, or index configuration actually improve the context the agent has to work with, without the noise of the rest of the system. This is the same logic behind treating agent improvement as a structured, experimental process rather than guesswork: isolate the change, measure its impact, and only promote what works.

What to Vary in a RAG Experiment

A RAG experiment holds the dataset and evals fixed so you can isolate the impact of a single change. The variable is whatever you are testing in the retrieval pipeline. The most common knobs:

• Chunking strategy. How you split source documents determines what can be retrieved at all. Chunks that are too large bury the relevant passage in noise. Chunks that are too small fracture context across boundaries. Test different chunk sizes and overlap settings against your dataset.
• Embedding model. The embedding model decides what counts as semantically similar. A model that performs well on general text may miss domain-specific matches in legal, medical, or technical content. Swap the model and measure whether retrieval quality holds.
• Index configuration. Index parameters, distance metrics, and filtering logic all shape what comes back. Small configuration changes can meaningfully shift which documents surface for a given query.
• Hybrid vs. near-text search. Pure semantic (near-text) search captures meaning but can miss exact terms, identifiers, or rare keywords. Hybrid search blends semantic and keyword matching. Comparing the two on the same dataset tells you which retrieves better context for your queries.

Change one variable at a time. If you swap the embedding model and adjust chunking in the same run, you lose the ability to attribute the result to either change.

How to Measure Improvement

The question every RAG experiment answers is simple: did the right context get retrieved for these known queries? Measuring it well comes down to choosing the right kind of check.

Because RAG experiments have a ground truth (the expected context for each query), they use supervised evals. The best practices for evals apply here:

• Binary, not scored on a range. Score each result pass or fail rather than on a 1-to-10 scale. Ranges push the judgment burden onto a human deciding what threshold matters, and LLM scorers are inconsistent across runs. When an eval fails, it should mean something requires attention.
• Specific, not generic. Each eval should target a concrete check with a clear definition of failure. "Did the retrieved documents contain the expected passage?" is specific enough to judge reliably. "Was the retrieval good?" is not.
• Explanation-backed. Require each eval to return an explanation alongside the pass/fail decision so you can spot patterns across failures instead of re-reading every result from scratch.

Match the check to what you are measuring. LLM-based evals are strong at generalizing over content, like judging whether retrieved context is relevant to a query. They are weaker at precise quantitative assessment. If you need to measure context recall or retrieval precision, where the answer is a calculation over which expected documents were and were not returned, a deterministic programmatic check is more reliable and cheaper than an LLM. Do not force an LLM eval where a simple function does the job better.

Start Narrow, Then Validate End to End

RAG experiments are fast and cheap because they skip the full agent. That makes them the right place to iterate quickly and find the retrieval configuration that works. But a retrieval improvement is only real if it holds up when the full system runs together.

The guidance is to start narrow. Use prompt and RAG experiments to iterate on individual components, then validate with an agent experiment that runs the agent end to end on known inputs before promoting changes to production. Better retrieval in isolation does not always translate to better agent behavior once the model, prompt, and tools are back in the loop. The agent-level run is where you confirm it does.

Once a change passes, the dataset that drove the improvement becomes a regression suite. Re-run your RAG experiments before every deployment so a future change to chunking, embeddings, or index configuration cannot silently undo the fix. Over time, that dataset grows into accumulated knowledge of how your retrieval system should behave.

TLDR

• Bad retrieval produces confidently wrong answers, and the failure is invisible at the agent level because the agent reasons over whatever context it received.
• A RAG experiment isolates the retrieval step and tests it against known queries with expected context as ground truth.
• Vary one thing at a time: chunking strategy, embedding model, index configuration, or hybrid vs. near-text search. Keep the dataset and evals fixed.
• Measure with supervised evals that are binary, specific, and explanation-backed. Use programmatic checks for quantitative measures like context recall and precision.
• Start narrow with RAG experiments, validate end to end with an agent experiment, then keep the dataset as a regression suite.

Bad retrieval should never be something your users discover for you. If you want to see how RAG experiments fit into the full agent development lifecycle, book a demo with an AI expert.

‍