Why generic test cases fall short for domain-specific RAG
We are building a RAG system for the IAEA’s marine radioactivity knowledge base. Its users range from domain scientists to the general public. We needed to test the system against all of them.
Most RAG evaluations use generic test cases or simple question templates, such as “What is the definition of X?” or “Summarize document Y.” These check basic functionality. They say little about how a system copes with the questions that real users of a specialized domain ask.
Consider two users of our system. A nuclear safety regulator asks “How do current Cs-137 levels in Pacific fish compare to pre-Fukushima baseline measurements for food safety assessment?” A consumer asks “Is it safe to eat fish from the Pacific Ocean after Fukushima?” Both questions are legitimate. They call for different levels of technical detail, different terminology and different kinds of evidence.
Real user data is the best test set. Nothing beats the queries your users actually send in production. A new system has little of that data, and neither does a system opening up to a new audience. Until the data arrives, you have to anticipate how people will use what you built. Synthetic questions are not a one-off exercise either. We revise ours as we learn more about our users and find new failure modes.
A test set that misses this range optimizes for the average user. The edge cases it skips can erode user trust or, worse, lead someone to misread safety-critical information.
The method below draws on the evaluation approach taught in Parlance Labs’ course on LLM evaluations.
Dimensions
We did not generate questions at random or fill in generic templates. We borrowed an approach from qualitative research: identify the dimensions along which your users and their queries vary, then sample across them.
A dimension is one axis of variation in how users interact with the system. After analyzing our target audiences and use cases, we chose three dimensions for the marine radioactivity RAG:
1. Query Type/Purpose
- Scientific/Technical (analytical methods, specific measurements, research findings)
- Regulatory/Safety (food safety assessments, contamination levels, risk evaluation)
- Educational/General Public (layman explanations of mechanisms, basic concepts)
- Metadata/Discovery (listing studies, regions, dates, most cited papers)
2. Audience Expertise Level
- Domain Expert (scientists, researchers familiar with radioactivity terminology)
- Professional (regulators, policymakers who need accurate info but may need context)
- General Public (requires simplified explanations, analogies, avoiding jargon)
3. Query Specificity/Complexity
- Direct Factual (specific measurements, dates, locations)
- Analytical/Comparative (trends, correlations, multi-parameter analysis)
- Conceptual/Explanatory (mechanisms, processes, “how” and “why” questions)
Without explicit dimensions, a hand-written test set drifts towards the common cases and towards whatever feels typical to the person writing the questions. Sampling across combinations of dimension values forces the set to span expertise levels, query types and complexity.
From dimensions to questions
A tuple is one combination of values, one from each dimension. Each tuple stands for a distinct use case the system must handle well. For example:
- (Scientific/Technical, Domain Expert, Direct Factual)
- (Educational/General Public, General Public, Conceptual/Explanatory)
- (Regulatory/Safety, Professional, Analytical/Comparative)
We built 20 tuples. You can write them by hand, sample combinations programmatically, or ask an LLM to propose a diverse set. Whichever you choose, check that the tuples spread across the dimension space instead of clustering around common combinations.
For each tuple, we then generated realistic questions that match its combination of values. Different tuples produce very different questions:
Tuple: (Scientific/Technical, Domain Expert, Direct Factual)
“What are the specific gamma spectrometry detection limits for Cs-137 in seawater samples from the Mediterranean studies?”
Tuple: (Educational/General Public, General Public, Conceptual/Explanatory)
“How does radioactive material from nuclear accidents end up in the fish we eat?”
Tuple: (Regulatory/Safety, Professional, Analytical/Comparative)
“How do current Cs-137 levels in Pacific fish compare to pre-Fukushima baseline measurements for food safety assessment?”
Tuple: (Metadata/Discovery, General Public, Direct Factual)
“Which countries have studied radioactivity in their coastal waters?”
The expert’s question names a measurement technique, an isotope and a sampling region. The general-public question asks how something happens, in everyday words. The regulator’s question asks for a comparison against a baseline and says what the comparison is for. The discovery question asks for a list.
Expert review
We shared our 100 generated questions with the IAEA’s lead domain expert on marine radioactivity. We wanted to know whether the dimensions produced realistic questions that match what users need. The reply was better than we had hoped:
“These questions are great! Would very much like to learn more about how they were generated. I made some changes but really very minor as they mostly made sense and have relevance.”
The expert made only minor edits. Questions written for the general public, for professionals and for domain experts all passed. That suggests the dimensions capture real variation in how people ask about marine radioactivity, and that the questions reflect real information needs rather than plausible but empty queries. We now trust the set enough to evaluate the RAG system against it.
What’s next
Good questions are where evaluation starts. Next we need to find out how and why the system fails on them. We’ll follow a method adapted from Grounded Theory in qualitative research:
- Trace collection. We’ll run the system on all 100 questions and record a complete trace for each one: the retrieval, reasoning and response generation steps.
- Open coding. We’ll read the traces without predefined error categories and note anything that looks wrong, such as hallucinations, irrelevant retrievals, the wrong tone or missing context.
- Axial coding. We’ll group similar observations into failure modes that do not overlap. The result is a taxonomy of what goes wrong and why.
- Quantification and prioritization. We’ll count how often each failure mode occurs across the test set and fix issues in order of frequency and impact.
- Automated evaluation. We’ll train LLM-as-judge evaluators to detect these specific failure modes at scale. They will let us monitor the system continuously as we change it.