Teams building their first evaluation set tend to worry about volume. The worry is misplaced. Beyond a few hundred items, additional questions mostly add maintenance cost, because someone has to keep them labelled as the corpus moves.
What matters is whether the set covers the ways the system can fail.
Stratify by failure mode, not by topic
Group by document family first, because families fail independently. A pipeline that handles native-text policy documents well and scanned medical reports badly will show an acceptable aggregate score and an unacceptable experience for one team.
Then group by question type, because those fail independently too.
- Lookup: a single fact in a single document
- Aggregation: a fact assembled across several documents
- Temporal: which version applied on a given date
- Negative: questions the corpus genuinely cannot answer, where the correct response is abstention
- Adversarial: content designed to elicit an unsupported claim, including instruction-like text inside documents
Report per stratum, always
A single headline number is the enemy of a useful evaluation. Report per family and per question type, every time, and set CI thresholds per stratum rather than on the mean. A four-point drop confined to one family will not move an aggregate enough to fail a build, and that is precisely the regression you most need to catch.
Grow it from production
The set should not be frozen at handover. Every correction a reviewer makes is a labelled example that arrived free, and the questions users actually ask drift away from the ones you imagined during the build.
Wire the feedback control into the interface, route corrections into a labelling queue, and add them in batches with the same adjudication standard as the original set. A set that grows every month is the clearest available signal that the system is being taken seriously by the people who own it.
