Testing RAG systems, agents, and the models that judge their answers.
01 · Context
Testing RAG systems, agents, and the models that judge their answers.
The problem
An answer can sound right without being supported by its sources. A model judging that answer can get it wrong, too. The evaluation needs its own tests.
My role
My independent project for exploring RAG evaluation, model judges, adversarial inputs, and agent reliability.
02 · Engineering decisions
How it works.
- 01Combine deterministic checks with model-based evaluation.
- 02Use replay data for repeatable checks without provider calls.
- 03Separate fast, judged, and live evaluation tiers.
- 04Compare a judge against a small labeled gold set and document adversarial findings.
- RAG · LangGraph
- Deterministic checks
- Judge evaluation
- Gold-set analysis
03 · Quality strategy
How I tested it.
The implementation includes scripted agent reliability tests for tool failures, loops, and state behavior, as well as replay-based evaluation and adversarial cases.
04 · Result & reflection
Results so far.
The project provides repeatable evaluation runs and documents where model judges fail. The findings include fixes for identified vulnerabilities and a judge-bias issue that remains unresolved.
Scope & limitations
The gold set is small. Calibration results depend on the chosen threshold and do not establish general safety or held-out accuracy.
What I learned
A judge’s final score doesn’t show where it disagrees with the labeled examples. Those disagreements and the chosen threshold need to be visible when interpreting results.
