Vibes-based testing is why so many retrieval systems regress silently. A small, honest eval harness — golden set, retrieval metrics, answer grading, and a regression gate — takes a couple of days and pays for itself immediately.
The most common reason a RAG system degrades after launch is that nobody can tell when it does. Someone swaps an embedding model, adjusts chunking, upgrades the generator — and the only feedback loop is a user eventually complaining.
Before I ship a retrieval system, I want four things in place. None of them are exotic.
1. A golden set that reflects real questions
150 to 300 question-answer pairs, written from actual user queries or support tickets, not invented by the team. Each entry records the question, an acceptable answer, and the document that should have been retrieved. Include the awkward ones: ambiguous phrasing, questions the corpus genuinely cannot answer, questions whose answer changed last quarter.
The unanswerable ones matter most. They are how you measure whether the system admits ignorance or invents something.
2. Retrieval metrics, separate from answer metrics
Grade retrieval on its own: recall@k and MRR against the expected document. If retrieval recall is 0.6, no prompt engineering will save the answers, and you will waste a week finding that out the slow way.
Keeping the two layers separate turns "the answers got worse" into either "retrieval got worse" or "generation got worse", which are different fixes.
3. Answer grading with a rubric
An LLM judge is fine here, provided it grades against a written rubric — factual correctness, grounding in retrieved context, appropriate refusal — and provided you spot-check 20 of its judgements by hand before trusting it. Score groundedness explicitly: an answer that is correct but not supported by the retrieved context is a lucky guess, and it will not stay lucky.
4. A regression gate in CI
Run the eval on every change to prompts, chunking, models, or retrieval parameters. Fail the build if any metric drops more than a defined threshold. This is what actually stops silent regressions, and it is the piece teams skip most often.
Keep it honest
Refresh the golden set quarterly with new real questions. Track the refusal rate alongside accuracy — a system that gets more accurate by refusing more is not necessarily better. And log production questions that scored poorly, because they are next quarter's golden set.
Two days of harness work buys you the ability to change your system with confidence for the rest of its life. There is no better return in this stack.