A practical regression workflow
-
Build one “golden” set per Intent Group
- Keep it focused: a smaller, high-signal set is easier to maintain and reason about.
- Mark it as Golden when appropriate.
-
Create a Test Run for every meaningful change
- Examples: a new system prompt revision, a model swap, a temperature change, or after a fine-tune.
- Use descriptions that make comparisons easy later (e.g. “prompt v4”, “temp 0.0”, “model X”, “post-finetune run”).
-
Compare runs instead of trusting a single metric
- Compare runs against the same set to see item-by-item score shifts, including runs against different consumers (e.g. a model baseline vs. an agent).
- Compare a single item across runs to track how that specific scenario changed over time.
-
Focus attention where it matters
- Watch the run’s match rate (or, for agent runs, pass rate) between runs.
- Use the per-criterion breakdown to understand which criterion degraded, not just that the score dropped, for numeric criteria, drill into the low buckets first.
Common “gotchas” when monitoring regressions
- Error % matters: a run with a higher match rate but a higher error rate can still be a regression.
- Coverage drift: as your product evolves, add new real-world failure cases to the test set so regressions don’t hide in untested corners.
Keeping Test Sets healthy
- Continuously add new failure modes discovered in Sessions/Requests via “Add to Test Set”.
- Tag requests consistently so you can quickly spot which categories are regressing.