> ## Documentation Index
> Fetch the complete documentation index at: https://docs.trymaitai.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Regression Monitoring

> Use Test Runs and comparisons to detect regressions over time

Regression monitoring is the practice of running Test Runs repeatedly as your configuration and models evolve, then using comparisons to catch quality regressions early.

## A practical regression workflow

* **Build one “golden” set per Intent Group**
  * Keep it focused: a smaller, high-signal set is easier to maintain and reason about.
  * Mark it as **Golden** when appropriate.

* **Create a Test Run for every meaningful change**
  * Examples: a new system prompt revision, a model swap, a temperature change, or after a fine-tune.
  * Use descriptions that make comparisons easy later (e.g. “prompt v4”, “temp 0.0”, “model X”, “post-finetune run”).

* **Compare runs instead of trusting a single metric**
  * Compare runs against the same set to see item-by-item score shifts, including runs against
    *different* consumers (e.g. a model baseline vs. an agent).
  * Compare a single item across runs to track how that specific scenario changed over time.

* **Focus attention where it matters**
  * Watch the run's **match rate** (or, for agent runs, **pass rate**) between runs.
  * Use the per-criterion breakdown to understand *which* [criterion](/test/test_sets/rubrics)
    degraded, not just that the score dropped, for numeric criteria, drill into the low buckets first.

## Common “gotchas” when monitoring regressions

* **Error % matters**: a run with a higher match rate but a higher error rate can still be a regression.
* **Coverage drift**: as your product evolves, add new real-world failure cases to the test set so regressions don’t hide in untested corners.

## Keeping Test Sets healthy

* **Continuously add new failure modes** discovered in Sessions/Requests via “Add to Test Set”.
* **Tag requests consistently** so you can quickly spot which categories are regressing.