Watching live production quality (Sentinels and Monitors) lives under
Observe. Test is for curated, offline evaluation; Observe is for real traffic.
The building blocks
Test Sets
Type-less sets of
input + expected output items, built from requests, workflow runs, agent
tasks, manual entries, or a file.Test Runs
Execute a set against a model, workflow, or agent, then read the results and track regressions.
Test Criteria
Reusable checks (deterministic, LLM-judge, or agent-trajectory) that score your Test Runs.
How they fit together
- A Test Set is a type-less batch of items.
- A Test Run executes that set against a chosen consumer (model, workflow, or agent).
- Evaluation Criteria score each run, so you can compare baseline vs. candidate and catch regressions.
Where to go next
- Build your first regression suite: Test Set Creation
- Fix a failure and prevent it from returning: Quality Loop
- Watch live quality instead: Sentinels and Monitors