Skip to main content
A Test Run executes every item in a test set against a consumer you choose, then scores the results. Because the set is type-less, the same set can be run against a model one time and a workflow or agent the next, which makes runs directly comparable through the set as the shared axis.

​
Create a Test Run

  1. Open a test set.
  2. Click New Run.
The wizard walks you through:
  • 1) Consumer: choose what runs each item:
    • Request: run each item through a single model with generation params you tune (temperature, top-p, max tokens, …).
    • Workflow: send each item through a workflow’s full pipeline.
    • Agent: replay each item as an agent session and score the final response.
  • 2) Description: a human-readable label so you remember why you ran it (e.g. “prompt v3”, “gpt-5.2 temp 0.2”, “post-finetune”). Required for Request and Workflow runs.
  • 3) Config: consumer-specific:
    • Request: pick the model and generation params.
    • Workflow: pick the workflow.
    • Agent: pick the agent (and optionally a max-iterations cap).
  • 4) Evaluations: pick the evaluation criteria that score the run.
    • Request/Workflow runs default to Exact Match; add more criteria as needed.
    • Agent runs require at least one criterion (trajectory, outcome, or custom).
  • 5) Review: confirm and launch.

​
What happens after you launch

A run moves through a short lifecycle:
  • Preparing inputs: each canonical item is adapted into the consumer’s native input (identity for a model; the workflow’s own transform for a workflow; a task payload for an agent). You can preview this before launch.
  • Running: the consumer executes every item.
  • Scoring: this runs as a separate pass after execution. Until scoring completes, an item’s score shows as a dash (-). You can also re-score a completed run after editing criteria.
While a run is preparing or running, its detail page reflects live progress.

​
Comparing runs

Because every run shares the set as its axis, you can compare runs against the same set, including runs against different consumers (e.g. a model baseline vs. an agent). Open a run to see per-item results, and use the comparison views to see how a specific item shifted across runs.

​
Deleting or resetting a run

From a run you can cancel an in-progress run, reset failed items to retry just the failures, or delete a run you no longer need. Next: Scoring & Criteria · Interpreting Results.