Skip to main content
A Test Run is scored on two complementary layers: a structural match rate (does the output match the expected output?) and evaluation criteria (richer, configurable checks). Scoring runs as a separate pass after execution, see Test Run Execution.

​
Layer 1: Structural match rate

For Request (model) and Workflow runs, each item’s actual output is compared against its expected output. The item is matched, mismatched, or no expected (when the item has no ground truth). The run’s headline Match Rate is:
Items without an expected output are excluded from the denominator. Agent runs don’t use structural matching, they’re scored entirely by criteria (below).

​
Layer 2: Evaluation criteria

Evaluation criteria are the configurable checks you attach to a run. Each item is scored against every selected criterion. Criteria come from your company’s catalog on the Test > Test Criteria page (see Evaluation Criteria); the built-in Exact Match is selected by default for Request and Workflow runs. Criteria fall into a few families, and they don’t all score the same way:
The 1–5 scale is just the default range for a numeric LLM-judge criterion, it is one option among several, not the scoring model for the whole run. Many criteria are simple pass/fail.

​
The per-item score (0–100%)

In the requests table, each item shows a single normalized percent score, aggregated across its criteria:
  • Boolean / pass-fail criteria contribute 1 (pass) or 0 (fail).
  • Numeric criteria contribute (score − min) / (max − min) using that criterion’s range.
  • Tag criteria are excluded from the score.
Hover an item’s score to see the criteria breakdown: each criterion’s individual result (pass/fail pill, numeric bar, or tag).

​
Agent trajectory scoring

Agent runs can be scored on the trajectory, not just the final answer. Trajectory criteria are deterministic checks such as: required tools were called, tools were called in the expected order, step count / sub-agent depth stayed within bounds, and latency or cost ceilings were respected. Agent runs can also use LLM-judge outcome criteria and custom Python-predicate criteria.

​
Using scores when debugging

  • Start with the distribution. For numeric criteria, the breakdown shows a score distribution, drill into the low buckets first.
  • Read the per-criterion breakdown. It tells you which criterion degraded, not just that the score dropped.
  • Compare responses. For any item, compare the expected output against the run’s actual output side by side (see Interpreting Results).