> ## Documentation Index
> Fetch the complete documentation index at: https://docs.trymaitai.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Scoring & Criteria

> How Test Runs are scored: structural match rate plus evaluation criteria.

A Test Run is scored on two complementary layers: a **structural match rate** (does the output match
the expected output?) and **evaluation criteria** (richer, configurable checks). Scoring runs as a
separate pass after execution, see [Test Run Execution](/test/test_sets/execution).

## Layer 1: Structural match rate

For **Request** (model) and **Workflow** runs, each item's actual output is compared against its
expected output. The item is `matched`, `mismatched`, or `no expected` (when the item has no ground
truth). The run's headline **Match Rate** is:

```
match_rate = matched / (matched + mismatched)
```

Items without an expected output are excluded from the denominator. **Agent** runs don't use
structural matching, they're scored entirely by criteria (below).

## Layer 2: Evaluation criteria

**Evaluation criteria** are the configurable checks you attach to a run. Each item is scored against
every selected criterion. Criteria come from your company's catalog on the
`Test > Test Criteria` page (see [Evaluation Criteria](/test/test_sets/criteria)); the built-in
**Exact Match** is selected by default for Request and Workflow runs.

Criteria fall into a few families, and they don't all score the same way:

| Family                                                                                        | How it scores                                                                                             |
| --------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------- |
| **Deterministic** (Exact Match, Contains, Regex, BLEU, ROUGE, Cosine Similarity, Levenshtein) | Pass/fail or a computed similarity value                                                                  |
| **LLM judge, boolean**                                                                        | PASS / FAIL                                                                                               |
| **LLM judge, numeric**                                                                        | An integer rubric over a configurable range (default **1–5**), optionally with named levels               |
| **LLM judge, tags**                                                                           | Classifies each output into one of N categories                                                           |
| **Agent trajectory** (agent runs only)                                                        | Deterministic checks on *how* the agent worked, e.g. tools used, tool sequence, step count, latency, cost |

<Note>
  The 1–5 scale is just the default range for a **numeric** LLM-judge criterion, it is one option
  among several, not the scoring model for the whole run. Many criteria are simple pass/fail.
</Note>

## The per-item score (0–100%)

In the requests table, each item shows a single normalized **percent** score, aggregated across its
criteria:

* **Boolean / pass-fail** criteria contribute `1` (pass) or `0` (fail).
* **Numeric** criteria contribute `(score − min) / (max − min)` using that criterion's range.
* **Tag** criteria are excluded from the score.

```
normalized_score = round( (sum of per-criterion contributions) / (criteria counted) × 100 )
```

Hover an item's score to see the **criteria breakdown**: each criterion's individual result
(pass/fail pill, numeric bar, or tag).

## Agent trajectory scoring

Agent runs can be scored on the *trajectory*, not just the final answer. Trajectory criteria are
deterministic checks such as: required tools were called, tools were called in the expected order,
step count / sub-agent depth stayed within bounds, and latency or cost ceilings were respected. Agent
runs can also use LLM-judge outcome criteria and custom Python-predicate criteria.

## Using scores when debugging

* **Start with the distribution.** For numeric criteria, the breakdown shows a score distribution,
  drill into the low buckets first.
* **Read the per-criterion breakdown.** It tells you *which* criterion degraded, not just that the
  score dropped.
* **Compare responses.** For any item, compare the expected output against the run's actual output
  side by side (see [Interpreting Results](/test/test_sets/results)).