Skip to main content
Evaluation Criteria are the reusable checks that score your Test Runs. You define them once in your company catalog, then select which ones apply when you create a run.

​
Where to manage criteria

In the Portal, go to Test > Test Criteria. You’ll see:
  • System criteria: built-in checks (like Exact Match). Read-only, but you can clone one to customize it.
  • Custom criteria: checks you create for your own quality bar.

​
Criteria types

​
Deterministic

Computed directly from the output. No LLM involved, fully repeatable:
  • Exact Match: output equals expected output
  • Contains: output contains an expected substring
  • Regex Match: output matches a pattern
  • BLEU / ROUGE / Cosine Similarity / Levenshtein: text-similarity scores against the expected output

​
LLM judge

A model grades the output against your instructions. Three scoring modes:
  • Boolean: PASS / FAIL against a yes/no question
  • Numeric: an integer rubric over a configurable range (default 1–5), optionally with named levels (e.g. 1 = Poor … 5 = Perfect)
  • Tags: classify each output into one of N categories
LLM-judge criteria let you pick the evaluation model and, where relevant, map which fields to compare.

​
Agent trajectory (agent runs)

For agent runs, criteria can score how the agent worked, not just its final answer:
  • Tool checks: required tools were called, only whitelisted tools were used, tools ran in the expected sequence
  • Shape checks: step count, sub-agent depth, failure rate within bounds
  • Budget checks: latency and cost ceilings respected
  • Outcome checks: an LLM judge or a custom Python predicate over the final result

​
How criteria attach to a run

You select criteria in the Evaluations step of the Test Run wizard:
  • Request and Workflow runs default to Exact Match; add more as needed.
  • Agent runs require at least one criterion (trajectory, outcome, or custom).
Each item is then scored against every selected criterion. How those results roll up into a single per-item percent (and the run’s match/pass rate) is covered in Scoring & Criteria.