Skip to main content
A Test Set is a reusable collection of test items you benchmark against, repeatedly, as your prompts, models, agents, and workflows change. Each item is a canonical input + expected output pair. A test set has no type. One set can hold items sourced from production requests, workflow runs, agent tasks, manual entries, or an uploaded file, mixed freely. You choose how to run it (against a model, a workflow, or an agent) later, at run time, see Test Run Execution.

​
Where to find Test Sets

In the Portal, go to Test > Test Sets. The list shows every test set in your company; each item carries a badge showing where it came from.

​
Create a test set (wizard)

Click New Test Set. The wizard has three steps:
  • 1) Details
    • Pick the Application the set belongs to.
    • Give it a Name (required) and optional Description.
    • Optionally mark it Golden: a small, high-signal set you trust as a quality bar (for example, to gate a fine-tuned model).
  • 2) Items: add items now, or skip and add them later. Two modes:
    • Pick items: add from any of four sources (see below). You can queue items from different sources into the same set.
    • Upload JSONL: bulk-import a file where each line declares its own type (request, workflow, agent, or manual).
  • 3) Review: confirm the details and the queued items, then create.
A set is just a named bucket of items scoped to an Application, there is no “request set” vs. “workflow set” anymore. What an item was (its source) is provenance; how you run the set is a separate choice made per test run.

​
Item sources

When picking items, you’ll see four tabs:

Request

Promote a real production chat completion you’ve observed. The request messages become the input; the assistant response becomes the expected output.

Workflow Run

Import a completed workflow run. Its input and final output are flattened into the canonical input/expected shape.

Agent Task

Import a completed agent task. The task’s opening message becomes the input; the agent’s final response becomes the expected output.

Manual

Author an item by hand, type the input messages and the expected assistant output yourself. Useful before you have production traffic.
Imports are idempotent: importing the same request, workflow run, or agent task into the same set twice won’t create a duplicate.

​
Add items from Observe pages

The fastest way to grow a set is to promote interesting traffic you’ve already found:
  • From a Session or Request page: use Add to Test Set to drop that request into an existing set (or create one inline).

​
Correct the expected output

An imported item’s expected output is the original production response, which may itself have been wrong. To encode “what should have happened,” edit an item’s expected output override. This stores your corrected answer for scoring without discarding the original imported ground truth, so you can always revert. This is what makes a test set a true regression bar: the expected output reflects the correct behavior, not just whatever the model happened to say at capture time.

​
Manage an existing test set

Open any test set to:
  • Browse its items (each badged by source).
  • Add more items or remove items.
  • Edit name, description, and Golden flag.
  • Create and review Test Runs (see Test Run Execution).
  • Clone the set (runs as a background job for large sets).

​
Practical guidance

  • Prefer real traffic. Test sets are strongest when they mirror what your users actually do.
  • Keep Golden sets small. A focused, high-signal set is easier to trust and maintain.
  • Keep sets evergreen. Add newly discovered failure modes from Sessions and Requests; retire items that no longer reflect expected behavior.