> ## Documentation Index
> Fetch the complete documentation index at: https://docs.trymaitai.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Test Set Creation

> Build a test set from any mix of production requests, workflow runs, agent tasks, manual entries, or a file upload.

A **Test Set** is a reusable collection of test items you benchmark against, repeatedly, as your
prompts, models, agents, and workflows change. Each item is a canonical **input + expected output**
pair.

A test set has **no type**. One set can hold items sourced from production requests, workflow runs,
agent tasks, manual entries, or an uploaded file, mixed freely. You choose *how* to run it (against
a model, a workflow, or an agent) later, at run time, see [Test Run Execution](/test/test_sets/execution).

## Where to find Test Sets

In the Portal, go to `Test > Test Sets`. The list shows every test set in your company; each item
carries a badge showing where it came from.

## Create a test set (wizard)

Click **New Test Set**. The wizard has three steps:

* **1) Details**
  * Pick the **Application** the set belongs to.
  * Give it a **Name** (required) and optional **Description**.
  * Optionally mark it **Golden**: a small, high-signal set you trust as a quality bar (for
    example, to gate a fine-tuned model).
* **2) Items**: add items now, or skip and add them later. Two modes:
  * **Pick items**: add from any of four sources (see below). You can queue items from *different*
    sources into the same set.
  * **Upload JSONL**: bulk-import a file where each line declares its own `type`
    (`request`, `workflow`, `agent`, or `manual`).
* **3) Review**: confirm the details and the queued items, then create.

<Note>
  A set is just a named bucket of items scoped to an Application, there is no "request set" vs.
  "workflow set" anymore. What an item *was* (its source) is provenance; how you *run* the set is a
  separate choice made per test run.
</Note>

## Item sources

When picking items, you'll see four tabs:

<CardGroup cols={2}>
  <Card title="Request" icon="arrow-right-arrow-left">
    Promote a real production **chat completion** you've observed. The request messages become the
    input; the assistant response becomes the expected output.
  </Card>

  <Card title="Workflow Run" icon="diagram-project">
    Import a completed **workflow run**. Its input and final output are flattened into the canonical
    input/expected shape.
  </Card>

  <Card title="Agent Task" icon="robot">
    Import a completed **agent task**. The task's opening message becomes the input; the agent's
    final response becomes the expected output.
  </Card>

  <Card title="Manual" icon="pen">
    Author an item by hand, type the input messages and the expected assistant output yourself.
    Useful before you have production traffic.
  </Card>
</CardGroup>

Imports are **idempotent**: importing the same request, workflow run, or agent task into the same
set twice won't create a duplicate.

## Add items from Observe pages

The fastest way to grow a set is to promote interesting traffic you've already found:

* **From a Session or Request page**: use **Add to Test Set** to drop that request into an existing
  set (or create one inline).

## Correct the expected output

An imported item's expected output is the *original* production response, which may itself have
been wrong. To encode "what *should* have happened," edit an item's **expected output override**.
This stores your corrected answer for scoring **without** discarding the original imported ground
truth, so you can always revert.

This is what makes a test set a true regression bar: the expected output reflects the correct
behavior, not just whatever the model happened to say at capture time.

## Manage an existing test set

Open any test set to:

* Browse its **items** (each badged by source).
* **Add** more items or **remove** items.
* Edit **name**, **description**, and **Golden** flag.
* Create and review **Test Runs** (see [Test Run Execution](/test/test_sets/execution)).
* **Clone** the set (runs as a background job for large sets).

## Practical guidance

* **Prefer real traffic.** Test sets are strongest when they mirror what your users actually do.
* **Keep Golden sets small.** A focused, high-signal set is easier to trust and maintain.
* **Keep sets evergreen.** Add newly discovered failure modes from Sessions and Requests; retire
  items that no longer reflect expected behavior.