# Add sub agent Source: https://docs.trymaitai.com/api-reference/agents/add-sub-agent /openapi.yaml post /agents/{agent_id}/subagents Attach an existing agent as a sub-agent of a parent agent. # Add supplied action Source: https://docs.trymaitai.com/api-reference/agents/add-supplied-action /openapi.yaml post /agents/{agent_id}/actions/supplied/{catalog_action_id} Clone a supplied action from the catalog onto an agent. The catalog row is never modified. The clone is a new agent_action with provenance stamped in ``meta.supplied``. Returns 409 when the agent already has an action with the same name. # Create agent Source: https://docs.trymaitai.com/api-reference/agents/create-agent /openapi.yaml post /agents Create a new agent under an application. Requires `application_id` and `agent_name`. # Create agent action Source: https://docs.trymaitai.com/api-reference/agents/create-agent-action /openapi.yaml post /agents/{agent_id}/actions Add a new action (LLM, API, or webhook capability) to an agent. # Create routing rule Source: https://docs.trymaitai.com/api-reference/agents/create-routing-rule /openapi.yaml post /agents/{agent_id}/routing/rules Create a new conditional routing rule for an agent. # Delete agent Source: https://docs.trymaitai.com/api-reference/agents/delete-agent /openapi.yaml delete /agents/{agent_id} Permanently delete an agent and its actions. # Delete agent action Source: https://docs.trymaitai.com/api-reference/agents/delete-agent-action /openapi.yaml delete /agents/actions/{action_id} Permanently remove an action from an agent. # Delete agent release Source: https://docs.trymaitai.com/api-reference/agents/delete-agent-release /openapi.yaml delete /agents/{agent_id}/releases/{name} Remove a named release. # Delete form fields Source: https://docs.trymaitai.com/api-reference/agents/delete-form-fields /openapi.yaml delete /agents/{agent_id}/form-fields Remove all form fields from an agent, clearing its input form. # Delete routing rule Source: https://docs.trymaitai.com/api-reference/agents/delete-routing-rule /openapi.yaml delete /agents/{agent_id}/routing/rules/{rule_id} Permanently delete a routing rule. # Diff agent versions Source: https://docs.trymaitai.com/api-reference/agents/diff-agent-versions /openapi.yaml get /agents/{agent_id}/versions/diff Structured diff between two agent versions. Pass `from` and `to` as version strings (e.g. "1.2.0"), a release name, or "draft" (the default for `to`). Returns per-section deltas: agent fields, actions added/removed/ changed (dotted config paths), sub-agents, and routing rules. # Disable action Source: https://docs.trymaitai.com/api-reference/agents/disable-action /openapi.yaml put /agents/actions/{action_id}/disable Disable an action so the agent no longer invokes it. # Disable agent system step Source: https://docs.trymaitai.com/api-reference/agents/disable-agent-system-step /openapi.yaml put /agents/{agent_id}/system-steps/{step}/disable Disable an optional runtime system step (`processing` or `progress`). # Disable sub agent Source: https://docs.trymaitai.com/api-reference/agents/disable-sub-agent /openapi.yaml put /agents/{agent_id}/subagents/{child_agent_id}/disable Disable a sub-agent relationship without removing it. # Enable action Source: https://docs.trymaitai.com/api-reference/agents/enable-action /openapi.yaml put /agents/actions/{action_id}/enable Re-enable a previously disabled action. # Enable agent system step Source: https://docs.trymaitai.com/api-reference/agents/enable-agent-system-step /openapi.yaml put /agents/{agent_id}/system-steps/{step}/enable Enable an optional runtime system step (`processing` or `progress`). # Enable sub agent Source: https://docs.trymaitai.com/api-reference/agents/enable-sub-agent /openapi.yaml put /agents/{agent_id}/subagents/{child_agent_id}/enable Re-enable a previously disabled sub-agent relationship. # Get agent Source: https://docs.trymaitai.com/api-reference/agents/get-agent /openapi.yaml get /agents/{agent_id} Retrieve an agent by ID. Use `detailed=true` to include actions, sub-agents, and full config. # Get agent action Source: https://docs.trymaitai.com/api-reference/agents/get-agent-action /openapi.yaml get /agents/actions/{action_id} Retrieve a single action by ID. Pass `include_intent=true` to also resolve the backing intent (application action): LLM actions are captured in the chat pipeline under `intent = action_name` within the agent's application, so intent-level tooling (test sets, configs with fallback models, monitors) keys off the returned `intent` ids. # Get agent invocation Source: https://docs.trymaitai.com/api-reference/agents/get-agent-invocation /openapi.yaml get /agents/{agent_id}/invocations/{task_id} Poll an asynchronous agent invocation started with `wait=false`. Requires the `request_id` returned by the invoke call. # Get agent session detail Source: https://docs.trymaitai.com/api-reference/agents/get-agent-session-detail /openapi.yaml get /agents/{agent_id}/sessions/{session_id} Retrieve an agent session including all its requests and responses. # Get agent session timeline Source: https://docs.trymaitai.com/api-reference/agents/get-agent-session-timeline /openapi.yaml get /agents/{agent_id}/sessions/{session_id}/timeline Retrieve the chronological timeline of events in an agent session. # Get agent status Source: https://docs.trymaitai.com/api-reference/agents/get-agent-status /openapi.yaml get /agents/{agent_id}/status One-shot release/version status: whether the draft has unpublished changes, the latest published version, total version count, and every named release with the version it pins. # Get agent system step config Source: https://docs.trymaitai.com/api-reference/agents/get-agent-system-step-config /openapi.yaml get /agents/{agent_id}/system-steps/{step}/config Get this agent's model/params config for a system step (model, temperature, max_tokens, etc.). Stored per-agent in agent_config, affects only this agent. # Get agent system step prompt Source: https://docs.trymaitai.com/api-reference/agents/get-agent-system-step-prompt /openapi.yaml get /agents/{agent_id}/system-steps/{step}/prompt Get a system step's prompt: `template` (ordered {{reference}} tokens) and `blocks` (segment metadata + content), plus whether it's an override. Stored per-agent in agent_config, affects only this agent. # Get agent task timeline Source: https://docs.trymaitai.com/api-reference/agents/get-agent-task-timeline /openapi.yaml get /agents/{agent_id}/tasks/{task_id}/timeline Retrieve the timeline for a specific task within an agent session. # Get agent task tree Source: https://docs.trymaitai.com/api-reference/agents/get-agent-task-tree /openapi.yaml get /agents/tasks/{task_id}/tree Retrieve the hierarchical tree of sub-tasks spawned from a root task. # Get agent version Source: https://docs.trymaitai.com/api-reference/agents/get-agent-version /openapi.yaml get /agents/{agent_id}/versions/{version} Retrieve a specific agent version snapshot by version string. # Get form fields Source: https://docs.trymaitai.com/api-reference/agents/get-form-fields /openapi.yaml get /agents/{agent_id}/form-fields Retrieve the form field definitions for an agent's input form. # Get routing config Source: https://docs.trymaitai.com/api-reference/agents/get-routing-config /openapi.yaml get /agents/{agent_id}/routing/config Retrieve the routing configuration and all routing rules for an agent. # Get routing result Source: https://docs.trymaitai.com/api-reference/agents/get-routing-result /openapi.yaml get /agents/{agent_id}/routing/results/{routing_result_id} Retrieve detailed routing result evaluations for an agent. # Import agents Source: https://docs.trymaitai.com/api-reference/agents/import-agents /openapi.yaml post /agents/import Import one or more code-defined agents into Maitai from a canonical manifest. The manifest describes the agents, their capabilities (actions), and handoffs (sub-agents) in a framework-agnostic shape. The endpoint: 1. Resolves or creates the target application. 2. Pre-flights every manifest agent against existing rows, so ``on_conflict='error'`` fails fast without any writes when a conflict is detected. 3. Applies the writes through the ``agent_service`` functions, which autocommit per call. These services perform read-backs and cache warming on separate asyncio tasks (``asyncio.gather`` / ``create_eager_task``); those child tasks open their own pooled connections, so the import must NOT run inside one outer ``connection.async_cursor()`` transaction or those read-backs cannot see the uncommitted rows (they raise ``ResourceNotFound``). The pre-flight in step 2 is what guards against partial writes for ``on_conflict='error'``. 4. ``on_conflict='update'`` is non-destructive: only fields the adapter actually owns are overwritten (Portal-edited fields like ``is_default``, ``invocation_mode``, action ``meta`` survive), and ``agent_config`` is merged rather than replaced so Portal-set keys like ``routing_config`` survive. 5. By default the import is additive, actions or sub-agent links removed from the source code are NOT deleted from Maitai. Pass ``prune=True`` (CLI: ``--prune``) for a destructive reconcile that makes the imported code the source of truth. # Invoke agent Source: https://docs.trymaitai.com/api-reference/agents/invoke-agent /openapi.yaml post /agents/{agent_id}/invoke Run an agent and wait for its final response. Targets the draft by default; pass `version` (e.g. "1.2.0") or `release` (e.g. "production") to pin. Body: `{message}` or `{messages: [...]}`, plus optional `session_id`, `max_iterations`, and `secrets` ({NAME: value} for actions that declare `{{secrets.NAME}}` placeholders). Set `wait=false` to enqueue asynchronously and poll with the task endpoint instead. # List action versions Source: https://docs.trymaitai.com/api-reference/agents/list-action-versions /openapi.yaml get /agents/actions/{action_id}/versions List published versions of an action. # List agent actions Source: https://docs.trymaitai.com/api-reference/agents/list-agent-actions /openapi.yaml get /agents/{agent_id}/actions List all actions (capabilities) configured for an agent. # List agent releases Source: https://docs.trymaitai.com/api-reference/agents/list-agent-releases /openapi.yaml get /agents/{agent_id}/releases List all named releases (e.g. production, staging) for an agent. # List agent sessions Source: https://docs.trymaitai.com/api-reference/agents/list-agent-sessions /openapi.yaml get /agents/{agent_id}/sessions List paginated agent sessions, optionally filtered by version. # List agent system steps Source: https://docs.trymaitai.com/api-reference/agents/list-agent-system-steps /openapi.yaml get /agents/{agent_id}/system-steps List the agent runtime system steps for this agent. Prompts and model/params are stored per-agent in agent_config, editing affects only this agent. # List agent versions Source: https://docs.trymaitai.com/api-reference/agents/list-agent-versions /openapi.yaml get /agents/{agent_id}/versions List published versions of an agent, ordered by most recent. # List agents Source: https://docs.trymaitai.com/api-reference/agents/list-agents /openapi.yaml get /agents List all agents belonging to your company. Optionally include sub-agents and actions. Pass ``application_id`` to scope the listing to one application, avoids O(total agents) work on large tenants when callers (e.g. ``maitai agents import --diff``) only need agents under one app. # List sub agents Source: https://docs.trymaitai.com/api-reference/agents/list-sub-agents /openapi.yaml get /agents/{agent_id}/subagents List all sub-agents attached to a parent agent. # List supplied actions Source: https://docs.trymaitai.com/api-reference/agents/list-supplied-actions /openapi.yaml get /agents/supplied-actions List Maitai-supplied actions available to clone onto an agent. Returns curated catalog entries only (no action_config). Cloned actions become normal agent actions and are managed via the standard action CRUD endpoints. # Patch agent Source: https://docs.trymaitai.com/api-reference/agents/patch-agent /openapi.yaml patch /agents/{agent_id} Partially update an agent using JSON Merge Patch semantics (RFC 7386): nested objects merge recursively, `null` deletes a key, scalars/arrays replace. Unlike PUT, `agent_config` and `meta` are merged onto the existing values instead of replaced wholesale, e.g. `{"agent_config": {"high_performance": true}}` flips one flag and leaves the rest of the config untouched. # Patch agent action Source: https://docs.trymaitai.com/api-reference/agents/patch-agent-action /openapi.yaml patch /agents/actions/{action_id} Partially update an action using JSON Merge Patch semantics (RFC 7386). Unlike PUT (which replaces `action_config` wholesale), nested objects merge recursively and `null` deletes a key, e.g. `{"action_config": {"prompt": "..."}}` changes just the prompt, and `{"action_config": {"model_config": {"model": "gpt-5.2"}}}` changes just the model. # Publish action version Source: https://docs.trymaitai.com/api-reference/agents/publish-action-version /openapi.yaml post /agents/actions/{action_id}/publish Publish the current action configuration as a new versioned snapshot. # Publish agent version Source: https://docs.trymaitai.com/api-reference/agents/publish-agent-version /openapi.yaml post /agents/{agent_id}/publish Publish the current draft agent configuration as a new versioned snapshot. # Remove sub agent Source: https://docs.trymaitai.com/api-reference/agents/remove-sub-agent /openapi.yaml delete /agents/{agent_id}/subagents/{child_agent_id} Detach a sub-agent from its parent. # Reorder routing rule Source: https://docs.trymaitai.com/api-reference/agents/reorder-routing-rule /openapi.yaml put /agents/{agent_id}/routing/rules/{rule_id}/reorder Reorder a routing rule to a new position. # Reset agent system step config Source: https://docs.trymaitai.com/api-reference/agents/reset-agent-system-step-config /openapi.yaml delete /agents/{agent_id}/system-steps/{step}/config Reset this agent's model/params config for a system step to the seed default. # Reset agent system step prompt Source: https://docs.trymaitai.com/api-reference/agents/reset-agent-system-step-prompt /openapi.yaml delete /agents/{agent_id}/system-steps/{step}/prompt Reset a system step's prompt to the canonical seed default (preserves the step's model/params config). Affects only this agent. # Restore action version Source: https://docs.trymaitai.com/api-reference/agents/restore-action-version /openapi.yaml post /agents/actions/{action_id}/versions/{version}/restore Restore a published action version into the editable draft, so new changes build on it. Overwrites the current action draft with that version's snapshot. # Restore agent version Source: https://docs.trymaitai.com/api-reference/agents/restore-agent-version /openapi.yaml post /agents/{agent_id}/versions/{version}/restore Restore a published version into the editable draft, so new changes build on it. Overwrites the current draft with that version's snapshot (publish again to create a new version from it). # Test routing rule Source: https://docs.trymaitai.com/api-reference/agents/test-routing-rule /openapi.yaml post /agents/{agent_id}/routing/rules/{rule_id}/test Test a routing rule against provided input values. # Update agent Source: https://docs.trymaitai.com/api-reference/agents/update-agent /openapi.yaml put /agents/{agent_id} Update agent fields such as name, description, config, or status. # Update agent action Source: https://docs.trymaitai.com/api-reference/agents/update-agent-action /openapi.yaml put /agents/actions/{action_id} Update an action's name, type, description, config, or invocation mode. # Update agent system step config Source: https://docs.trymaitai.com/api-reference/agents/update-agent-system-step-config /openapi.yaml put /agents/{agent_id}/system-steps/{step}/config Update this agent's model/params for a system step. Body is a config object, e.g. `{model, temperature, max_tokens}`. For the respond step, set structured JSON via `response_format`. Top-level keys merge. Stored per-agent in agent_config, affects only this agent. # Update agent system step prompt Source: https://docs.trymaitai.com/api-reference/agents/update-agent-system-step-prompt /openapi.yaml put /agents/{agent_id}/system-steps/{step}/prompt Update a system step's prompt. Body: `{template, blocks:[{reference, content}]}`. Only editable block content and template token order may change, blocks cannot be added or removed. Stored per-agent in agent_config, affects only this agent. # Update form fields Source: https://docs.trymaitai.com/api-reference/agents/update-form-fields /openapi.yaml put /agents/{agent_id}/form-fields Replace all form field definitions for an agent. # Update routing config Source: https://docs.trymaitai.com/api-reference/agents/update-routing-config /openapi.yaml put /agents/{agent_id}/routing/config Update the routing mode (llm or rules) and default route for an agent. # Update routing rule Source: https://docs.trymaitai.com/api-reference/agents/update-routing-rule /openapi.yaml put /agents/{agent_id}/routing/rules/{rule_id} Update the conditions, route, or label of an existing routing rule. # Upsert agent release Source: https://docs.trymaitai.com/api-reference/agents/upsert-agent-release /openapi.yaml put /agents/{agent_id}/releases/{name} Create or update a named release, optionally pinning it to a version. # Get agent request analytics Source: https://docs.trymaitai.com/api-reference/analytics/get-agent-request-analytics /openapi.yaml get /analytics/agents/{agent_id}/requests Get request analytics for an agent. # Get application action fault rate analytics Source: https://docs.trymaitai.com/api-reference/analytics/get-application-action-fault-rate-analytics /openapi.yaml get /analytics/applications/{application_id}/intents/{action_id}/fault-rate Get fault-rate analytics for an application intent. # Get application action request analytics Source: https://docs.trymaitai.com/api-reference/analytics/get-application-action-request-analytics /openapi.yaml get /analytics/applications/{application_id}/intents/{action_id}/requests Get request analytics for an application intent. # Get application fault rate analytics Source: https://docs.trymaitai.com/api-reference/analytics/get-application-fault-rate-analytics /openapi.yaml get /analytics/applications/{application_id}/fault-rate Get fault-rate analytics for an application. # Get application request analytics Source: https://docs.trymaitai.com/api-reference/analytics/get-application-request-analytics /openapi.yaml get /analytics/applications/{application_id}/requests Get request analytics for an application. # Get company application activity analytics Source: https://docs.trymaitai.com/api-reference/analytics/get-company-application-activity-analytics /openapi.yaml get /analytics/company/application-activity Get per-application request activity for the company. # Get company model usage analytics Source: https://docs.trymaitai.com/api-reference/analytics/get-company-model-usage-analytics /openapi.yaml get /analytics/company/model-usage Get per-model token usage for the company. # Get company request analytics Source: https://docs.trymaitai.com/api-reference/analytics/get-company-request-analytics /openapi.yaml get /analytics/company/requests Get company-level request analytics. # Get intent group fault rate analytics Source: https://docs.trymaitai.com/api-reference/analytics/get-intent-group-fault-rate-analytics /openapi.yaml get /analytics/intent-groups/{intent_group_id}/fault-rate Get fault-rate analytics for an intent group. # Get intent group request analytics Source: https://docs.trymaitai.com/api-reference/analytics/get-intent-group-request-analytics /openapi.yaml get /analytics/intent-groups/{intent_group_id}/requests Get request analytics for an intent group. # Create application Source: https://docs.trymaitai.com/api-reference/applications/create-application /openapi.yaml post /applications Create a new application under your company. # Delete application Source: https://docs.trymaitai.com/api-reference/applications/delete-application /openapi.yaml delete /applications/{application_id} Permanently delete an application and all its associated intents. # Get application Source: https://docs.trymaitai.com/api-reference/applications/get-application /openapi.yaml get /applications/{application_id} Retrieve a single application by ID with full detail. # Get application config Source: https://docs.trymaitai.com/api-reference/applications/get-application-config /openapi.yaml get /applications/{application_id}/config Retrieve the configuration object for an application. # List application models Source: https://docs.trymaitai.com/api-reference/applications/list-application-models /openapi.yaml get /applications/{application_id}/models List all models assigned to an application. # List application sessions Source: https://docs.trymaitai.com/api-reference/applications/list-application-sessions /openapi.yaml get /applications/{application_id}/sessions List paginated chat sessions for an application. # List application workflow runs Source: https://docs.trymaitai.com/api-reference/applications/list-application-workflow-runs /openapi.yaml get /applications/{application_id}/workflow-runs List paginated workflow execution runs for an application. # List applications Source: https://docs.trymaitai.com/api-reference/applications/list-applications /openapi.yaml get /applications List all applications belonging to your company. Use `detailed=true` to include full configuration. # Update application Source: https://docs.trymaitai.com/api-reference/applications/update-application /openapi.yaml put /applications/{application_id} Update an application's name, reference name, or other fields. # Update application config Source: https://docs.trymaitai.com/api-reference/applications/update-application-config /openapi.yaml put /applications/{application_id}/config Replace the configuration object for an application. # Create composition Source: https://docs.trymaitai.com/api-reference/compositions/create-composition /openapi.yaml post /compositions Create a dataset composition. # Delete composition Source: https://docs.trymaitai.com/api-reference/compositions/delete-composition /openapi.yaml delete /compositions/{composition_id} Delete a composition. # Get composition Source: https://docs.trymaitai.com/api-reference/compositions/get-composition /openapi.yaml get /compositions/{composition_id} Retrieve a single composition by ID. # List composition versions Source: https://docs.trymaitai.com/api-reference/compositions/list-composition-versions /openapi.yaml get /compositions/{composition_id}/versions List all versions for a composition lineage. # List compositions Source: https://docs.trymaitai.com/api-reference/compositions/list-compositions /openapi.yaml get /compositions List dataset compositions for your company. # List compositions by application intent Source: https://docs.trymaitai.com/api-reference/compositions/list-compositions-by-application-intent /openapi.yaml get /compositions/applications/{application_id}/intents/{intent_id} List compositions for an application intent. # List compositions by intent group Source: https://docs.trymaitai.com/api-reference/compositions/list-compositions-by-intent-group /openapi.yaml get /compositions/intent-groups/{intent_group_id} List compositions for an intent group. # Update composition Source: https://docs.trymaitai.com/api-reference/compositions/update-composition /openapi.yaml put /compositions/{composition_id} Update a composition by creating a new version. # Validate rl composition Source: https://docs.trymaitai.com/api-reference/compositions/validate-rl-composition /openapi.yaml post /compositions/validate-rl Validate that datasets can form an RL composition. # API Coverage Source: https://docs.trymaitai.com/api-reference/coverage What the Developer API covers, what is intentionally excluded, and where to find generated references. The Maitai Developer API exposes the product resources most teams manage in the Portal: applications, intents, agents, workflows, datasets, test sets, test runs, sentinels, models, analytics, and quality workflows. The API is organized under: ```text theme={null} https://api.trymaitai.ai/api/v1 ``` ## Covered resource groups Use the API Reference sidebar for endpoint-level details. The public API currently includes: * Applications, intents, intent groups, sessions, requests, and evaluations * Agents, routing rules, and conversation trees * Workflows, workflow versions/releases, workflow artifacts, workflow runs, workflow test sets, and workflow test runs * Datasets, test sets, test runs, finetune runs, models, and model drift checks * Sentinels and evaluation criteria * Analytics, reports, tagging, continuous learning, and Compass searches The CLI mirrors these top-level resources where a generic list/get/create/update/delete command is useful. For specialized actions, use `maitai api` with the endpoint path. ## Response shapes Most `/api/v1` endpoints use the standard response envelope: ```json theme={null} { "data": {} } ``` Some endpoints delegate to the same backend handlers used by the Portal for long-running or complex operations. These may return a Portal-shaped response rather than the standard envelope. The OpenAPI reference is the source of truth for each endpoint. ## Not exposed The Developer API intentionally does not expose internal account or operations controls, including: * Billing and account administration * API-key management * User permissions and company administration * Email/notification delivery controls * Infrastructure-admin and VAST instance controls There are also a few Portal-only product helpers that are not yet part of `/api/v1`: application context file management, dataset review/public-review workflows, shared-resource sharing, sandbox conversations, and some import/search/validation helper endpoints. ## Choosing an interface * Use the [API Reference](/api-reference) when integrating directly over HTTP. * Use the [CLI Reference](/cli-reference/overview) for terminal workflows and scripts. * Use the generated `maitai api` command for specialized endpoints that do not have a dedicated top-level CLI command. # Add dataset requests bulk Source: https://docs.trymaitai.com/api-reference/datasets/add-dataset-requests-bulk /openapi.yaml put /datasets/{dataset_id}/requests/add Add requests to a dataset using the Portal bulk payload shape. # Add dataset tag Source: https://docs.trymaitai.com/api-reference/datasets/add-dataset-tag /openapi.yaml post /datasets/{dataset_id}/tags Add a tag to a dataset. # Add requests to dataset Source: https://docs.trymaitai.com/api-reference/datasets/add-requests-to-dataset /openapi.yaml post /datasets/{dataset_id}/requests Add request/response pairs to a dataset. Accepts either body shape: - `{"requests": [{"request_id": "..."}]}` (canonical) - `{"request_ids": ["...", "..."]}` (convenience, flat string array) # Apply dataset prune Source: https://docs.trymaitai.com/api-reference/datasets/apply-dataset-prune /openapi.yaml post /datasets/{dataset_id}/prune/apply Apply dataset pruning. # Apply dataset regex replacements Source: https://docs.trymaitai.com/api-reference/datasets/apply-dataset-regex-replacements /openapi.yaml post /datasets/{dataset_id}/regex-replacements Apply regex replacements to dataset rows. # Augment dataset reasoning Source: https://docs.trymaitai.com/api-reference/datasets/augment-dataset-reasoning /openapi.yaml post /datasets/{dataset_id}/reasoning/augment Augment dataset rows with reasoning. # Cancel dataset generation Source: https://docs.trymaitai.com/api-reference/datasets/cancel-dataset-generation /openapi.yaml post /datasets/{dataset_id}/generate/cancel Cancel AI dataset generation. # Cancel dataset s3 import Source: https://docs.trymaitai.com/api-reference/datasets/cancel-dataset-s3-import /openapi.yaml post /datasets/{dataset_id}/s3/cancel Cancel an in-progress S3 dataset import. # Clone dataset Source: https://docs.trymaitai.com/api-reference/datasets/clone-dataset /openapi.yaml post /datasets/{dataset_id}/clone Clone a dataset. # Create dataset Source: https://docs.trymaitai.com/api-reference/datasets/create-dataset /openapi.yaml post /datasets Create a new training dataset linked to an intent group. # Delete dataset Source: https://docs.trymaitai.com/api-reference/datasets/delete-dataset /openapi.yaml delete /datasets/{dataset_id} Permanently delete a dataset and disassociate its requests. Pass ``?cascade=lineage`` to delete the entire version lineage instead of only the specified version. Returns 409 if a continuous learning run still references any targeted version. # Estimate dataset generation cost Source: https://docs.trymaitai.com/api-reference/datasets/estimate-dataset-generation-cost /openapi.yaml post /datasets/generate/estimate-cost Estimate AI dataset generation cost. # Get dataset Source: https://docs.trymaitai.com/api-reference/datasets/get-dataset /openapi.yaml get /datasets/{dataset_id} Retrieve a single dataset by ID. # Get dataset delete impact Source: https://docs.trymaitai.com/api-reference/datasets/get-dataset-delete-impact /openapi.yaml get /datasets/{dataset_id}/delete-impact Describe what would be deleted if this dataset (or its lineage) were removed. # Get dataset prompt Source: https://docs.trymaitai.com/api-reference/datasets/get-dataset-prompt /openapi.yaml get /datasets/{dataset_id}/prompt Get a generated prompt summary for a dataset. # Get dataset regex replacements applicable count Source: https://docs.trymaitai.com/api-reference/datasets/get-dataset-regex-replacements-applicable-count /openapi.yaml post /datasets/{dataset_id}/regex-replacements/count Count dataset rows that would be affected by regex replacements. # Get dataset request distribution Source: https://docs.trymaitai.com/api-reference/datasets/get-dataset-request-distribution /openapi.yaml get /datasets/{dataset_id}/request-distribution Get request distribution facets for a dataset. # Get dataset request reasoning Source: https://docs.trymaitai.com/api-reference/datasets/get-dataset-request-reasoning /openapi.yaml get /datasets/{dataset_id}/requests/{request_id}/reasoning Get reasoning for one dataset request. # Get dataset s3 syncer info Source: https://docs.trymaitai.com/api-reference/datasets/get-dataset-s3-syncer-info /openapi.yaml get /datasets/s3/syncer-info Get S3 syncer service account and external ID details. # Get dataset status Source: https://docs.trymaitai.com/api-reference/datasets/get-dataset-status /openapi.yaml get /datasets/{dataset_id}/status Get dataset import/generation status. # Get estimated dataset request count Source: https://docs.trymaitai.com/api-reference/datasets/get-estimated-dataset-request-count /openapi.yaml get /datasets/estimated-request-count Estimate dataset request count for the provided filters. # List dataset request ids Source: https://docs.trymaitai.com/api-reference/datasets/list-dataset-request-ids /openapi.yaml get /datasets/{dataset_id}/request-ids List request IDs included in a dataset. # List dataset requests Source: https://docs.trymaitai.com/api-reference/datasets/list-dataset-requests /openapi.yaml get /datasets/{dataset_id}/requests List the request/response pairs included in a dataset. # List dataset tags Source: https://docs.trymaitai.com/api-reference/datasets/list-dataset-tags /openapi.yaml get /datasets/{dataset_id}/tags List tags for a dataset. # List dataset versions Source: https://docs.trymaitai.com/api-reference/datasets/list-dataset-versions /openapi.yaml get /datasets/{dataset_id}/versions List dataset versions. # List datasets Source: https://docs.trymaitai.com/api-reference/datasets/list-datasets /openapi.yaml get /datasets List all training datasets for your company. # List datasets by intent Source: https://docs.trymaitai.com/api-reference/datasets/list-datasets-by-intent /openapi.yaml get /datasets/applications/{application_id}/intents/{intent_id} List datasets for an application intent. # List datasets by intent group Source: https://docs.trymaitai.com/api-reference/datasets/list-datasets-by-intent-group /openapi.yaml get /datasets/intent-groups/{intent_group_id} List datasets for an intent group. # List datasets by tags Source: https://docs.trymaitai.com/api-reference/datasets/list-datasets-by-tags /openapi.yaml get /datasets/by-tags List datasets matching tag filters. # List datasets by test set Source: https://docs.trymaitai.com/api-reference/datasets/list-datasets-by-test-set /openapi.yaml get /datasets/test-sets/{test_set_id} List datasets that are associated with a test set. # List eligible dataset requests Source: https://docs.trymaitai.com/api-reference/datasets/list-eligible-dataset-requests /openapi.yaml get /datasets/{dataset_id}/eligible-requests List requests eligible for addition to a dataset. # List valid dataset tags Source: https://docs.trymaitai.com/api-reference/datasets/list-valid-dataset-tags /openapi.yaml get /datasets/tags/valid List valid dataset tags. # Pause dataset generation Source: https://docs.trymaitai.com/api-reference/datasets/pause-dataset-generation /openapi.yaml post /datasets/{dataset_id}/generate/pause Pause AI dataset generation. # Preview dataset generation Source: https://docs.trymaitai.com/api-reference/datasets/preview-dataset-generation /openapi.yaml post /datasets/generate/preview Preview AI-generated dataset rows. # Preview dataset prune Source: https://docs.trymaitai.com/api-reference/datasets/preview-dataset-prune /openapi.yaml post /datasets/{dataset_id}/prune/preview Preview dataset pruning. # Reimport dataset s3 source Source: https://docs.trymaitai.com/api-reference/datasets/reimport-dataset-s3-source /openapi.yaml post /datasets/{dataset_id}/s3/reimport Re-run S3 import for a dataset. # Remove dataset requests bulk Source: https://docs.trymaitai.com/api-reference/datasets/remove-dataset-requests-bulk /openapi.yaml put /datasets/{dataset_id}/requests/remove Remove requests from a dataset using the Portal bulk payload shape. # Remove dataset tag Source: https://docs.trymaitai.com/api-reference/datasets/remove-dataset-tag /openapi.yaml delete /datasets/{dataset_id}/tags/{tag} Remove a tag from a dataset. # Remove requests from dataset Source: https://docs.trymaitai.com/api-reference/datasets/remove-requests-from-dataset /openapi.yaml delete /datasets/{dataset_id}/requests Remove request/response pairs from a dataset. Accepts either body shape: - `{"requests": [{"request_id": "..."}]}` (canonical) - `{"request_ids": ["...", "..."]}` (convenience, flat string array) # Resume dataset generation Source: https://docs.trymaitai.com/api-reference/datasets/resume-dataset-generation /openapi.yaml post /datasets/{dataset_id}/generate/resume Resume AI dataset generation. # Sample clone dataset Source: https://docs.trymaitai.com/api-reference/datasets/sample-clone-dataset /openapi.yaml post /datasets/{dataset_id}/sample-clone Create a sampled clone of a dataset. # Search added dataset requests Source: https://docs.trymaitai.com/api-reference/datasets/search-added-dataset-requests /openapi.yaml post /datasets/{dataset_id}/added-requests/search Search requests already added to a dataset. # Search eligible dataset requests Source: https://docs.trymaitai.com/api-reference/datasets/search-eligible-dataset-requests /openapi.yaml post /datasets/{dataset_id}/eligible-requests/search Search requests eligible for addition to a dataset. # Set dataset tags Source: https://docs.trymaitai.com/api-reference/datasets/set-dataset-tags /openapi.yaml put /datasets/{dataset_id}/tags Replace all tags on a dataset. # Start dataset generation Source: https://docs.trymaitai.com/api-reference/datasets/start-dataset-generation /openapi.yaml post /datasets/generate Start AI dataset generation. # Update dataset Source: https://docs.trymaitai.com/api-reference/datasets/update-dataset /openapi.yaml put /datasets/{dataset_id} Update a dataset's name, description, or sampling configuration. # Validate dataset s3 source Source: https://docs.trymaitai.com/api-reference/datasets/validate-dataset-s3-source /openapi.yaml post /datasets/s3/validate Validate an S3 source before dataset import. # Create evaluation criteria Source: https://docs.trymaitai.com/api-reference/evaluation-criteria/create-evaluation-criteria /openapi.yaml post /evaluation-criteria Create evaluation criteria. # Delete evaluation criteria Source: https://docs.trymaitai.com/api-reference/evaluation-criteria/delete-evaluation-criteria /openapi.yaml delete /evaluation-criteria/{criteria_id} Delete evaluation criteria. # List available evaluation criteria Source: https://docs.trymaitai.com/api-reference/evaluation-criteria/list-available-evaluation-criteria /openapi.yaml get /evaluation-criteria/available List built-in evaluation criteria options. # List evaluation criteria Source: https://docs.trymaitai.com/api-reference/evaluation-criteria/list-evaluation-criteria /openapi.yaml get /evaluation-criteria List saved evaluation criteria. # Update evaluation criteria Source: https://docs.trymaitai.com/api-reference/evaluation-criteria/update-evaluation-criteria /openapi.yaml put /evaluation-criteria/{criteria_id} Update evaluation criteria. # Create evaluation Source: https://docs.trymaitai.com/api-reference/evaluations/create-evaluation /openapi.yaml post /evaluations Start a batch evaluation run against selected sentinels and requests. # Get evaluation Source: https://docs.trymaitai.com/api-reference/evaluations/get-evaluation /openapi.yaml get /evaluations/{evaluation_run_id} Retrieve an evaluation run. Use `full=true` to include all request data. # Get evaluation results Source: https://docs.trymaitai.com/api-reference/evaluations/get-evaluation-results /openapi.yaml get /evaluations/{evaluation_run_id}/results Retrieve per-request evaluation scores and sentinel verdicts for a run. # List evaluations Source: https://docs.trymaitai.com/api-reference/evaluations/list-evaluations /openapi.yaml get /evaluations List all evaluation runs for your company. # Cancel finetune deploy Source: https://docs.trymaitai.com/api-reference/finetune-runs/cancel-finetune-deploy /openapi.yaml post /finetune-runs/{run_id}/deploy/cancel Cancel an in-progress hosted-model deployment for a finetune run. Only works while the deployment is still active; a completed or already-terminal deployment returns 400. # Cancel finetune run Source: https://docs.trymaitai.com/api-reference/finetune-runs/cancel-finetune-run /openapi.yaml post /finetune-runs/{run_id}/cancel Cancel a running finetune job. # Create finetune run Source: https://docs.trymaitai.com/api-reference/finetune-runs/create-finetune-run /openapi.yaml post /finetune-runs Start a new model finetuning job using a training dataset. Pass ``?force=true`` to bypass the pre-flight VRAM estimator; otherwise a config predicted to OOM returns 409 with a `recommendation` payload. # Deploy finetune run Source: https://docs.trymaitai.com/api-reference/finetune-runs/deploy-finetune-run /openapi.yaml post /finetune-runs/{run_id}/deploy Deploy a completed finetune run as a hosted model. Promotes the finetune into a servable model addressed by `model_name`: - a new `model_name` publishes the finetune as a new model; - an existing model's `model_name` with `promote_existing: true` swaps that model's underlying finetune to this run while keeping the same model reference (so callers targeting that model name pick up the new weights). Deployment is asynchronous: this returns immediately with the run in a `SCHEDULED` deployment state, poll `GET /finetune-runs/{run_id}/status` until `phase` reaches `deployed` (or `deploy_failed`/`deploy_cancelled`). Once deployed, run a test set against it by creating a test run whose `config.model` is this `model_name`. Hosted-model deployment is billed, so an account over its plan limit or with billing blocked returns 402. # Estimate finetune vram Source: https://docs.trymaitai.com/api-reference/finetune-runs/estimate-finetune-vram /openapi.yaml post /finetune-runs/estimate Run the pre-flight VRAM estimator against a proposed finetune config. # Get finetune run Source: https://docs.trymaitai.com/api-reference/finetune-runs/get-finetune-run /openapi.yaml get /finetune-runs/{run_id} Retrieve a finetune run with its configuration and current status. # Get finetune run metrics Source: https://docs.trymaitai.com/api-reference/finetune-runs/get-finetune-run-metrics /openapi.yaml get /finetune-runs/{run_id}/metrics Retrieve training loss and accuracy metrics for a finetune run. # Get finetune status Source: https://docs.trymaitai.com/api-reference/finetune-runs/get-finetune-status /openapi.yaml get /finetune-runs/{run_id}/status Get the current status of a finetune run. Reports a single `phase`: one of `training`, `trained`, `testing`, `deploying`, `deployed`, `deploy_failed`, `deploy_cancelled`, `failed`, `cancelled`, `deleted`: that folds together the training lifecycle, any hosted-model deployment, and any auto-test run. The component fields (`training_status`, `deployment_status`, `auto_test_status`, `progress`, and the deployed `model_id`) are included for detail. # List finetune runs Source: https://docs.trymaitai.com/api-reference/finetune-runs/list-finetune-runs /openapi.yaml get /finetune-runs List all finetune runs for your company. Optionally filter by `status`. # Help query Source: https://docs.trymaitai.com/api-reference/help/help-query /openapi.yaml post /help Answer a free-form question about building with Maitai in natural-language markdown. The information-only sibling of ``POST /search``: instead of a structured execution plan, returns an expert prose answer with working code examples (SDK integration, Developer API, CLI, or Portal) for agent IDEs (Cursor, Claude Code) to apply directly. Never takes action. # Get intent group Source: https://docs.trymaitai.com/api-reference/intent-groups/get-intent-group /openapi.yaml get /intent-groups/{intent_group_id} Retrieve a single intent group by ID. Use `detailed=true` (default) for full details. # Get intent group config Source: https://docs.trymaitai.com/api-reference/intent-groups/get-intent-group-config /openapi.yaml get /intent-groups/{intent_group_id}/config Retrieve the configuration for an intent group. # List intent group compositions Source: https://docs.trymaitai.com/api-reference/intent-groups/list-intent-group-compositions /openapi.yaml get /intent-groups/{intent_group_id}/compositions List all dataset compositions for an intent group. # List intent group datasets Source: https://docs.trymaitai.com/api-reference/intent-groups/list-intent-group-datasets /openapi.yaml get /intent-groups/{intent_group_id}/datasets List all training datasets for an intent group. # List intent group models Source: https://docs.trymaitai.com/api-reference/intent-groups/list-intent-group-models /openapi.yaml get /intent-groups/{intent_group_id}/models List all models assigned to an intent group. # List intent group requests Source: https://docs.trymaitai.com/api-reference/intent-groups/list-intent-group-requests /openapi.yaml get /intent-groups/{intent_group_id}/requests List paginated requests for an intent group with optional date range and message filtering. # List intent group sentinels Source: https://docs.trymaitai.com/api-reference/intent-groups/list-intent-group-sentinels /openapi.yaml get /intent-groups/{intent_group_id}/sentinels List all sentinels attached to an intent group. # List intent group test sets Source: https://docs.trymaitai.com/api-reference/intent-groups/list-intent-group-test-sets /openapi.yaml get /intent-groups/{intent_group_id}/test-sets List all test sets associated with an intent group. # List intent groups Source: https://docs.trymaitai.com/api-reference/intent-groups/list-intent-groups /openapi.yaml get /intent-groups List all intent groups for your company with pagination. # List intents by group Source: https://docs.trymaitai.com/api-reference/intent-groups/list-intents-by-group /openapi.yaml get /intent-groups/{intent_group_id}/intents List all intents belonging to an intent group across applications. # Update intent group config Source: https://docs.trymaitai.com/api-reference/intent-groups/update-intent-group-config /openapi.yaml put /intent-groups/{intent_group_id}/config Replace the configuration for an intent group. # Create application intent Source: https://docs.trymaitai.com/api-reference/intents/create-application-intent /openapi.yaml post /applications/{application_id}/intents Create a new intent (application action) for an application. # Delete application intent Source: https://docs.trymaitai.com/api-reference/intents/delete-application-intent /openapi.yaml delete /applications/{application_id}/intents/{intent_id} Delete an intent (application action) from an application. # Disable intent notifications Source: https://docs.trymaitai.com/api-reference/intents/disable-intent-notifications /openapi.yaml put /applications/{application_id}/intents/{intent_id}/notifications/disable Disable fault notifications for an intent. # Enable intent notifications Source: https://docs.trymaitai.com/api-reference/intents/enable-intent-notifications /openapi.yaml put /applications/{application_id}/intents/{intent_id}/notifications/enable Enable fault notifications for an intent. # Get application intent Source: https://docs.trymaitai.com/api-reference/intents/get-application-intent /openapi.yaml get /applications/{application_id}/intents/{intent_id} Retrieve a single intent by ID within an application. # Get intent config Source: https://docs.trymaitai.com/api-reference/intents/get-intent-config /openapi.yaml get /applications/{application_id}/intents/{intent_id}/config Retrieve the configuration for a specific intent. # List application intents Source: https://docs.trymaitai.com/api-reference/intents/list-application-intents /openapi.yaml get /applications/{application_id}/intents List all intents (application actions) for an application with details. # Reset intent config Source: https://docs.trymaitai.com/api-reference/intents/reset-intent-config /openapi.yaml post /applications/{application_id}/intents/{intent_id}/config/reset Reset an intent's configuration to the application/default configuration. # Update intent config Source: https://docs.trymaitai.com/api-reference/intents/update-intent-config /openapi.yaml put /applications/{application_id}/intents/{intent_id}/config Replace the configuration for a specific intent. # Attach model inference pool Source: https://docs.trymaitai.com/api-reference/models/attach-model-inference-pool /openapi.yaml post /models/{model_id}/inference-pool/attach Attach an inference pool to a model. # Create base model Source: https://docs.trymaitai.com/api-reference/models/create-base-model /openapi.yaml post /models/base-model Create a Maitai base model registry entry. # Detach model inference pool Source: https://docs.trymaitai.com/api-reference/models/detach-model-inference-pool /openapi.yaml post /models/{model_id}/inference-pool/detach Detach an inference pool from a model. # Disable model Source: https://docs.trymaitai.com/api-reference/models/disable-model /openapi.yaml post /models/{model_id}/disable Disable a model so it is no longer used for inference. # Enable model Source: https://docs.trymaitai.com/api-reference/models/enable-model /openapi.yaml post /models/{model_id}/enable Re-enable a previously disabled model for inference. # Execute swap model ref Source: https://docs.trymaitai.com/api-reference/models/execute-swap-model-ref /openapi.yaml post /models/{model_id}/swap-model-ref/execute Execute a model ref replacement. # Get model Source: https://docs.trymaitai.com/api-reference/models/get-model /openapi.yaml get /models/{model_id} Get model details. # Get model inference pool Source: https://docs.trymaitai.com/api-reference/models/get-model-inference-pool /openapi.yaml get /models/{model_id}/inference-pool Get the inference pool attached to a model. # Get model inference pool debug Source: https://docs.trymaitai.com/api-reference/models/get-model-inference-pool-debug /openapi.yaml get /models/{model_id}/inference-pool/debug Get debug details for a model's inference pool. # Get model repoint options Source: https://docs.trymaitai.com/api-reference/models/get-model-repoint-options /openapi.yaml get /models/{model_id}/repoint-options Get repointing options for a model. # List available models Source: https://docs.trymaitai.com/api-reference/models/list-available-models /openapi.yaml get /models/available List all models available to your company, including base and finetuned models. # List model inference pool attach candidates Source: https://docs.trymaitai.com/api-reference/models/list-model-inference-pool-attach-candidates /openapi.yaml get /models/{model_id}/inference-pool/attach-candidates List inference pools that can be attached to a model. # List models Source: https://docs.trymaitai.com/api-reference/models/list-models /openapi.yaml get /models List enabled models for your company. Optionally filter by `model_type`. # List models by application Source: https://docs.trymaitai.com/api-reference/models/list-models-by-application /openapi.yaml get /models/applications/{application_id} List models for an application. # List models by intent Source: https://docs.trymaitai.com/api-reference/models/list-models-by-intent /openapi.yaml get /models/intents/{intent_id} List models for an intent. # List models by intent group Source: https://docs.trymaitai.com/api-reference/models/list-models-by-intent-group /openapi.yaml get /models/intent-groups/{intent_group_id} List models for an intent group. # Patch model registry Source: https://docs.trymaitai.com/api-reference/models/patch-model-registry /openapi.yaml patch /models/{model_id}/registry Patch model registry metadata. # Preview swap model ref Source: https://docs.trymaitai.com/api-reference/models/preview-swap-model-ref /openapi.yaml post /models/{model_id}/swap-model-ref/preview Preview replacing a model ref. # Repoint model Source: https://docs.trymaitai.com/api-reference/models/repoint-model /openapi.yaml post /models/{model_id}/repoint Repoint a model. # Activate monitor Source: https://docs.trymaitai.com/api-reference/monitors/activate-monitor /openapi.yaml post /monitors/{monitor_id}/activate Set status to ACTIVE so production sampling fires. Idempotent. # Create monitor Source: https://docs.trymaitai.com/api-reference/monitors/create-monitor /openapi.yaml post /monitors Create a reusable production monitor. New monitors land in `DRAFT`. # Create monitor target Source: https://docs.trymaitai.com/api-reference/monitors/create-monitor-target /openapi.yaml post /monitors/{monitor_id}/targets Attach the monitor to an intent / workflow / agent with a sample rate. # Delete monitor Source: https://docs.trymaitai.com/api-reference/monitors/delete-monitor /openapi.yaml delete /monitors/{monitor_id} Soft-delete a monitor (also detaches all targets). # Delete monitor release Source: https://docs.trymaitai.com/api-reference/monitors/delete-monitor-release /openapi.yaml delete /monitors/{monitor_id}/releases/{release_name} Delete a named release. # Delete monitor target Source: https://docs.trymaitai.com/api-reference/monitors/delete-monitor-target /openapi.yaml delete /monitors/{monitor_id}/targets/{monitor_target_id} Detach (soft-delete) a monitor target. # Disable monitor target Source: https://docs.trymaitai.com/api-reference/monitors/disable-monitor-target /openapi.yaml post /monitors/{monitor_id}/targets/{monitor_target_id}/disable Disable a target without detaching it (sampling stops). Idempotent. # Enable monitor target Source: https://docs.trymaitai.com/api-reference/monitors/enable-monitor-target /openapi.yaml post /monitors/{monitor_id}/targets/{monitor_target_id}/enable Re-enable a monitor target so sampling resumes for it. Idempotent. # Get default monitor version Source: https://docs.trymaitai.com/api-reference/monitors/get-default-monitor-version /openapi.yaml get /monitors/{monitor_id}/releases/default Return the version runtime would use right now (default release or latest). # Get monitor Source: https://docs.trymaitai.com/api-reference/monitors/get-monitor /openapi.yaml get /monitors/{monitor_id} Retrieve a monitor by ID. Use `include_targets=false` to skip target loading. # Get monitor activity Source: https://docs.trymaitai.com/api-reference/monitors/get-monitor-activity /openapi.yaml get /monitors/{monitor_id}/activity Time-bucketed monitor outcome counts (overall or per target). Query params: `monitor_target_id`, `bucket_seconds`, `start_seconds`, `end_seconds`. Defaults: 24h lookback, 1h buckets. # Get monitor metrics Source: https://docs.trymaitai.com/api-reference/monitors/get-monitor-metrics /openapi.yaml get /monitors/{monitor_id}/metrics Aggregated counts + rates for the monitor over a window. Query params: `monitor_target_id` (scope to one target), `start_seconds`, `end_seconds`. Defaults match the activity endpoint (24h lookback). # Get monitor metrics by target Source: https://docs.trymaitai.com/api-reference/monitors/get-monitor-metrics-by-target /openapi.yaml get /monitors/{monitor_id}/metrics/by-target Per-target metrics rollup (one row per attached target). Targets with zero activity in the window still appear (with zeroed counts) so the caller can compare against the configured sample rate. # Get monitor target metrics Source: https://docs.trymaitai.com/api-reference/monitors/get-monitor-target-metrics /openapi.yaml get /monitors/{monitor_id}/targets/{monitor_target_id}/metrics Aggregated counts + rates for a single attached target. # Get monitor target sparkline Source: https://docs.trymaitai.com/api-reference/monitors/get-monitor-target-sparkline /openapi.yaml get /monitors/{monitor_id}/targets/{monitor_target_id}/sparkline Per-target error rate sparkline (used by the At-a-Glance cards). # Get monitor version Source: https://docs.trymaitai.com/api-reference/monitors/get-monitor-version /openapi.yaml get /monitors/{monitor_id}/versions/{version_id} Retrieve a specific monitor version snapshot. # Get run global Source: https://docs.trymaitai.com/api-reference/monitors/get-run-global /openapi.yaml get /monitors/runs/{run_id} Drill into a single ``monitor_run`` (resolution trace, source ids, etc.). # Get run scoped Source: https://docs.trymaitai.com/api-reference/monitors/get-run-scoped /openapi.yaml get /monitors/{monitor_id}/runs/{run_id} Get a single run and verify it belongs to the given monitor. # Get target sample Source: https://docs.trymaitai.com/api-reference/monitors/get-target-sample /openapi.yaml get /monitors/sample-target Return the most recent production payload for a monitor target. Powers the wizard's "design against a real sample" UX (autocomplete JSON paths, preview the runner-input shape) and is also useful for agents asking "what does this intent / workflow / agent actually emit?" before they author resolution rules. The payload mirrors the shape the runtime sampler pushes at the runner, so paths the caller picks here resolve identically in production. When the target has no completed invocations yet, ``payload`` / ``payload_paths`` are returned as ``null`` / ``[]``. Required query params: ``target_type`` (intent|workflow|agent), ``target_id`` (the resource id). # List monitor releases Source: https://docs.trymaitai.com/api-reference/monitors/list-monitor-releases /openapi.yaml get /monitors/{monitor_id}/releases List named releases for a monitor (default release first). # List monitor targets Source: https://docs.trymaitai.com/api-reference/monitors/list-monitor-targets /openapi.yaml get /monitors/{monitor_id}/targets List targets attached to a monitor. # List monitor versions Source: https://docs.trymaitai.com/api-reference/monitors/list-monitor-versions /openapi.yaml get /monitors/{monitor_id}/versions List published versions for a monitor (newest first). # List monitors Source: https://docs.trymaitai.com/api-reference/monitors/list-monitors /openapi.yaml get /monitors List monitors for your company. Supports `search`, `status`, pagination. # List monitors by target Source: https://docs.trymaitai.com/api-reference/monitors/list-monitors-by-target /openapi.yaml get /monitors/by-target Find every active monitor attached to ``(target_type, target_id)``. Useful for agents asking "what monitors are watching workflow 123?" or "is anything monitoring this intent group?". Required query params: ``target_type``, ``target_id``. Optional: ``enabled_only=true``. # List runs by source Source: https://docs.trymaitai.com/api-reference/monitors/list-runs-by-source /openapi.yaml get /monitors/runs/by-source Reverse lookup: every monitor run that evaluated the given event. Required: ``source_type`` (request|workflow|agent), ``source_type_id``. Optional: ``limit`` (default 25, max 100). # List runs for monitor Source: https://docs.trymaitai.com/api-reference/monitors/list-runs-for-monitor /openapi.yaml get /monitors/{monitor_id}/runs List runs scoped to a single monitor. # List runs global Source: https://docs.trymaitai.com/api-reference/monitors/list-runs-global /openapi.yaml get /monitors/runs List monitor runs across every monitor in your company. All filters are optional. Useful for "show me failures from the last hour" style queries, pass ``outcome=failed&start_seconds=...``. # Pause monitor Source: https://docs.trymaitai.com/api-reference/monitors/pause-monitor /openapi.yaml post /monitors/{monitor_id}/pause Set status to PAUSED so production sampling stops. Idempotent. # Preview monitor Source: https://docs.trymaitai.com/api-reference/monitors/preview-monitor /openapi.yaml post /monitors/{monitor_id}/preview Run the monitor judge + resolution against an ad-hoc payload. Body is :class:`MonitorPreviewRequest`. Defaults to the working draft so agents iterating on ``monitor.config`` can see the verdict immediately. Pass ``use=version`` / ``use=release`` / ``use=adhoc`` to preview a different config. No ``monitor_run`` row is inserted, but the runner still creates a real chat completion / workflow run server-side (returned as ``runner_type_id``) so agents can drill in later if they need to. # Promote monitor release Source: https://docs.trymaitai.com/api-reference/monitors/promote-monitor-release /openapi.yaml post /monitors/{monitor_id}/releases/{release_name}/promote Make ``release_name`` the default release for the monitor. Convenience for agents, same as ``POST /releases`` with the existing name + ``is_default=true``, but doesn't require knowing the version. # Publish monitor version Source: https://docs.trymaitai.com/api-reference/monitors/publish-monitor-version /openapi.yaml post /monitors/{monitor_id}/versions Publish a new version of the monitor (snapshot + semver bump). # Update monitor Source: https://docs.trymaitai.com/api-reference/monitors/update-monitor /openapi.yaml patch /monitors/{monitor_id} Update name, description, status, runner config, resolution config, or meta. # Update monitor target Source: https://docs.trymaitai.com/api-reference/monitors/update-monitor-target /openapi.yaml patch /monitors/{monitor_id}/targets/{monitor_target_id} Update the attachment's sample rate, enabled flag, or meta bag. # Upsert monitor release Source: https://docs.trymaitai.com/api-reference/monitors/upsert-monitor-release /openapi.yaml put /monitors/{monitor_id}/releases Create or update a named release pointing at a specific monitor version. # API Reference Source: https://docs.trymaitai.com/api-reference/overview Programmatically manage Maitai resources using the REST API. The Maitai Platform API gives you programmatic access to the main product resources available in the [Portal](https://portal.trymaitai.com), applications, intents, agents, workflows, datasets, test sets, finetune runs, sentinels, analytics, and more. For details on covered and intentionally excluded surfaces, see [API Coverage](/api-reference/coverage). ## Base URL All endpoints are served from: ``` https://api.trymaitai.ai/api/v1 ``` ## Authentication Every request must include your Maitai API key in the `X-Maitai-Api-Key` header. ```bash theme={null} curl https://api.trymaitai.ai/api/v1/applications \ -H "X-Maitai-Api-Key: YOUR_MAITAI_API_KEY" ``` You can create API keys in the Portal under **Settings > API Keys**. ## Response format Most `/api/v1` responses follow a consistent envelope: ```json Success theme={null} { "data": { ... } } ``` ```json Paginated theme={null} { "data": [ ... ], "pagination": { "total": 42, "offset": 0, "limit": 25 } } ``` ```json Error theme={null} { "error": { "status": 401, "message": "Missing company_id" } } ``` Some specialized endpoints share backend handlers with the Portal and may return a resource-specific response shape. Check the endpoint reference for the exact schema. ## Pagination Paginated endpoints accept `offset` and `limit` query parameters: | Parameter | Default | Max | Description | | --------- | ------- | ---- | ----------------------- | | `offset` | 0 | none | Number of items to skip | | `limit` | 25 | 100 | Max items to return | ## Resources Use the sidebar to navigate to specific endpoints. Each resource section lists all available operations. Manage applications and their configuration. Application actions nested under applications. Cross-application intent grouping. Agents, actions, sub-agents, versions, releases, and routing. Workflow definitions, versions, releases, artifacts, runs, and workflow test runs. Evaluation watchers for monitoring model output quality. Classic (non-agent) chat sessions. Individual request/response pairs. Curated request sets for model training. Curated request sets for model evaluation. Test set execution and results. Model finetuning jobs. Evaluation runs and results. Available and finetuned models. Request volume, fault-rate analytics, reports, drift, and continuous learning. # List fallbacks Source: https://docs.trymaitai.com/api-reference/reports/list-fallbacks /openapi.yaml get /reports/fallbacks List recent fallbacks. # List fallbacks by intent Source: https://docs.trymaitai.com/api-reference/reports/list-fallbacks-by-intent /openapi.yaml get /reports/intents/{intent_id}/fallbacks List recent fallbacks for an intent. # List faults Source: https://docs.trymaitai.com/api-reference/reports/list-faults /openapi.yaml get /reports/faults List recent faults. # List faults by intent Source: https://docs.trymaitai.com/api-reference/reports/list-faults-by-intent /openapi.yaml get /reports/intents/{intent_id}/faults List recent faults for an intent. # Get request Source: https://docs.trymaitai.com/api-reference/requests/get-request /openapi.yaml get /requests/{request_id} Retrieve a single request and its corresponding response. # List requests Source: https://docs.trymaitai.com/api-reference/requests/list-requests /openapi.yaml get /requests List requests filtered by application_id + intent_id, with date range. # Update request response Source: https://docs.trymaitai.com/api-reference/requests/update-request-response /openapi.yaml put /requests/{request_id}/response Update/correct a response for a request. Send `message_content` (string) in the body. # Search Source: https://docs.trymaitai.com/api-reference/search/search /openapi.yaml post /search Return a recommended Maitai CLI/API execution plan for a free-form query. Useful for agent IDEs (Cursor, Claude Code) that want to translate a natural-language ask like "how do I import an OpenAI agent" into a concrete sequence of Maitai operations, known pitfalls, and related tools, the same shape Composio's `composio.search` returns for their toolkits. # Create sentinel Source: https://docs.trymaitai.com/api-reference/sentinels/create-sentinel /openapi.yaml post /sentinels Create a new sentinel to monitor model output quality for an intent group. # Delete sentinel Source: https://docs.trymaitai.com/api-reference/sentinels/delete-sentinel /openapi.yaml delete /sentinels/{sentinel_id} Permanently delete a sentinel. # Generate new sentinel Source: https://docs.trymaitai.com/api-reference/sentinels/generate-new-sentinel /openapi.yaml post /sentinels/generate/new Generate a new sentinel. # Generate sentinels Source: https://docs.trymaitai.com/api-reference/sentinels/generate-sentinels /openapi.yaml post /sentinels/generate Generate sentinels. # Get sentinel Source: https://docs.trymaitai.com/api-reference/sentinels/get-sentinel /openapi.yaml get /sentinels/{sentinel_id} Retrieve a single sentinel by ID. # List sentinels Source: https://docs.trymaitai.com/api-reference/sentinels/list-sentinels /openapi.yaml get /sentinels List all sentinels (evaluation watchers) for your company. # List sentinels by application Source: https://docs.trymaitai.com/api-reference/sentinels/list-sentinels-by-application /openapi.yaml get /sentinels/applications/{application_id} List sentinels for an application. # List sentinels by application human needs Source: https://docs.trymaitai.com/api-reference/sentinels/list-sentinels-by-application-human-needs /openapi.yaml get /sentinels/applications/{application_id}/human-needs List human-needs sentinels for an application. # List sentinels by intent group Source: https://docs.trymaitai.com/api-reference/sentinels/list-sentinels-by-intent-group /openapi.yaml get /sentinels/intent-groups/{intent_group_id} List sentinels for an intent group. # List sentinels by intent name Source: https://docs.trymaitai.com/api-reference/sentinels/list-sentinels-by-intent-name /openapi.yaml get /sentinels/intent-groups/by-name/{intent_name} List sentinels for an intent group name. # List simple sentinels Source: https://docs.trymaitai.com/api-reference/sentinels/list-simple-sentinels /openapi.yaml get /sentinels/simple List lightweight sentinel rows. # Regenerate sentinel Source: https://docs.trymaitai.com/api-reference/sentinels/regenerate-sentinel /openapi.yaml post /sentinels/{sentinel_id}/regenerate Regenerate a sentinel. # Regenerate sentinel correction Source: https://docs.trymaitai.com/api-reference/sentinels/regenerate-sentinel-correction /openapi.yaml post /sentinels/{sentinel_id}/regenerate/correction Regenerate a sentinel correction prompt. # Regenerate sentinel evaluation Source: https://docs.trymaitai.com/api-reference/sentinels/regenerate-sentinel-evaluation /openapi.yaml post /sentinels/{sentinel_id}/regenerate/evaluation Regenerate a sentinel evaluation prompt. # Regenerate sentinel qualification Source: https://docs.trymaitai.com/api-reference/sentinels/regenerate-sentinel-qualification /openapi.yaml post /sentinels/{sentinel_id}/regenerate/qualification Regenerate a sentinel qualification prompt. # Regenerate sentinels Source: https://docs.trymaitai.com/api-reference/sentinels/regenerate-sentinels /openapi.yaml post /sentinels/regenerate Regenerate multiple sentinels. # Update sentinel Source: https://docs.trymaitai.com/api-reference/sentinels/update-sentinel /openapi.yaml put /sentinels/{sentinel_id} Update a sentinel's name, severity, directives, or configuration. # Update sentinel directive Source: https://docs.trymaitai.com/api-reference/sentinels/update-sentinel-directive /openapi.yaml put /sentinels/{sentinel_id}/directive Update a sentinel directive. # Get session Source: https://docs.trymaitai.com/api-reference/sessions/get-session /openapi.yaml get /sessions/{session_id} Retrieve a single session by ID. Requires `application_id` query param. # Get session timeline Source: https://docs.trymaitai.com/api-reference/sessions/get-session-timeline /openapi.yaml get /sessions/{session_id}/timeline Retrieve the chronological request timeline for a session. Requires `application_id` query param. # List sessions Source: https://docs.trymaitai.com/api-reference/sessions/list-sessions /openapi.yaml get /sessions List classic (non-agent) chat sessions. Optionally filter by `application_id`. # Set session feedback Source: https://docs.trymaitai.com/api-reference/sessions/set-session-feedback /openapi.yaml post /sessions/{session_id}/feedback Submit feedback (rating, notes) for a session. # Cancel unified test run Source: https://docs.trymaitai.com/api-reference/unified-test-runs/cancel-unified-test-run /openapi.yaml post /unified-test-runs/{test_run_id}/cancel Stop an in-flight run and mark it FAILED. Fails any items still in ``CREATED``/``RUNNING`` and stamps ``error_message='Cancelled by user'``, then signals the conductor (in-proc asyncio task in dev, K8s Job in prod) to unwind. Returns ``400`` if the run has already terminated (``COMPLETED`` / ``FAILED``); callers should refresh and use ``DELETE`` instead if they want the run gone. # Create unified test run Source: https://docs.trymaitai.com/api-reference/unified-test-runs/create-unified-test-run /openapi.yaml post /unified-test-runs Create a run in ``status=CREATED``. Body: ``CreateUnifiedTestRunRequest``. Consumer config is validated against the Pydantic variant matching ``consumer_type`` at the service layer, so wire-shape mistakes surface as a clean 400. Insertion is atomic: no items are created here, callers hit ``/prepare`` next to snapshot the adapted inputs. # Delete unified test run Source: https://docs.trymaitai.com/api-reference/unified-test-runs/delete-unified-test-run /openapi.yaml delete /unified-test-runs/{test_run_id} Hard-delete the run and its items. ``unified_test_run_item.fk_utri_run`` is ``ON DELETE CASCADE``, so items are removed in the same statement. Test set items and their ``response_sub`` overrides are untouched. # Execute unified test run Source: https://docs.trymaitai.com/api-reference/unified-test-runs/execute-unified-test-run /openapi.yaml post /unified-test-runs/{test_run_id}/execute Schedule execution across MODEL / WORKFLOW / AGENT and return immediately. Requires ``status=RUNNING`` (i.e. ``/prepare`` was called). **Non-blocking**: the service validates state, resolves the API key when required, and schedules a conductor (in-proc asyncio task in dev, K8s Job named ``unified-test-run-{id}`` in prod). HTTP returns the run row while it is still processing. Callers poll ``GET /{id}/progress`` to watch counts advance and to observe the final transition to COMPLETED (or FAILED for setup errors like missing API key or missing workflow). Per-item failures do not abort the run, items are marked FAILED with an ``error_message`` and the run still lands in COMPLETED. # Get unified test run Source: https://docs.trymaitai.com/api-reference/unified-test-runs/get-unified-test-run /openapi.yaml get /unified-test-runs/{test_run_id} Fetch a single run row. # Get unified test run progress Source: https://docs.trymaitai.com/api-reference/unified-test-runs/get-unified-test-run-progress /openapi.yaml get /unified-test-runs/{test_run_id}/progress Aggregate counts + duration percentiles for a run. Returns:: { "test_run_id": int, "run_status": "CREATED" | "PREPARING_INPUTS" | "RUNNING" | ..., "total": int, # every item on the run "done": int, # COMPLETED + FAILED "matched": int, # matched == true after scoring "mismatched": int, # matched == false after scoring "no_expected": int,# matched IS NULL after scoring "errored": int, # status == 'FAILED' "percent": float, # done / total (0.0 when total == 0) "duration_ms": {"avg", "p50", "p90", "p95"} } Portal polls this to watch a running conductor advance, matches the shape the workflow-family ``/workflow-test-runs/{id}/progress`` endpoint returns so the UI can share progress components. # List unified test run items Source: https://docs.trymaitai.com/api-reference/unified-test-runs/list-unified-test-run-items /openapi.yaml get /unified-test-runs/{test_run_id}/items List all items on a run, in insertion order. Not paginated: run items are bounded by test-set size (the same set that scored `run.test_set_id`), and every UI that consumes this endpoint needs all of them at once for the results table. # List unified test runs Source: https://docs.trymaitai.com/api-reference/unified-test-runs/list-unified-test-runs /openapi.yaml get /unified-test-runs List runs for this company, newest first. Query params: * ``test_set_id`` (int, optional), narrow to runs against one set. * ``application_id`` (int, optional), narrow to runs whose set belongs to one application. * ``offset`` / ``limit``, standard pagination. # Prepare unified test run Source: https://docs.trymaitai.com/api-reference/unified-test-runs/prepare-unified-test-run /openapi.yaml post /unified-test-runs/{test_run_id}/prepare Snapshot adapted inputs for every item and move to RUNNING. Walks every item on the set, calls the input adapter registered for the run's ``consumer_type``, and inserts a run-item row with the adapted payload. Idempotent-ish: rejects any status other than CREATED with a 400 so an accidental double-call from the UI can't duplicate rows. # Preview adapted input Source: https://docs.trymaitai.com/api-reference/unified-test-runs/preview-adapted-input /openapi.yaml post /unified-test-runs/preview-adapted-input Dry-run the input adapter for one item + hypothetical consumer config. Body: * ``test_set_id`` (int, required) * ``test_set_item_id`` (int, required) * ``consumer_type`` (str, required), MODEL / WORKFLOW / AGENT * ``consumer_id`` (int, optional) * ``consumer_config`` (object, optional) Same validation as ``/`` create (so a successful preview means the create+prep path will accept the same config), but nothing is persisted. Returns the adapted payload the item would carry into a run. # Reset failed unified test run items Source: https://docs.trymaitai.com/api-reference/unified-test-runs/reset-failed-unified-test-run-items /openapi.yaml post /unified-test-runs/{test_run_id}/reset-failed Reset items on a terminal run so it can be re-executed. Default mode (``?include_completed=false`` or body ``{"include_completed": false}``) resets only the ``FAILED`` items, a "rerun failed only" operation that preserves successful items' outcomes. Full-rerun mode (``include_completed=true``) also resets ``COMPLETED`` items, mirroring the legacy Workflow Retry action. Item PKs are preserved either way. Returns the reloaded run. # Score unified test run Source: https://docs.trymaitai.com/api-reference/unified-test-runs/score-unified-test-run /openapi.yaml post /unified-test-runs/{test_run_id}/score Kick off a background scoring pass, **non-blocking**. Requires ``status=COMPLETED``. Transitions the run to ``status=SCORING`` and returns the updated ``UnifiedTestRun`` row immediately; the actual structural-diff + evaluation-criteria pass runs as an ``asyncio`` task in the API process. Poll ``GET /{id}/progress`` (or refetch the run) until ``status`` flips back to ``COMPLETED``, at that point the aggregate scoring result lives at ``run.meta["score_summary"]`` (same ``UnifiedTestRunScoreSummary`` shape this endpoint used to return inline). Idempotent-ish: safe to re-invoke on a COMPLETED run to force a re-score (e.g. after adding a ``response_sub`` override). Returns ``400`` if the run is already in ``SCORING``, wait for it to settle and try again. # Clone unified test set Source: https://docs.trymaitai.com/api-reference/unified-test-sets/clone-unified-test-set /openapi.yaml post /unified-test-sets/{test_set_id}/clone Full-clone a unified test set. Returns a job id (async). Body: * ``new_test_set_name`` (str, required) -- must be unique per ``(company_id, application_id)``. * ``new_test_set_description`` (str, optional) -- defaults to source's description. Response: ``202 Accepted`` with ``{job_id, status: "pending"}``. The portal subscribes to the ``TEST_SET_JOB_PROGRESS`` websocket keyed on ``job_id`` and polls ``GET /api/v1/test-set-jobs/{job_id}`` as fallback. On completion the job's ``result`` is the newly-cloned ``TestSet`` payload -- same shape ``GET /api/v1/unified-test-sets/{id}`` returns. Because the unified surface stores item payload inline, the clone is one SQL round-trip per phase (set + items+response_subs). Sample cloning isn't offered for the unified family yet -- add a ``sample_mode`` branch here when it lands. # Create unified manual test set item Source: https://docs.trymaitai.com/api-reference/unified-test-sets/create-unified-manual-test-set-item /openapi.yaml post /unified-test-sets/{test_set_id}/items Insert a hand-authored (MANUAL) item. Body: ``CreateManualTestSetItemRequest``, ``input`` is required and must validate against ``CanonicalInput`` (i.e. the canonical chat-completion params shape). ``expected_output``, if present, must be a ``CanonicalAssistantMessage``. Source-import items go through the ``/import-*`` routes above and never hit this handler. # Create unified test set Source: https://docs.trymaitai.com/api-reference/unified-test-sets/create-unified-test-set /openapi.yaml post /unified-test-sets Create an empty unified test set. Body: ``CreateTestSetRequest``, ``test_set_name`` and ``application_id`` are required. The unique constraint on ``(company_id, application_id, test_set_name)`` is surfaced as a 400 by the service (``ValueError``) so callers get a clear duplicate message. # Delete unified test set item Source: https://docs.trymaitai.com/api-reference/unified-test-sets/delete-unified-test-set-item /openapi.yaml delete /unified-test-sets/{test_set_id}/items/{item_id} Delete a single item. Any ``test_set_item_response_sub`` override on the item cascades via FK, and the item is dropped from future runs. Existing ``unified_test_run_item`` rows are unaffected (they FK to the item id but were snapshotted at prepare-time). # Get unified test set Source: https://docs.trymaitai.com/api-reference/unified-test-sets/get-unified-test-set /openapi.yaml get /unified-test-sets/{test_set_id} Fetch a single set (with ``item_count`` populated). # Get unified test set item Source: https://docs.trymaitai.com/api-reference/unified-test-sets/get-unified-test-set-item /openapi.yaml get /unified-test-sets/{test_set_id}/items/{item_id} Fetch a single item, scoped to the set + company. # Import agent task into unified test set Source: https://docs.trymaitai.com/api-reference/unified-test-sets/import-agent-task-into-unified-test-set /openapi.yaml post /unified-test-sets/{test_set_id}/import-agent-task Import a completed agent task into this test set. Body: * ``agent_task_id`` (str, required), the top-level agent task's UUID * ``flatten_variant`` (str, optional), one of ``agent_first_user_message`` (default; drops system prompt) or ``agent_first_user_message_with_system_prompt``. Same variants the AGENT-consumer executor accepts via ``consumer_config``. * ``meta`` (object, optional) Idempotent via ``idx_test_set_item_unique_agent_task``. # Import request into unified test set Source: https://docs.trymaitai.com/api-reference/unified-test-sets/import-request-into-unified-test-set /openapi.yaml post /unified-test-sets/{test_set_id}/import-request Import a production `chat_completion_request` into this test set. Body: * ``request_id`` (str, required), the `chat_completion_request` id * ``meta`` (object, optional), free-form provenance for the item Idempotent at the service layer via the ``idx_test_set_item_unique_request`` partial unique index; re-posting the same request_id returns the existing item, not an error. # Import workflow run into unified test set Source: https://docs.trymaitai.com/api-reference/unified-test-sets/import-workflow-run-into-unified-test-set /openapi.yaml post /unified-test-sets/{test_set_id}/import-workflow-run Import a `workflow_run` into this test set. Body: * ``workflow_run_id`` (int, required) * ``meta`` (object, optional) Idempotent via ``idx_test_set_item_unique_workflow_run``. # List unified test set items Source: https://docs.trymaitai.com/api-reference/unified-test-sets/list-unified-test-set-items /openapi.yaml get /unified-test-sets/{test_set_id}/items List items on a set, in insertion order. Query params: ``offset`` / ``limit`` (standard pagination). # List unified test sets Source: https://docs.trymaitai.com/api-reference/unified-test-sets/list-unified-test-sets /openapi.yaml get /unified-test-sets List unified test sets for this company, newest first. Runs the lazy legacy-shell backfill on every call (idempotent), so Intent Group / company lists see both native unified sets and pending shells for still-unmigrated legacy rows. Query params: * ``application_id`` (int, optional), narrow to one application. * ``intent_group_id`` (int, optional), native sets with ``meta.intent_group_id`` plus legacy REQUEST shells whose source row belongs to that intent group. * ``offset`` / ``limit``, standard pagination (defaults 0/25, limit capped at 100 by ``parse_pagination``). # Patch unified test set Source: https://docs.trymaitai.com/api-reference/unified-test-sets/patch-unified-test-set /openapi.yaml patch /unified-test-sets/{test_set_id} Patch mutable metadata (name / description / is_golden). Body: ``UpdateTestSetRequest``, every field is optional. The unique ``(company_id, application_id, test_set_name)`` constraint is surfaced as a 400 via ``ValueError``. # Upload unified test set jsonl Source: https://docs.trymaitai.com/api-reference/unified-test-sets/upload-unified-test-set-jsonl /openapi.yaml post /unified-test-sets/upload Create a unified test set from a typed JSONL upload (non-blocking). Multipart form fields: * ``file`` (required), ``.jsonl`` with one object per line; each line must include ``type`` in ``request|workflow|agent|manual`` * ``name`` (required) * ``application_id`` (required) * ``description`` (optional) * ``is_golden`` (optional, default false) Returns the new test set immediately with ``meta.import_status=IMPORT_PENDING``. A background job parses the file and imports lines via the unified import helpers. # Upsert unified test set item expected output override Source: https://docs.trymaitai.com/api-reference/unified-test-sets/upsert-unified-test-set-item-expected-output-override /openapi.yaml put /unified-test-sets/{test_set_id}/items/{item_id}/expected-output-override Insert or replace this item's `expected_output` override. Body must validate against ``UpdateTestSetItemExpectedOutputRequest``: a single ``output_content`` field carrying a canonical assistant message. When present, scoring reads from the override table instead of the item's own ``expected_output``, the same behavior the legacy request- and workflow- families provided via their ``_response_sub`` tables. # Check workflow transforms Source: https://docs.trymaitai.com/api-reference/workflows/check-workflow-transforms /openapi.yaml get /workflows/{workflow_id}/has-transforms Check if a workflow script defines transform_session. # Clear workflow default release Source: https://docs.trymaitai.com/api-reference/workflows/clear-workflow-default-release /openapi.yaml post /workflows/{workflow_id}/releases/clear-default Clear the workflow default release pointer. # Create workflow Source: https://docs.trymaitai.com/api-reference/workflows/create-workflow /openapi.yaml post /workflows Create a workflow and upload its script. # Create workflow chat completion Source: https://docs.trymaitai.com/api-reference/workflows/create-workflow-chat-completion /openapi.yaml post /workflows/chat/completions Execute a workflow using the chat-completions contract. # Create workflow release Source: https://docs.trymaitai.com/api-reference/workflows/create-workflow-release /openapi.yaml post /workflows/{workflow_id}/releases Create a named workflow release pointer. # Delete workflow Source: https://docs.trymaitai.com/api-reference/workflows/delete-workflow /openapi.yaml delete /workflows/{workflow_id} Soft-delete a workflow. # Delete workflow accessory Source: https://docs.trymaitai.com/api-reference/workflows/delete-workflow-accessory /openapi.yaml delete /workflows/{workflow_id}/accessories Delete a workflow accessory file by key. # Delete workflow datastore Source: https://docs.trymaitai.com/api-reference/workflows/delete-workflow-datastore /openapi.yaml delete /workflows/{workflow_id}/datastores/{name} Delete a workflow datastore schema. # Delete workflow release Source: https://docs.trymaitai.com/api-reference/workflows/delete-workflow-release /openapi.yaml delete /workflows/{workflow_id}/releases/{name} Delete a named workflow release pointer. # Get default workflow script Source: https://docs.trymaitai.com/api-reference/workflows/get-default-workflow-script /openapi.yaml get /workflows/default-script Fetch the default workflow script template. # Get workflow Source: https://docs.trymaitai.com/api-reference/workflows/get-workflow /openapi.yaml get /workflows/{workflow_id} Retrieve a workflow by ID. # Get workflow artifact files Source: https://docs.trymaitai.com/api-reference/workflows/get-workflow-artifact-files /openapi.yaml get /workflows/{workflow_id}/artifact-files List deployed workflow script and accessory artifacts. # Get workflow run Source: https://docs.trymaitai.com/api-reference/workflows/get-workflow-run /openapi.yaml get /workflows/runs/{session_id} Fetch a single workflow run by session ID. # Get workflow schema Source: https://docs.trymaitai.com/api-reference/workflows/get-workflow-schema /openapi.yaml get /workflows/{workflow_id}/schema Fetch workflow input schema and transform availability. # Get workflow script Source: https://docs.trymaitai.com/api-reference/workflows/get-workflow-script /openapi.yaml get /workflows/{workflow_id}/script Fetch a workflow script decomposed into code, schemas, and transform. # Get workflow transform info Source: https://docs.trymaitai.com/api-reference/workflows/get-workflow-transform-info /openapi.yaml get /workflows/{workflow_id}/transform-info Fetch workflow transform_session availability. # Get workflow version artifact files Source: https://docs.trymaitai.com/api-reference/workflows/get-workflow-version-artifact-files /openapi.yaml get /workflows/{workflow_id}/versions/{version_number}/artifact-files List artifact files frozen in a workflow version. # List workflow datastores Source: https://docs.trymaitai.com/api-reference/workflows/list-workflow-datastores /openapi.yaml get /workflows/{workflow_id}/datastores List workflow datastores that can provide data to executions. # List workflow releases Source: https://docs.trymaitai.com/api-reference/workflows/list-workflow-releases /openapi.yaml get /workflows/{workflow_id}/releases List workflow release pointers. # List workflow runs Source: https://docs.trymaitai.com/api-reference/workflows/list-workflow-runs /openapi.yaml get /workflows/runs List workflow runs by workflow_id or application_id. # List workflow versions Source: https://docs.trymaitai.com/api-reference/workflows/list-workflow-versions /openapi.yaml get /workflows/{workflow_id}/versions List published workflow versions. # List workflows Source: https://docs.trymaitai.com/api-reference/workflows/list-workflows /openapi.yaml get /workflows List workflows available to your company. # Log workflow run Source: https://docs.trymaitai.com/api-reference/workflows/log-workflow-run /openapi.yaml put /workflows/runs/log Log an externally executed (bring-your-own) workflow run. # Preview workflow transform Source: https://docs.trymaitai.com/api-reference/workflows/preview-workflow-transform /openapi.yaml post /workflows/{workflow_id}/transform-preview Dry-run transform_session for one session without persisting output. # Publish workflow version Source: https://docs.trymaitai.com/api-reference/workflows/publish-workflow-version /openapi.yaml post /workflows/{workflow_id}/versions Publish a new immutable workflow version. # Update workflow Source: https://docs.trymaitai.com/api-reference/workflows/update-workflow /openapi.yaml put /workflows/{workflow_id} Update workflow metadata, script, schemas, or transform code. # Update workflow release Source: https://docs.trymaitai.com/api-reference/workflows/update-workflow-release /openapi.yaml put /workflows/{workflow_id}/releases/{name} Update a named workflow release pointer. # Upload workflow accessory Source: https://docs.trymaitai.com/api-reference/workflows/upload-workflow-accessory /openapi.yaml post /workflows/{workflow_id}/accessories Upload or replace a workflow accessory file. # Upload workflow bundle Source: https://docs.trymaitai.com/api-reference/workflows/upload-workflow-bundle /openapi.yaml post /workflows/upload Upload or upsert a workflow bundle (script, accessories, optional version publish). # Upload workflow datastore Source: https://docs.trymaitai.com/api-reference/workflows/upload-workflow-datastore /openapi.yaml post /workflows/{workflow_id}/datastores/{name}/upload Upload records or link an S3 URI for a workflow datastore. # Upsert workflow datastore Source: https://docs.trymaitai.com/api-reference/workflows/upsert-workflow-datastore /openapi.yaml post /workflows/{workflow_id}/datastores/{name} Create or update a workflow datastore schema. # Validate workflow script Source: https://docs.trymaitai.com/api-reference/workflows/validate-workflow-script /openapi.yaml post /workflows/validate Validate workflow code, transform code, or schema snippets. # Agent Flow Steps Source: https://docs.trymaitai.com/build/agents/agent_flow The reasoning loop, step by step, and how to configure each one Agents in `reasoning` mode process every request through a pipeline of **flow steps**. The entire pipeline is exposed on the agent's **Configuration** tab, there are no hidden system instructions. Each LLM-powered step is a card you can open, inspect, and customize: what you see in the pipeline is exactly what runs. ## The pipeline | Step | Required | What it does | | ------------------ | ------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Orchestrate** | Yes | Plans the next move, reviews the request, current [state](/build/agents/agent_state), and prior results, then picks one capability to run. | | **Action** | Yes | Where your capabilities execute (API calls, LLM calls, workflows, webhooks, sub-agents). Not an LLM step, manage these on the **Capabilities** tab. | | **Process** | Optional, on by default | Distills each action result into a rolling summary and findings, and writes state updates. Two sub-calls: **Summary & Findings** and **Form Updates**. Disable it to keep a raw message transcript instead. | | **Progress Probe** | Optional, off by default | After each result, checks whether the agent should continue, respond, or rewind. Adds a guardrail LLM call per iteration. | | **Respond** | Yes | Produces the final, user-facing answer once the loop decides the task is done. Freeform text by default; you can bake in a structured JSON schema on the step (**Response Format** in the step config), and callers can still override per request via `response_format` (see [Agent Call) structured final response](/sdk/agent_call#structured-final-response-response_format). | ## Configuring a step Click any step in the pipeline to open its configuration: * **System prompt**: every step ships with Maitai's default prompt template. Edit it to tailor the step to your agent; a badge on the step shows **Default** vs **Customized**, and you can reset back to the default at any time. * **Model & parameters**: each step runs as its own intent, so each can use its own model and hyperparameters. A common pattern: a fast, cheap model for **Orchestrate** and **Process**, and a premium model for **Respond**. Guidance that previously lived in "special instructions" now belongs directly in the step prompts: * Prioritization logic, checklists before calling tools go in **Orchestrate** * How to interpret and synthesize capability results goes in **Process** * Tone, formatting, and response policy go in **Respond** * Structured JSON answer shape (schema) goes in **Respond**, under **Response Format** Flow steps run through your Application like any other intent, which means each step shows up in [request monitoring](/observe/request_overview) and can be independently fine-tuned over time. ## Toggling optional steps **Process** and **Progress Probe** have switches directly on the pipeline. Turning Process off trades context quality for latency; turning Progress Probe on trades latency for tighter loop control. ## Per-request overrides (SDK) You can toggle flow steps for a single request without touching the published agent, for example, skipping the Progress Probe on a latency-sensitive call: ```python theme={null} agent = client.agent.get("YOUR_AGENT_NAME", version="prod") agent.config.system_steps.progress = False request = agent.to_request(messages=[...], session_id="YOUR_SESSION_ID") response = client.agent.completions.create(request=request) ``` The step keys are `orchestration`, `processing`, `progress`, and `respond`. Overrides are validated against the published version and only whitelisted toggles apply (see [Agent Call) per-request overlays](/sdk/agent_call#per-request-overlays-agent-get). ## Next * Define the fields your agent maintains: [Agent State](/build/agents/agent_state) * Full walkthrough: [Build an Agent](/examples/build_agent) # Agent State Source: https://docs.trymaitai.com/build/agents/agent_state Structured, session-scoped memory your agent maintains and acts on **Agent State** is a set of typed fields the agent is responsible for keeping up to date throughout a session. Define the fields on the agent's **State** tab in the Portal; the agent fills them in as it converses and runs capabilities. State acts as a structured scratchpad that persists across turns, so the agent never loses track of critical information, and it feeds directly into how the agent behaves: * **Orchestration**: in `reasoning` mode, the planner sees current state when deciding what to do next, and the **Process** step writes updates back after each action. * **Routing**: in `route` mode, rules-based routing can match on state values (e.g. route to the billing action once `order_id` is set). * **SDK**: read resolved state and seed values per request via `agent.get` (see below). ## Defining a field Each state field has: * **Name**: how the agent and your code reference the field (e.g. `order_id`). * **Description**: what the field holds and when to fill it. The agent relies on this, so be specific. * **Type**: `string`, `number`, `boolean`, or `date`. Values written from the SDK are validated against this type. * **Update Rules** (optional), extra guidance for when and how the agent should update the value. * **Read / Write Scope**: access control, below. ## Access control (ACLs) Each field carries its own ACL controlling which agents in the hierarchy can see or modify it, critical for data integrity in multi-agent workflows. * **Read Scope**: who can view the field's value. * **Write Scope**: who can update it. Write access requires read access. Both scopes support three levels: 1. **Private** (default): only the agent that defines the field can access it. Best for internal session state that sub-agents shouldn't see. 2. **Public**: any agent in the session hierarchy can access the field. Useful for shared context like `user_id` or `session_language`. 3. **Custom**: explicitly select which sub-agents get access, a "need-to-know" architecture where data is only shared with the specialists that require it. ### Use cases * **Data isolation**: keep sensitive internal reasoning **Private**. * **Selective delegation**: share `payment_token` with only the "Billing" sub-agent via **Custom**, hidden from "General Support". * **Global context**: make essential metadata **Public** so every agent in the tree can use it. ## Seeding state from code You can read an agent's resolved state and seed field values for a single request through the SDK: ```python theme={null} agent = client.agent.get("YOUR_AGENT_NAME", version="prod") agent.state["user_facts"] = "VIP customer since 2021" request = agent.to_request(messages=[...], session_id="YOUR_SESSION_ID") response = client.agent.completions.create(request=request) ``` Patches are field-level and validated against the field types defined on the State tab: absent keys are untouched, and an explicit `None` clears a field. See [Agent Call, per-request overlays](/sdk/agent_call#per-request-overlays-agent-get). ## Next * Guide the steps that read and write state: [Agent Flow Steps](/build/agents/agent_flow) * Full walkthrough: [Build an Agent](/examples/build_agent) # API Action Source: https://docs.trymaitai.com/build/agents/capabilities/api_action Integrating external services into your agent workflows **API Actions** grant your agents the ability to interact with the real world by calling external services.
### Configuration * **Endpoint & Schema**: Define the HTTP method, URL, timeout, and a JSON schema for the action's parameters, including which are required. Use `{{parameter_name}}` placeholders for dynamic values in the URL path or query. * **Importing Specs**: Bootstrap quickly by uploading an **OpenAPI spec**: create a single action or a full set of API Actions in bulk. * **Usage Guidance**: **When to use** / **When not to use** hints that feed routing and orchestration decisions. Specific guidance here directly improves capability selection. * **Authentication**: Store API keys and headers in Maitai, or reference per-request secrets (below). * **Response Handling**: Make **Real** API calls or return **Mock** responses for testing and prototyping. ## Invocation Mode Every action is either **Foreground** or **Background**. What that means in practice depends on the agent's [execution mode](/build/agents/execution_mode): | | Foreground | Background | | -------------------- | ------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **`route` mode** | Eligible to be the routed capability, its output becomes the agent's response. | Never the primary route. The router may select **one background action** to run in parallel with the routed capability, purely to enrich state/context for later turns. | | **`reasoning` mode** | Results feed the loop and can be surfaced in the agent's final response. | Context-only: results are stored for other actions and sub-agents to use, but never shown directly to the user. | *Example*: a background `get_loan_options` action gathers data silently, which a "Loan Advisor" sub-agent then uses to formulate a tailored recommendation. **Webhook Creation** actions are always foreground, they pause execution and need to drive the session, so they can't run as background enrichment. ## Per-request secrets Instead of storing credentials in Maitai, you can reference them by name and supply the value at request time. In the action's **auth** or **headers** config, use a `{{secrets.NAME}}` placeholder: ```text theme={null} Authorization: Bearer {{secrets.stripe_key}} ``` Then pass the value when calling the agent: ```python theme={null} agent = client.agent.get("YOUR_AGENT_NAME", version="prod") request = agent.to_request( messages=[...], session_id="YOUR_SESSION_ID", secrets={"stripe_key": "sk_live_..."}, ) ``` Secrets are validated up front (a `400` lists any missing names), resolved only inside auth/headers, never exposed to the LLM, and redacted before any request record is persisted. Sub-agents invoked during the run can resolve the same secrets. Full details: [Agent Call, per-request secrets](/sdk/agent_call#per-request-secrets). ## Per-request control (SDK) Beyond secrets, the SDK can adjust how a published agent uses its API Actions for a single request, without changing the agent definition: * **Capability mask**: subtractively disable an action (or a whole sub-agent) for one request via `agent.disable_action("name")`. You can only disable capabilities the published version exposes, never enable disabled ones. * **Config overlay**: toggle [flow steps](/build/agents/agent_flow) via `agent.config.system_steps.`. See [Agent Call, per-request overlays](/sdk/agent_call#per-request-overlays-agent-get). # LLM Action Source: https://docs.trymaitai.com/build/agents/capabilities/llm_action Specialized sub-tasks with unique prompts and model configs An **LLM Action** allows your agent to perform a specialized reasoning task using a separate prompt and model configuration.
### When to use LLM Actions Use LLM Actions for atomic tasks that require reasoning but don't necessarily need a full, stateful subagent. * **Summarization**: Condensing a long session history or a document. * **Translation**: Converting user input or agent output into a different language. * **Formatting**: Transforming unstructured text into a specific JSON schema or markdown format. ### Configuration You can define a unique system prompt, select a specific model (e.g., a smaller, faster model for simple tasks), and set hyperparameters specifically for this action. Like all actions, LLM Actions have an [Invocation Mode](/build/agents/capabilities/api_action#invocation-mode): **Foreground** output can stream back as part of the agent's response, while **Background** output only enriches session context and is never shown to the user. LLM Actions run through your Application like any other intent, each action shows up in [request monitoring](/observe/request_overview) and can be independently fine-tuned as you collect traffic. # Subagent Source: https://docs.trymaitai.com/build/agents/capabilities/subagent Recursive hierarchical delegation to specialized agents Maitai supports an **infinite recursive hierarchy**. A **Subagent** is another agent that is added as a capability to a "manager" agent. ### The Manager Pattern A high-level agent (e.g., "Customer Support Lead") can delegate complex tasks to specialized subagents (e.g., "Billing Specialist" or "Technical Troubleshooter"). ### Key Features * **Recursive Nature**: Subagents can themselves have their own subagents, allowing you to build deeply nested "Agent Trees." * **Shared Context**: Subagents inherit relevant context from their parents but operate within their own specialized prompts and capability sets. [State](/build/agents/agent_state) fields flow between them according to each field's read/write scope. * **Modular Design**: Build and test specialized agents independently, then compose them together into a powerful multi-agent system. ### Subagents and execution mode How a subagent is invoked depends on the parent's [execution mode](/build/agents/execution_mode): * In **`reasoning`** mode, the orchestrator delegates a scoped objective to the subagent inside the loop, then processes its result like any other action. * In **`route`** mode, a subagent can be the routing target, the original conversation is passed through, and the subagent's answer becomes the response. The subagent runs with its **own** execution mode, so a routing parent can delegate to a reasoning child and vice versa. For latency-sensitive deployments with deep hierarchies, the `high_performance` request flag flattens subagent capabilities into the root agent and runs everything in-process. See [Execution Mode, performance options](/build/agents/execution_mode#performance-options). # Supplied Actions Source: https://docs.trymaitai.com/build/agents/capabilities/supplied_actions Ready-made capabilities from the Maitai catalog you can add to any agent **Supplied Actions** are pre-built, Maitai-maintained capabilities you can add to any agent in one click, no configuration, no credentials. Adding one clones it onto your agent as a normal action you fully own. ## Current catalog | Action | Type | What it does | | ------------------------------------- | ---------- | ------------------------------------------------------------------------------------------------------------------------------------ | | **Web Search** (`web_search`) | API Action | Live web search via the Perplexity Search API, using Maitai-managed credentials, you never need your own API key | | **Scrape Webpage** (`scrape_webpage`) | API Action | Scrape any webpage via Firecrawl, using Maitai-managed credentials. Returns the page as markdown plus metadata | | **Ask Webpage** (`ask_webpage`) | API Action | Scrape a webpage via Firecrawl and answer a natural-language question about its content. Returns markdown, metadata, and an `answer` | | **Build Strategy** (`build_strategy`) | LLM Action | A curated strategy-building prompt with a tuned model configuration | The catalog grows over time, check the portal or API for the live list. ## Adding from the portal On your agent's page, open **Add Capability** and choose **Supplied by Maitai**. Pick an action and add it. A lock icon indicates the action uses Maitai-managed credentials. If your agent already has an action with the same name, you'll see an "already added" notice instead of a duplicate. ## Adding via API / CLI List the catalog: ```bash theme={null} maitai agents list-supplied-actions # GET /agents/supplied-actions ``` Each entry includes `id` (the catalog action id), `action_name`, `action_type`, `description`, and `uses_platform_credentials`. Clone one onto your agent: ```bash theme={null} maitai agents add-supplied-action # POST /agents/{agent_id}/actions/supplied/{catalog_action_id} ``` Returns the newly created action (`201`). A `409` means the agent already has an action with that name. ## After adding The clone is yours, it behaves like any other capability: * Edit the prompt, schema, or usage guidance; disable or delete it at will. * It lands on your agent's **draft**. Publish the action and a new agent version, then point a release at it, before it serves traffic. * The clone's `meta.supplied` block records which catalog entry (and version) it came from. Catalog entries themselves are read-only, adding always clones, so later edits to your copy never affect the catalog (and vice versa). ## Maitai-managed credentials Some supplied actions (like Web Search) call third-party APIs with Maitai-owned keys. These reference a `{{platform.NAME}}` token in their auth config: * The token resolves **server-side at execution time**: the raw key never appears in your action config, API responses, or the portal. * Tokens are **host-pinned**: the credential only resolves when the request targets the intended provider. Re-pointing the action at another host fails the call rather than exposing the key. * You never supply these values, unlike [per-request secrets](/build/agents/capabilities/api_action#per-request-secrets), there's nothing to pass at request time. To call a different provider with your own credentials, build a regular [API Action](/build/agents/capabilities/api_action) instead. # Webhook Source: https://docs.trymaitai.com/build/agents/capabilities/webhook Handling asynchronous callbacks and long-running tasks **Webhooks** are used for asynchronous workflows where the agent must wait for a callback from an external system before continuing. ### How it Works 1. **Trigger**: The agent runs a **Webhook Creation** capability. 2. **Register**: Maitai creates a unique, per-session `webhook_url` on the Maitai hooks service (e.g., `https://hooks.trymaitai.ai/agent/`). 3. **Wait**: The agent returns **WAITING\_WEBHOOK** and pauses. 4. **Callback**: Your external system POSTs an event payload to the `webhook_url`. 5. **Resume**: Maitai routes the payload to the correct agent + session and resumes execution. ### Notes * The `webhook_url` is a Maitai-hosted callback endpoint. The webhook key is used to look up which agent/session is waiting. * Maitai does **not** verify provider-specific signatures (e.g., Stripe’s `Stripe-Signature`) at webhook ingress. Verify signatures in your own webhook handler before forwarding to Maitai if needed. * Webhook Creation actions are always **foreground**: they pause the session, so they can't run as background enrichment. They work in both [execution modes](/build/agents/execution_mode): the reasoning loop pauses mid-task, and a routed webhook action pauses until the callback arrives. ### Use Cases * **Payment Processing**: Waiting for a transaction to clear. * **Human-in-the-Loop**: Waiting for a human agent to approve a specific action. * **Long-running Jobs**: Waiting for a video to finish rendering or a report to be generated. # Workflow Action Source: https://docs.trymaitai.com/build/agents/capabilities/workflow Run a Maitai Workflow as an agent capability A **Workflow Action** lets your agent invoke a Maitai Workflow (a versioned, multi-step script hosted on the platform) as a single capability. ### When to use Workflow Actions Use a Workflow Action when a task is too procedural for prompt-driven orchestration: deterministic multi-step logic, data transformations, or an existing workflow your team already maintains in **Forge > Workflows**. * **Deterministic pipelines**: The agent decides *when* to run the workflow; the workflow guarantees *how* the steps execute. * **Reuse**: The same workflow can back multiple agents, and be tested and versioned independently of any of them. ### Configuration 1. Add a capability with type **Maitai Workflow**. 2. Select the workflow to run (or enter a workflow reference directly). 3. Define the action's name, description, and parameter schema like any other action, the agent passes the arguments it plans as the workflow's input, and the workflow's output returns to the agent as the action result. Like all actions, Workflow Actions have an [Invocation Mode](/build/agents/capabilities/api_action#invocation-mode), run them in the background to enrich context, or in the foreground to drive the response. # Create Agent Source: https://docs.trymaitai.com/build/agents/create_agent Defining the identity, context, and execution mode of your AI Agent The first step in building a Maitai Agent is defining its identity: what it does, how it executes, and where it lives in your hierarchy. Go to **Forge > Agents** and click **Create Agent**. ### Basic Setup * **Name & Description**: Clearly identify the agent's purpose. The description does double duty, it helps you organize, and it's used as a routing signal whenever this agent is invoked as a [sub-agent](/build/agents/capabilities/subagent). * **Execution Mode**: How the agent processes each request, **Reasoning** (full planning loop, the default) or **Route** (single-step capability selection). This is part of the agent's identity, set here at build time; you can change it later from the agent's **Configuration** tab, but not per-request. See [Execution Mode](/build/agents/execution_mode). * **Application**: Associate the agent with an Application. This determines which context documents are available and where the agent's LLM traffic (including its [flow steps](/build/agents/agent_flow)) is monitored. * **Base URL (Optional)**: A default host applied to all of this agent's [API Actions](/build/agents/capabilities/api_action), so individual actions only need relative paths. Actions can override it when needed. ### Drafts, versions, and releases Agents are versioned. Edits in the Portal land on the agent's **draft**; when you're happy with it, **Publish** creates an immutable version (major / minor / patch, with release notes). You can label versions with named releases like `prod` and pin requests to a version or release from the SDK, unpinned requests run the current draft. Manage versions from the agent's **Versions** tab; see [Agent Call](/sdk/agent_call) for pinning from code. ## Next * Configure how it executes: [Execution Mode](/build/agents/execution_mode) * Add tools/capabilities: [LLM Action](/build/agents/capabilities/llm_action) · [API Action](/build/agents/capabilities/api_action) · [Subagent](/build/agents/capabilities/subagent) * Full walkthrough: [Build an Agent](/examples/build_agent) * Call your agent from code: [Agent Call](/sdk/agent_call) # Execution Mode Source: https://docs.trymaitai.com/build/agents/execution_mode How your agent processes each request: full reasoning loop or single-step routing Every agent has an **Execution Mode** that determines how it handles each request. You choose it when you [create the agent](/build/agents/create_agent), and you can change it at any time from the agent's **Configuration** tab in the Portal. Execution mode is part of the agent's identity. It is **not** set per-request. The `AgentRequest` no longer accepts an `execution_mode` parameter; if you were passing it from code, remove it and configure the mode on the agent instead. ## Reasoning: multi-step loop (default) The agent runs the full reasoning loop, powered by Cortex, Maitai's event-driven orchestration engine: 1. **Orchestrate**: an LLM planner reviews the request, the agent's [state](/build/agents/agent_state), and the available capabilities, then picks the next action. 2. **Act**: the chosen capability runs: an API call, LLM call, workflow, webhook registration, or sub-agent delegation. 3. **Process**: the result is distilled into a rolling summary, findings, and state updates. 4. **Repeat or respond**: the loop continues until the agent has what it needs, then it composes the final response. Every LLM step in this loop is visible and configurable on the Configuration tab, see [Agent Flow Steps](/build/agents/agent_flow). Choose `reasoning` when: * Requests take **multiple actions** to complete (gather data, act on it, synthesize an answer) * The agent **delegates to sub-agents** * The agent needs to **pause on webhooks** and resume when an external system calls back * Task completion matters more than minimal latency ## Route: single-step routing The agent sends each request **directly to a single capability** and returns that capability's result. There is no planning loop: one routing decision, one action, one response. To keep responses fast without giving up context gathering, the router can also select **one background action** to run in parallel with the routed capability. Background results are never shown to the user; they enrich session state for later turns. See [Invocation Mode](/build/agents/capabilities/api_action#invocation-mode). Choose `route` when: * Each request maps cleanly to one capability * **Latency matters**: route mode skips the orchestration loop entirely and streams the routed capability's output straight back * You want **predictable, auditable behavior**, especially with rules-based routing ## Choosing a mode | | `reasoning` | `route` | | -------------------- | ------------------------------- | -------------------------------------- | | Capability selection | Iterative LLM planning | One-shot: rules or LLM | | Actions per request | As many as the task needs | One foreground (+ optional background) | | Latency | Higher, multiple LLM steps | Lowest | | Sub-agents | Full delegation inside the loop | Can be the routing target | | Webhook waits | Supported | Supported | | Best for | Complex, multi-step tasks | High-volume request-to-action mapping | ## Routing strategy (route mode) When an agent is in `route` mode, the Configuration tab exposes **how** the routing decision is made: * **LLM Routing** (default): a routing model selects the best capability using the conversation, the agent's current state, and each capability's name, description, and usage guidance. Well-written descriptions and **When to use / When not to use** guidance directly improve routing accuracy. * **Rules-Based Routing**: an ordered list of deterministic rules, first match wins. * Conditions evaluate the **request** content or **state** fields. * Operators include equality and numeric comparisons, pattern matching (`like`, regex), and **semantic matching** against a reference phrase. * Each rule routes to a specific **action**, a **sub-agent**, or falls through to **LLM routing**. * A **default route** handles requests that match no rule. ## Flow configuration (reasoning mode) When the agent is in `reasoning` mode, the Configuration tab shows the **Agent Flow** pipeline instead: every step of the loop, each with its own prompt and model configuration. See [Agent Flow Steps](/build/agents/agent_flow). ## Performance options These optional request-time flags tune how an agent executes without changing what it can do. See [Agent Call](/sdk/agent_call) for the full parameter reference. | Parameter | Default | Description | | ------------------ | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `high_performance` | `false` | Flattens sub-agent capabilities into the root agent and runs everything in-process. Eliminates inter-process overhead for agents with sub-agent hierarchies. | | `max_iterations` | `100` | Caps reasoning-loop iterations. A safety limit for runaway loops in `reasoning` mode. | ## Next * Configure the reasoning pipeline: [Agent Flow Steps](/build/agents/agent_flow) * Define structured session state: [Agent State](/build/agents/agent_state) * Full walkthrough: [Build an Agent](/examples/build_agent) # Import from code Source: https://docs.trymaitai.com/build/agents/import_from_code Translate your existing Python agent code into Maitai Agents: any framework, or none at all If you already have agents written in Python (using a framework like the OpenAI Agents SDK, LangChain/LangGraph, CrewAI, AutoGen, Pydantic AI, or LlamaIndex, or hand-written on top of a raw provider SDK) you don't need to redefine them in the Portal. Point the Maitai CLI at your source and it will translate the code into Maitai [Agents](/build/agents/create_agent), [capabilities](/build/agents/capabilities/llm_action), and [sub-agent](/build/agents/capabilities/subagent) links. The CLI ships your code to Maitai, which uses Mojo (the same translator that powers Maitai's natural-language tooling) to read the code and produce a canonical agent manifest. The manifest is then applied to your account. ## Quickstart ```bash theme={null} maitai agents import path/to/your_agent.py \ --application-name "My App" --create-application ``` That single invocation: 1. Reads your source file (or every `.py` file in a directory). 2. Sends it to Maitai for translation. 3. Creates the agents, attaches their tools as capabilities, and wires sub-agent handoffs. 4. Prints a summary including the agent IDs you can open in the Portal. ## Single file vs. directory You can point `agents import` at either a single `.py` file or a directory: * **Single file**: best for small agents that live in one module. The entire file is shipped as the source. * **Directory**: best for real projects (CrewAI crews, LangGraph subgraphs, custom agent packages). Every `.py` file in the directory tree is concatenated and shipped as one project, with file separators preserved so the translator knows which code came from which file. For directory imports, everything in the directory is treated as part of the same agent project. If your repo has unrelated code mixed in, point at a smaller subdirectory containing just the agent files. ```bash theme={null} maitai agents import path/to/your_crewai_project/ \ --application-name "Marketing Crew" --create-application ``` The total concatenated source is capped at 1 MB. If you hit that, prune non-agent code (tests, generated fixtures, vendored deps) or point at a smaller subdirectory. ## What gets imported Maitai's translator looks for any unit of code that combines (1) a system prompt or instruction string conditioning an LLM with (2) an LLM call. Tools and handoffs are optional. That means: * **Framework agents** (`Agent(name=..., instructions=..., tools=[...])` and similar) are recognized directly. * **Hand-written loops** that build a messages array with a system prompt and call `client.chat.completions.create(...)` are also recognized. * **Tools / functions** that the agent invokes become [API Actions](/build/agents/capabilities/api_action) or [LLM Actions](/build/agents/capabilities/llm_action) depending on what they do. * **Handoffs** between agents, whether explicit (`handoff()`), task-pipeline (CrewAI), or implicit (one agent invoking another as a tool), are translated into [sub-agent](/build/agents/capabilities/subagent) links. When the translator recognizes a known framework (CrewAI, LangGraph, Pydantic AI, etc.) it tags the import with `Source recognized as: `. When it doesn't recognize anything, it still imports based on the code's structure, the translator errs on the side of importing rather than silently skipping. ## Flags | Flag | Effect | | -------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `--application-name "..."` | Target application by name. Required unless `--application-id` is set. | | `--application-id ` | Target an existing application by ID. | | `--create-application` | Create the application if it doesn't exist (paired with `--application-name`). | | `--update` | If an agent with the same name already exists, update it in place instead of failing. | | `--prune` | With `--update`, also remove actions and handoffs that exist in Maitai but are absent from the source. | | `--dry-run` | Translate only, print the manifest, don't create anything. | | `--print-manifest` | Print the translated manifest alongside the apply summary. | | `--diff` | Show what would change against the current Maitai state before applying. | | `--diff-only` | Like `--diff`, but stop before applying. | | `--exclude ''` | When importing a directory, skip `.py` files matching this glob (relative to the target dir). Repeatable. Well-known noise dirs (`.venv`, `__pycache__`, `node_modules`, `build`/`dist`, `*.egg-info`, …) are always skipped. | ## How long does it take? The import runs asynchronously: when you press enter, the CLI submits a translation job and polls its status until completion. A typical single-file agent finishes in 20-60 seconds; large multi-file projects can take 1-3 minutes. The CLI shows a live status indicator that moves through `Queued…`, `Translating source…`, and `Applying manifest…` so you can see what's happening. If the job fails (e.g. the translator couldn't find any agents in the source), the CLI surfaces a clear error message describing what went wrong. ## After import Each imported agent shows up under your target application in the Portal. From there you can: * Edit prompts and capabilities like any other Maitai agent. * Publish a version and pin requests to it from your SDK calls. * Wire additional [sub-agents](/build/agents/capabilities/subagent) or [supplied actions](/build/agents/capabilities/supplied_actions) that aren't in the original source. * Configure routing and execution mode per agent. Re-running `agents import` with `--update` is the easiest way to keep imported agents in sync with code changes; add `--prune` to also delete capabilities/handoffs that were removed from the source. ## Examples The Maitai repo includes a small fixture suite covering different frameworks and project layouts. Each is suitable as a starting template: * `examples/agent_import/raw_openai_loop.py`, hand-written agent on the raw OpenAI SDK * `examples/agent_import/crewai_marketing_crew/`, multi-file CrewAI project (agents, tools, tasks, crew) * `examples/agent_import/langgraph_support_pipeline/`, LangGraph with subgraphs in nested directories * `examples/agent_import/pydantic_ai_support_bot.py`, Pydantic AI with typed `Agent[Deps, Output]` * `examples/agent_import/custom_devops_assistant/`, no framework, hand-written base class across multiple files ## Next * Inspect and edit your imported agents: [Create Agent](/build/agents/create_agent) * Call them from your application: [Agent Call](/sdk/agent_call) * Add Maitai-native capabilities: [LLM Action](/build/agents/capabilities/llm_action) · [API Action](/build/agents/capabilities/api_action) # Compositions Source: https://docs.trymaitai.com/build/models/compositions Creating dataset recipes for optimal fine-tuning A **Composition** is the "recipe" for your fine-tune. Rather than just selecting a single dataset, a composition allows you to mix multiple data sources to find the optimal balance for your model. Compositions ### Compositions as Recipes Think of a composition as a way to combine various data sources: * **Mix Multiple Datasets**: Combine real production data with synthetic data from **Synthetic Conversation Trees**. * **Balance Data Ratios**: You might want 70% core tasks, 20% edge cases, and 10% "negative" examples (what NOT to do). * **Experimentation**: Create multiple compositions with different dataset ratios to see which one yields the best performance during validation. ## Next * Start training: [Fine-tune Runs](/build/models/fine_tune_runs) * Full end-to-end walkthrough: [Fine-tune a Model](/examples/fine_tuning) # Dataset Creation Source: https://docs.trymaitai.com/build/models/dataset_creation How to capture and generate data for fine-tuning The foundation of any fine-tune is the data. You can build your datasets using three primary methods in Maitai: Dataset Creation ### Production Traffic Promote real-world interactions directly from the **Observe** view into a dataset. This ensures your model is trained on the actual distributions of your users' requests. ### Manual Upload Import your existing "gold standard" data via CSV or JSON. This is ideal for bootstrapping a model when you already have a verified corpus of high-quality interactions. ### Synthetic Generation Use the **Synthetic Conversation Tree (SCT)** visualizer to map out the logic of your application and generate thousands of unique, structured training examples. This is powerful for covering edge cases that haven't occurred in production yet. ## Next * Prepare and refine your data: [Dataset Preparation](/build/models/dataset_preparation) * Full end-to-end walkthrough: [Fine-tune a Model](/examples/fine_tuning) # Dataset Preparation Source: https://docs.trymaitai.com/build/models/dataset_preparation Reviewing, fixing, and augmenting your training data Preparation is where you turn raw data into a high-quality training set. The quality of your model is directly proportional to the quality of this data. ### Data Hygiene * **Avoid Overlap**: Ensure your Fine-tune Dataset does not overlap with your **Test Set**. You want to train on one set of data and validate on a completely different set to ensure the model generalizes well. * **Curate for Quality**: Remove incorrect, ambiguous, or low-value examples. ### AI Review & Modification * **AI Review**: Use high-capability models to systematically check your dataset for accuracy and consistency. * **Broad Modification**: Perform bulk updates using **Regex Keyword Replacement** to instantly align thousands of responses with new requirements or brand guidelines across your entire dataset. ### Augmentations * **Reasoning Augmentation**: Teach your model to think step-by-step by attaching structured chain-of-thought reasoning to each training example. [Learn more](/build/models/reasoning_augmentation) * **Generate from Examples**: Use your best examples as seeds to generate dozens of similar but unique training cases. * **Variation Injection**: Automatically create variations in tone and phrasing to make your model more robust to different user styles. ## Next * Add structured reasoning to your data: [Reasoning Augmentation](/build/models/reasoning_augmentation) * Combine datasets into a training recipe: [Compositions](/build/models/compositions) * Full end-to-end walkthrough: [Fine-tune a Model](/examples/fine_tuning) # Deployment Source: https://docs.trymaitai.com/build/models/deployment Promoting fine-tuned models to production endpoints Once you are satisfied with the validation results, you can deploy your model to production. ### Deployment Workflow 1. **Deploy Weights**: Make the new model available via a dedicated endpoint. 2. **Studio Testing**: Perform a final verification in **Maitai Studio** to ensure the model captures the desired nuances in a live sandbox environment. 3. **A/B Test**: Use Maitai's routing rules to gradually introduce the new model to production traffic. 4. **Full Promotion**: If the new model performs better in production, promote it to 100% of traffic. ## Next * Monitor real traffic and fault rate: [Intent Groups](/observe/resources/intent_groups) * Prevent regressions with a golden suite: [Regression Monitoring](/test/test_sets/regression) # Fine-tune Runs Source: https://docs.trymaitai.com/build/models/fine_tune_runs Executing and monitoring managed training Once your Composition is ready, you can define the execution parameters and kick off a **Fine-tune Run**.
### Configuration Parameters * **Base Model**: Choose your starting point (e.g., `mistral-7b`, `llama-3-8b`). * **Hyperparameters**: Configure learning rate, batch size, and epochs to optimize training. ### Monitoring Maitai handles the underlying GPU infrastructure automatically. You can monitor the progress in real-time: * **Training Logs**: View detailed output from the training process. * **Loss Curves**: Track the training and validation loss to ensure the model is learning effectively. ## Next * Review automated validation: [Validation](/build/models/validation) * Full end-to-end walkthrough: [Fine-tune a Model](/examples/fine_tuning) # Reasoning Augmentation Source: https://docs.trymaitai.com/build/models/reasoning_augmentation Teach your model to think step-by-step with structured chain-of-thought augmentation **Reasoning Augmentation** enriches your fine-tuning dataset by adding structured chain-of-thought reasoning to each training example. An LLM analyzes every request in your dataset and extracts key information based on a schema you define, then prepends that reasoning to the assistant's response during training. The result: your fine-tuned model learns to "think before it speaks," producing more accurate and consistent outputs. ### How It Works 1. You define a **reasoning schema**: a set of fields with extraction instructions tailored to your use case. 2. Maitai runs an LLM over every request in your dataset to extract values for each field. 3. The extracted reasoning is stored as structured metadata on each dataset request. 4. During fine-tuning, the reasoning is automatically prepended to each response and converted to model-specific tags (e.g., `` for DeepSeek-style models). ### Starting a Reasoning Augmentation Navigate to your dataset's **Augmentations** tab. You'll see both **Reasoning Augmentation** and **Variation Augmentation** buttons. Augmentations Tab Click **Reasoning Augmentation** to open the schema editor. ### Defining a Reasoning Schema The modal lets you define the fields you want the LLM to extract from each request in your dataset. Each field has two parts: * **Field name**: a short label for the extracted value (e.g., `Intent`, `Sentiment`, `Key Entities`). * **Instruction**: a natural-language prompt telling the LLM what to look for (e.g., "Determine what the user is asking for"). Reasoning Augmentation Modal Click **Add Field** to add more fields to your schema. You can remove any field with the trash icon. Once you've defined at least one valid field, click **Start Augmentation** to kick off the job. **Design your schema around the decision points of your task.** Think about what intermediate facts or reasoning steps your model needs before producing its final answer. For example: * **Customer support:** `Customer Intent`, `Urgency`, `Relevant Policy` * **Content moderation:** `Topic`, `Tone`, `Violation Type` * **Data extraction:** `Key Entities`, `Relationships`, `Confidence` * **Code generation:** `Requirements`, `Edge Cases`, `Approach` ### Monitoring Progress After starting the augmentation, a new entry appears in the **Augmentations** table. You can track its status as it progresses from `CREATED` to `RUNNING` to `COMPLETED`. Once completed, click **View** to inspect the extracted reasoning for each request. Each entry shows the key-value pairs the LLM extracted based on your schema. ### Using Reasoning in Fine-tuning When you include a reasoning-augmented dataset in a **Composition** and start a **Fine-tune Run**, the reasoning is automatically handled: * The structured reasoning is prepended to each assistant response in the training data. * The reasoning tags are automatically converted to the appropriate model-specific format for your chosen base model (e.g., `` for DeepSeek-style models). No extra configuration is needed, once reasoning is attached to your dataset, it flows through the entire fine-tuning pipeline. ## Next * Combine datasets into a training recipe: [Compositions](/build/models/compositions) * Start training: [Fine-tune Runs](/build/models/fine_tune_runs) * Full end-to-end walkthrough: [Fine-tune a Model](/examples/fine_tuning) # Validation Source: https://docs.trymaitai.com/build/models/validation Reviewing automated results and checking for regressions After the run completes, Maitai automatically executes a **Validation Run** to verify model accuracy and performance. Validation ### Key Metrics * **Training vs. Validation Loss**: Ensure the model is learning without over-fitting. * **Accuracy Improvement**: Verify the model correctly predicts outputs in your validation set compared to the base model. * **Regression Check**: Run your **Test Set** against the new weights to ensure general performance remains high and no regressions were introduced. ## Next * Deploy a successful run: [Deployment](/build/models/deployment) * Benchmark the candidate model on golden cases: [Test Sets](/test/test_sets/creation) # Build Source: https://docs.trymaitai.com/build/overview Improve the model and orchestrate real work: fine-tuned models, agents, and workflows. **Build** is the higher-investment pillar: once you can [observe](/observe/overview) your traffic and [test](/test/overview) its quality, this is where you improve the underlying model and compose it into real applications: fine-tunes, agents, and multi-step workflows. Come here when you want to answer: *"How do I make the model better at my task, and how do I wire it into something that takes action?"* In the Portal, everything in **Build** lives under the **Forge** section. ## What you can build Fine-tune a model end-to-end: datasets, compositions, runs, validation, then deployment. Tool-using runtimes that plan, call capabilities, and synthesize results. Multi-step AI pipelines you invoke with a single API call. ## End-to-end walkthroughs If you'd rather follow a complete recipe than read reference pages: Create, capabilities, orchestration, state, then test. Turn an OpenAPI spec into an agent with API-call actions and routing. Dataset, composition, run, validation, then deployment. ## Where to go next Build depends on data that [Observe](/observe/overview) produces and quality gates that [Test](/test/overview) enforces, most teams arrive here after both. To call what you build from code, see the [SDK Reference](/sdk/agent_call). # Workflow Context (ctx) Source: https://docs.trymaitai.com/build/workflows/context The runtime API passed to every workflow: LLMs, agents, HTTP, datastores, streaming, and logging. `execute(ctx)` receives a single argument: the **workflow context**. It's how your script reads its input and reaches the outside world, calling models, agents, and other workflows, making HTTP requests, querying datastores, streaming progress, and logging. Everything here is synchronous. The methods handle their own network I/O; you call them directly, no `await`. ```python theme={null} def execute(ctx: WorkflowContext): query = ctx.input["query"] resp = ctx.chat.completions.create( messages=[{"role": "user", "content": query}], application="Support", intent="answer", ) return {"answer": resp["choices"][0]["message"]["content"]} ``` ## Input & request data These attributes carry what the caller sent. See [Invoking workflows](/build/workflows/overview) for how they're populated. The structured payload from the request's `input` field, usually a dict. `None` if the caller sent only `messages`. This is the primary input for most workflows. Chat messages (OpenAI format) from the request's `messages` field. Use for chat-style workflows. Runtime credentials passed via the request's `secrets` field (e.g. third-party API keys). Never stored, available only for the duration of the run. Arbitrary metadata passed by the caller. Config from the workflow's stored configuration, including accessory S3 keys. Also mirrored into `os.environ`. Inference params from the request: `max_tokens`, `temperature`, `stream`. The session ID for this run. Pass it through to `ctx.chat`/`ctx.agents` calls to correlate every step under one session in Maitai analytics. ## Call a model Run an LLM call through Maitai, fully monitored, with your intent's configured model, fallbacks, and Sentinels applied. ```python theme={null} resp = ctx.chat.completions.create( messages=[ {"role": "system", "content": "You are a financial analyst."}, {"role": "user", "content": "Summarize Q3 earnings."}, ], application="Finance", # defaults to the workflow's APPLICATION binding intent="summarize", # required session_id=ctx.session_id, ) content = resp["choices"][0]["message"]["content"] ``` The intent name. Determines which model, prompt config, and Sentinels apply. (`action_type` is accepted as an alias.) The application this call belongs to. Defaults to the workflow's bound application; pass explicitly to override. (`application_ref_name` is an alias.) Chat messages. Defaults to `ctx.messages` if omitted. Optional model override. When omitted, the intent's configured model is used. When `True`, returns an iterator of chunk dicts instead of a single response. Default `False`. Any additional keyword arguments (`temperature`, `max_tokens`, `response_format`, …) are forwarded to the underlying completion. The non-streaming call returns an OpenAI-shaped dict (`choices`, `usage`, `model`, `request_id`, …). `ctx.chat.completions.create(...)` and the flat alias `ctx.chat_completion(...)` are equivalent. The namespaced form mirrors the OpenAI SDK and is recommended. ### Streaming a model call ```python theme={null} for chunk in ctx.chat.completions.create( messages=[{"role": "user", "content": "Write a poem."}], application="Content", intent="compose", stream=True, ): delta = chunk["choices"][0]["delta"].get("content", "") ctx.emit({"type": "token", "content": delta}) ``` ## Call an agent Invoke a Maitai [Agent](/build/agents/create_agent) (including its full reasoning loop) from inside a workflow. ```python theme={null} result = ctx.agents.completions.create( agent="support-triage", messages=[{"role": "user", "content": "My card was declined."}], max_iterations=6, ) ``` The agent reference to invoke. Conversation to send the agent. Defaults to `ctx.messages`. The full agent request surface, same overlays as the [agent SDK](/sdk/agent_call): capability mask (`actions`), form seed (`state`), per-request `secrets`, `config` overlay, iteration cap, and structured `response_format`. Optional LLM override, separate from the agent reference. Stream the agent's events as an iterator. Default `False`. The flat alias is `ctx.agent_completion(agent, ...)`. A `400` from the agent (e.g. a missing required secret) surfaces as a `ValueError` you can catch. ## Call another workflow Compose workflows by invoking one from another. ```python theme={null} rate = ctx.workflows.completions.create( workflow="hs-rate-line-classifier", input={"subheading": subheading, "product": product}, ) ``` Reference of the workflow to call (optionally `ref:tag`). Structured input for the nested workflow. Returns the nested workflow's result dict (with `output` and `session_id`). The flat alias is `ctx.workflow_completion(workflow, input=...)`. ## Make HTTP requests `ctx.request(method, url, ...)` is a thin, pooled wrapper over [httpx](https://www.python-httpx.org/). Any keyword httpx accepts, `json`, `data`, `params`, `headers`, `auth`, `timeout`, `files`, …, is forwarded verbatim. ```python theme={null} resp = ctx.post( "https://api.example.com/score", json={"text": ctx.input["text"]}, headers={"Authorization": f"Bearer {ctx.secrets['EXAMPLE_KEY']}"}, retries=2, ) if resp["ok"]: score = resp["json"]["score"] ``` The call returns a plain, JSON-serializable dict: ```python theme={null} { "status_code": 200, "ok": True, # True for 2xx "headers": {...}, "json": {...}, # parsed body, or None if not JSON "text": "...", # raw body text "url": "https://...", # final URL after redirects "elapsed_ms": 142.5, } ``` Additional attempts on a transport error or a transient status. `0` means a single attempt. Status codes treated as transient and retried when `retries > 0`. When `True`, raise on a 4xx/5xx. Default `False`, branch on `status_code`/`ok` instead. Verb shorthands exist for every method: `ctx.get`, `ctx.post`, `ctx.put`, `ctx.patch`, `ctx.delete`, `ctx.head`, `ctx.options`. `follow_redirects` defaults to `True`. Transport-level failures (DNS, connection refused, timeout) raise after retries are exhausted. ### Streaming an HTTP response `ctx.request_stream(method, url, ...)` is the streaming sibling of `ctx.request`. Instead of buffering the whole body, it returns a context-managed handle you iterate incrementally, ideal for forwarding an upstream NDJSON or SSE stream to your caller one event at a time. ```python theme={null} with ctx.request_stream("POST", "https://api.example.com/run", json=payload, retries=2) as resp: if not resp.ok: return {"error": resp.read()} for line in resp.iter_lines(): if line: ctx.emit(json.loads(line)) # forward each event live ``` The handle exposes `status_code`, `ok`, `headers`, and `url` **before** you read the body (so you can branch on status first), plus: | Method | Returns | Use for | | ---------------------------------- | ----------------- | ---------------------------------------- | | `resp.iter_lines()` | `Iterator[str]` | NDJSON / line-delimited streams | | `resp.iter_bytes(chunk_size=None)` | `Iterator[bytes]` | Binary, SSE, custom framing | | `resp.read()` | `str` | Buffer the remaining body (escape hatch) | It accepts the same `retries` / `retry_backoff` / `retry_statuses` / `raise_for_status` options as `ctx.request`, with one caveat: **retries apply only while opening the stream.** Once you start iterating the body, a mid-stream failure can't be replayed and propagates to you. Always use it as a context manager (`with ... as resp:`), the connection is released on exit, including on early `break` or an exception. ## Stream progress to the caller `ctx.emit(data)` pushes an intermediate chunk to the caller. When the workflow was invoked with `stream=True`, each chunk is written to the response stream immediately; otherwise chunks are buffered and returned with the final result, so the same code works in both modes. ```python theme={null} ctx.emit({"type": "status", "content": "Step 2 of 3: scoring candidates"}) ``` Pass any dict. It's wrapped as a `WorkflowChunk` (`type`, `content`, `metadata`) and stamped with an epoch-millisecond `timestamp` so consumers can order and debug events. See the [consumer side](/build/workflows/overview#streaming-responses) for how clients read these. ## Query a datastore If your workflow has an attached [datastore](/build/workflows/data#datastores), reach it through `ctx.datastore`. ```python theme={null} from maitai_workflow import Tag, Text, all_, any_, not_ # Fetch a full record by id ruling = ctx.datastore.get("ruling-12345") # Vector (semantic) search hits = ctx.datastore.search(vector=embedding, k=10) # Keyword / hybrid search with the filter DSL hits = ctx.datastore.search( where=all_( Tag("hts_codes", ["6202"], prefix=True), any_(Text("leather boots"), Tag("keywords", ["leather"])), not_(Tag("keywords", ["children"])), ), k=20, ) ``` Fetch one full record by id, or pass a list of ids for a batch (missing ids are omitted). Reads from S3 with in-process caching. Query the index. Provide `vector` for KNN semantic search (sorted by similarity), and/or `where` to filter. `full=True` hydrates complete records from S3; the default returns only indexed fields. `datastore_name` is required only when the workflow has more than one datastore. The `where` filter accepts a composable `Filter` (`Tag`, `Text`, combined with `all_`/`any_`/`not_` or the `&` `|` `~` operators), a raw RediSearch string, or `None` (matches everything). Full DSL details live in [Datastores & Accessories](/build/workflows/data#querying). ## Load an accessory file Read a [bundled reference file](/build/workflows/data#accessory-files) by key. Resolution order is cache, then Redis, then S3, with an optional local fallback for development. ```python theme={null} hts_codes = ctx.load_accessory("hts_codes", fallback_path="./hts_codes.json") ``` ## Embeddings & web search ```python theme={null} vecs = ctx.embeddings.create(input=["text one", "text two"]) answer = ctx.web_search("latest USTR tariff announcements") ``` Embed a string or list of strings. Returns the embeddings API response. Web search (Perplexity). Returns plain text, concatenated snippets by default, or an LLM-composed answer when `system_prompt` is given. Returns an empty string if unavailable. ## Logging & profiling ```python theme={null} ctx.log("scoring complete", candidates=len(hits), top_score=hits[0]["score"]) ctx.profile_span("retrieval") # time since the last span / script start ctx.profile_span("generation") ``` Structured log line. Keyword fields are attached to the log record and rendered inline, so they show up in your observability tooling. Record a timing span from the last checkpoint to now. No-op unless profiling is enabled (send `X-Profile: true`). `ctx.profile_span_from_start(label)` measures from script start instead. Spans also drive the **step** boundaries in the [flow diagram](/build/workflows/overview). ## Next * How a workflow file is laid out: [Workflow Structure](/build/workflows/structure) * Attach reference data: [Datastores & Accessories](/build/workflows/data) * Call your workflow from code: [Invoking workflows](/build/workflows/overview) # Datastores & Accessories Source: https://docs.trymaitai.com/build/workflows/data Give a workflow reference data: small bundled files, or large indexed, queryable record stores. Workflows often need reference data: a lookup table, a prompt library, a corpus of documents to search. Maitai offers two mechanisms, and the right one depends on **size** and **access pattern**. | | [Accessory file](#accessory-files) | [Datastore](#datastores) | | -------------------- | ----------------------------------------------- | ------------------------------------------------------ | | **What it is** | One read-only JSON blob bundled with the script | An S3-backed, indexed record store | | **You read it with** | `ctx.load_accessory("key")` | `ctx.datastore.get(...)` / `ctx.datastore.search(...)` | | **Access pattern** | Load the whole file into memory | Query by id, keyword, or vector similarity | | **Best for** | Small lookup tables, enums, config, prompt sets | Large corpora, semantic/keyword search, retrieval | | **Size** | Up to \~50 MB per file | Effectively unlimited | A rule of thumb: if you'll hold the entire file in memory and index into it yourself, use an **accessory**. If you need to *search* or *selectively fetch* from a large collection, use a **datastore**. ## Accessory files An accessory is a single JSON file you upload alongside your workflow under a short **key**. At runtime, [`ctx.load_accessory(key)`](/build/workflows/context#load-an-accessory-file) returns its parsed contents. ```python theme={null} def execute(ctx: WorkflowContext): # Loaded once, then cached in-process and in Redis for the worker's lifetime. hts_codes = ctx.load_accessory("hts_codes") notes = ctx.load_accessory("hts_notes", fallback_path="./hts_notes.json") ... ``` **How the key maps to storage.** When you upload an accessory under key `hts_codes`, Maitai stores the file in S3 and sets an env var `HTS_CODES_S3_KEY` in the workflow's config. `load_accessory("hts_codes")` reads that env var to locate the object. The key is uppercased and hyphens become underscores (`hts-codes` becomes `HTS_CODES_S3_KEY`), so **keep your `load_accessory` key identical to the upload key.** **Resolution order.** `load_accessory` checks, in order: in-process cache, then Redis (24h TTL), then S3 (then writes back to Redis). If no S3 key is configured and you passed `fallback_path`, it loads that local file, convenient for local development before the accessory is uploaded. **Limits.** Up to \~50 MB per accessory and 100 accessories per workflow. Keys are lowercase alphanumeric plus `_`/`-`. Files must be valid UTF-8 JSON. See [Uploading a workflow](#uploading-a-workflow) for the `--accessory` flag. ## Datastores A datastore is a queryable collection of JSON records. You define which fields to index and how; Maitai stores the raw records in S3 and builds a search index over the indexed fields. At runtime your workflow queries it through [`ctx.datastore`](/build/workflows/context#query-a-datastore) (by record id, by keyword/tag filter, or by vector similarity) without managing any of the underlying infrastructure. Each workflow has at most one datastore. (Need several datasets? Split them across workflows and compose with [nested calls](/build/workflows/context#call-another-workflow).) ### Schema A datastore is defined by a small schema: which field is the record key, where the raw records live, and how each indexed field is treated. ```yaml theme={null} record_key: ruling_id # primary id field on each record records_prefix: avalara/rulings/enriched # S3 prefix holding the raw JSON records fields: ruling_id: type: tag hts_codes: type: tag separator: "," # multi-value tag field product_short: type: text # full-text searchable summary: type: text embedding: type: vector # dense-vector similarity search dim: 1536 algorithm: HNSW # HNSW | FLAT distance_metric: COSINE # COSINE | L2 | IP ``` **Field types:** | Type | Query with | Notes | | --------- | -------------------- | ------------------------------------------------------------------------------------------------------ | | `tag` | `Tag(field, [...])` | Exact or prefix match; OR across values. `separator` enables multi-value fields. | | `text` | `Text("...")` | Tokenized full-text search. | | `numeric` | `Raw(...)` | Numeric range filters via the raw escape hatch. | | `vector` | `search(vector=...)` | KNN similarity. Requires `dim`; `algorithm` (HNSW/FLAT) and `distance_metric` (COSINE/L2/IP) optional. | Only the fields you list are indexed; `ctx.datastore.get` always returns the complete record from S3. ### Querying Import the filter helpers from `maitai_workflow` and pass a filter as `where`. Helpers compose with `all_`/`any_`/`not_` (or the `&` `|` `~` operators). ```python theme={null} from maitai_workflow import Tag, Text, Raw, all_, any_, not_ # Exact id fetch (full record from S3) record = ctx.datastore.get("ruling-12345") # Semantic search: pass an embedding vector hits = ctx.datastore.search(vector=query_embedding, k=10) # Keyword search with a tag filter hits = ctx.datastore.search(where=Tag("keywords", ["leather", "boots"]), k=20) # Hybrid: vector + structured filter hits = ctx.datastore.search( vector=query_embedding, where=all_( Tag("hts_codes", ["6202"], prefix=True), # prefix match any_(Text("leather boots"), Tag("keywords", ["leather"])), not_(Tag("keywords", ["children"])), ), k=10, ) # Power user: raw RediSearch (phrases, wildcards, numeric ranges) hits = ctx.datastore.search(where=Text('"steel toe" leather*', op="raw"), k=20) ``` * **`Tag(field, values, prefix=False)`**, match a tag field against one or more values (OR). `prefix=True` matches by prefix (`"6202"` matches `6202.10.4080`, …). * **`Text(terms, field=None, op="any")`**, full-text search. `op="any"` (OR, recall), `op="all"` (AND, precision), or `op="raw"` for verbatim RediSearch (phrases, `leather*`, etc.). * **`Raw(expr)`**, drop in a literal RediSearch fragment for anything the DSL doesn't model (numeric/geo ranges). `search` returns only the **indexed** fields by default; pass `full=True` to hydrate the complete records from S3 (slower). With a `vector`, results are sorted by similarity; without one, by text relevance. ### Creating a datastore 1. **Define the schema** for the workflow (a YAML/JSON document like above): ```bash theme={null} POST /workflows/{workflow_id}/datastores/{name} ``` 2. **Upload records**: a JSON array of records, or an S3 URI to ingest: ```bash theme={null} POST /workflows/{workflow_id}/datastores/{name}/upload ``` Maitai stores the records, computes/loads the indexed fields, and builds the search index in the background. Once ready, workers serve `ctx.datastore` queries against it. Manage datastores with `GET /workflows/{workflow_id}/datastores` and `DELETE /workflows/{workflow_id}/datastores/{name}`. ## Uploading a workflow Register a workflow (and bundle its accessories) with the upload script, run from the repository root: ```bash theme={null} python scripts/db_ops/upload_workflow.py [options] ``` This uploads the script to S3, upserts the workflow row (timeout, execution mode, application binding), and uploads any accessory files next to the script. Callers then invoke it as `model="workflow:"`. ```bash theme={null} # Minimal python scripts/db_ops/upload_workflow.py 22 my-workflow ./workflows/my_workflow.py # Bind to an application (created if missing) and bundle two accessories python scripts/db_ops/upload_workflow.py 2 hs-classification ./workflows/avalara/hs_classification.py \ --application-name "HS Classification" \ --accessory hts_codes=./workflows/avalara/hts_codes_v2.json \ --accessory hts_notes=./workflows/avalara/hts_notes.json ``` | Option | Meaning | | ---------------------- | ---------------------------------------------------------------------------------------------------- | | `--name` | Display name (defaults to a title-cased ref name). | | `--desc` | Description. | | `--timeout` | Worker timeout in **seconds** (default `600`). | | `--mode` | Execution mode: `sync`, `async`, or `stream` (default `sync`). | | `--application-name` | Bind to an application by name, creating it if needed. Don't combine with `--application-id`. | | `--application-id` | Bind to an existing application id (its company is used; `company_id` may then be omitted). | | `--accessory KEY=path` | Repeatable. Uploads the file and exposes it to `ctx.load_accessory("KEY")` via `{KEY_UPPER}_S3_KEY`. | | `--publish-version` | After upsert, publish an immutable version (like the Portal's **Publish**). | | `--version-notes` | Notes for the published version. | | `--version-bump` | `major` / `minor` / `patch` (default `minor`) when publishing. | Use `company_id = -1` for a global workflow. Run from `maitai-backend/` so relative paths like `./workflows/...` resolve, with AWS + database credentials configured for your target environment. ### Execution modes | Mode | Behavior | | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- | | `sync` | Caller blocks until the workflow finishes; the response carries the final output. The default. | | `async` | The run is queued; the call returns immediately and you poll for the result. | | `stream` | Intermediate chunks (from [`ctx.emit`](/build/workflows/context#stream-progress-to-the-caller)) stream to the caller, followed by the final completion. | ## Next * Use this data in your script: [Workflow Context (`ctx`)](/build/workflows/context) * How a workflow file is laid out: [Workflow Structure](/build/workflows/structure) * Invoke your workflow: [Invoking workflows](/build/workflows/overview) # Workflows Source: https://docs.trymaitai.com/build/workflows/overview Execute complex AI workflows through a single API call. Maitai Workflows let you execute multi-step AI pipelines through a single API call. A workflow is a Python script you write; Maitai runs it in a managed worker pool and handles execution, scaling, monitoring, and caching. Orchestrate models, agents, HTTP services, and custom logic, then invoke the whole thing the way you'd call any model. ## What a workflow is A workflow is **one Python file** with an `execute(ctx)` function. Maitai calls `execute`, passes it a [context object](/build/workflows/context), and returns whatever you return. ```python theme={null} from maitai_workflow import WorkflowContext def execute(ctx: WorkflowContext): resp = ctx.chat.completions.create( messages=[{"role": "user", "content": ctx.input["query"]}], application="Support", intent="answer", ) return {"answer": resp["choices"][0]["message"]["content"]} ``` The `ctx` object is the workflow's connection to everything: its input, Maitai model and agent calls, arbitrary HTTP, queryable datastores, streaming, and logging. ### Building blocks The script layout, the `execute` entrypoint, input/output schemas, and helpers. The runtime API, models, agents, HTTP, streaming, logging, and more. Attach reference data, bundled files or large, queryable record stores. Call a workflow from the SDK or raw API, including streaming. ### Lifecycle 1. **Write** the [script](/build/workflows/structure) and (optionally) attach [datastores or accessories](/build/workflows/data). 2. **Upload** it, see [Uploading a workflow](/build/workflows/data#uploading-a-workflow). Maitai registers it under a `workflow_ref_name`. 3. **Invoke** it as `model="workflow:"` (below). Maitai runs it, monitors every model call, and returns the result. The Maitai team can also configure workflows for your account during onboarding. ## Invoke a workflow Set `model` to `workflow:`. You can pass either `messages` (chat-style) or `input` (arbitrary JSON payload), or both. ### With messages (chat-style) ```python Python theme={null} from maitai import Maitai maitai = Maitai() response = maitai.workflow.completions.create( model="workflow:my-workflow", messages=[ {"role": "user", "content": "Summarize this quarter's earnings report."}, ], ) print(response.choices[0].message.content) ``` ```javascript Node theme={null} import Maitai from "maitai"; const maitai = new Maitai(); const response = await maitai.workflow.completions.create({ model: "workflow:my-workflow", messages: [ { role: "user", content: "Summarize this quarter's earnings report." }, ], }); console.log(response.choices[0].message.content); ``` ### With input (structured payload) For workflows that expect structured data rather than chat messages, use `input`: ```python theme={null} response = maitai.workflow.completions.create( model="workflow:my-etl-pipe", input={"document_url": "https://...", "output_format": "markdown"}, ) ``` ```javascript theme={null} const response = await maitai.workflow.completions.create({ model: "workflow:my-etl-pipe", input: { document_url: "https://...", output_format: "markdown" }, }); ``` ### Raw API (cURL) ```bash theme={null} curl -X POST https://api.trymaitai.ai/workflow/chat/completions \ -H "Content-Type: application/json" \ -H "x-api-key: $MAITAI_API_KEY" \ -d '{ "model": "workflow:my-workflow", "input": {"document_url": "https://...", "output_format": "markdown"} }' ``` ### With session\_id (for correlation) Pass `session_id` to correlate all model calls within a workflow run in Maitai analytics: ```python theme={null} response = maitai.workflow.completions.create( model="workflow:my-workflow", messages=[{"role": "user", "content": "..."}], session_id="my-session-123", # Optional; overrides auto-generated ) ``` ### Streaming responses To receive workflow output in real-time as the script executes, set `stream=True`. The workflow will yield intermediate chunks as they are emitted (via `ctx.emit()`), followed by the final completion. ```python theme={null} stream = maitai.workflow.completions.create( model="workflow:my-workflow", input={"document_url": "https://...", "output_format": "markdown"}, stream=True, ) for chunk in stream: if hasattr(chunk, "type"): # This is an intermediate workflow chunk (from ctx.emit) print(f"[{chunk.type}] {chunk.content}") elif hasattr(chunk, "choices"): # This is the final ChatCompletion object print("Final result:", chunk.choices[0].message.content) ``` ```javascript theme={null} const stream = await maitai.workflow.completions.create({ model: "workflow:my-workflow", input: { document_url: "https://...", output_format: "markdown" }, stream: true, }); for await (const chunk of stream) { if (chunk.type) { // Intermediate chunk console.log(`[${chunk.type}]`, chunk.content); } else if (chunk.choices) { // Final completion console.log("Final result:", chunk.choices[0].message.content); } } ``` ## Passing secrets If your workflow requires runtime credentials (e.g. third-party API keys), pass them via `secrets`. These are never stored, they're only available for the duration of the request. ```python theme={null} response = maitai.workflow.completions.create( model="workflow:my-workflow", messages=[{"role": "user", "content": "Fetch and summarize the latest data."}], secrets={"EXTERNAL_API_KEY": "tok-..."}, ) ``` ```javascript theme={null} const response = await maitai.workflow.completions.create({ model: "workflow:my-workflow", messages: [{ role: "user", content: "Fetch and summarize the latest data." }], secrets: { EXTERNAL_API_KEY: "tok-..." }, }); ``` ## Request format The workflow to invoke, in the format `workflow:`. An array of chat messages (OpenAI format). Required unless `input` is provided. ```json theme={null} [ {"role": "system", "content": "You are an analyst."}, {"role": "user", "content": "Summarize the report."} ] ``` Arbitrary JSON payload passed to the workflow. Required unless `messages` is provided. Use this for Lambda-style invocations where chat messages don't apply. ```json theme={null} {"document_url": "https://...", "output_format": "markdown"} ``` Optional session ID for the workflow run. If provided, overrides the auto-generated session ID. All model calls within the workflow use this session, enabling correlation across steps in Maitai analytics. Optional. When `true`, returns a stream of intermediate workflow chunks (emitted via `ctx.emit()`) followed by a final `chat.completion` object, allowing you to display real-time progress. Default `false`. Optional key-value pairs available to the workflow at runtime (e.g. API keys the workflow needs to call external services). Not stored. Optional metadata passed to the workflow. ## Response format Workflows return a standard OpenAI-compatible `chat.completion` response: ```json theme={null} { "id": "chatcmpl-abc123", "object": "chat.completion", "created": 1770848541, "model": "workflow:my-workflow", "choices": [ { "index": 0, "message": { "role": "assistant", "content": "Here is the earnings summary..." }, "finish_reason": "stop" } ], "usage": { "prompt_tokens": 0, "completion_tokens": 0, "total_tokens": 0 }, "request_id": "d4f7e2a1-..." } ``` Unique identifier for the completion. The workflow that was invoked. Array containing the workflow output. `choices[0].message.content` holds the result. Maitai request identifier, useful for debugging. ## Authentication Workflows use the same API key authentication as all Maitai endpoints. See [Authentication](/get_started/authentication). ## Error handling | Status | Meaning | | ------ | ------------------------------------------------------------------------- | | `400` | Invalid request (missing both `input` and `messages`, bad `model` format) | | `401` | Invalid or missing API key | | `404` | Workflow not found for your account | | `408` | Workflow exceeded its timeout | | `500` | Workflow execution error (check `error.message` for details) | Error responses follow the standard format: ```json theme={null} { "error": { "message": "Workflow not found: my-workflow", "type": "invalid_request_error" } } ``` ## Next * Write your first workflow: [Workflow Structure](/build/workflows/structure) * The full runtime API: [Workflow Context (`ctx`)](/build/workflows/context) * Attach reference data and upload: [Datastores & Accessories](/build/workflows/data) # Workflow Structure Source: https://docs.trymaitai.com/build/workflows/structure How a workflow script is organized: the execute entrypoint, input/output schemas, and helpers. A Maitai workflow is a single Python file. Maitai loads it into the [workflow worker pool](/build/workflows/overview), calls one function, `execute(ctx)`, and returns whatever that function returns. Everything you need at runtime arrives on the [`ctx` object](/build/workflows/context): input, secrets, LLM access, HTTP, datastores, and more. Here is a complete, minimal workflow: ```python theme={null} from maitai_workflow import WorkflowContext APPLICATION = "My Application" INPUT_SCHEMA = { "query": {"type": "string", "description": "The user's question", "required": True}, "context": {"type": "string", "description": "Optional background context"}, } OUTPUT_SCHEMA = { "answer": {"type": "string", "description": "The generated answer"}, "confidence": {"type": "number", "description": "Confidence between 0 and 1"}, } def execute(ctx: WorkflowContext): inp = ctx.input or {} query = inp.get("query", "") if not query: return {"error": "query is required"} resp = ctx.chat.completions.create( messages=[{"role": "user", "content": query}], application=APPLICATION, intent="answer_query", ) answer = resp["choices"][0]["message"]["content"] return {"answer": answer, "confidence": 0.85} ``` The repo ships this as a copy-paste starting point at `workflows/workflow.example.py`. You can also fetch the latest template from the API at `GET /workflows/default-script`. ## The `execute(ctx)` entrypoint Every workflow **must** define exactly one top-level `execute` function. * **Required and unique.** There must be precisely one module-level `def execute(...)`. Upload validation rejects a script with zero or multiple. * **Synchronous.** Write `def execute(ctx):`, not `async def`. The worker runs your function on a thread, use `ctx`'s synchronous methods (they handle their own I/O) rather than `await`. * **Takes the context.** It receives a single argument, the [`WorkflowContext`](/build/workflows/context), conventionally named `ctx`. * **Returns a value.** Return your result, typically a `dict` matching `OUTPUT_SCHEMA`. A bare `return` or `return None` is rejected at upload. The returned object is exposed verbatim as `response.output`, and a JSON-stringified copy becomes `choices[0].message.content` for OpenAI-shaped consumers. You can define as many helper functions, constants, and imports at module level as you like, only `execute` is special. Keep helpers as plain top-level `def`s; the [flow graph](/build/workflows/overview) inlines them when visualizing your workflow. ```python theme={null} SYSTEM_PROMPT = "You are a careful financial analyst." def _build_messages(query: str) -> list: return [ {"role": "system", "content": SYSTEM_PROMPT}, {"role": "user", "content": query}, ] def execute(ctx: WorkflowContext): resp = ctx.chat.completions.create( messages=_build_messages(ctx.input["query"]), application="Finance", intent="analyze", ) return {"answer": resp["choices"][0]["message"]["content"]} ``` ## The `WorkflowContext` import ```python theme={null} from maitai_workflow import WorkflowContext ``` If you reference `WorkflowContext` anywhere, most commonly as the type hint on `execute(ctx: WorkflowContext)`, you must import it, or the script fails at runtime with a `NameError`. Upload validation flags a missing import. The import is also what makes editor autocomplete work for the full `ctx` surface. ## Input and output schemas `INPUT_SCHEMA` and `OUTPUT_SCHEMA` are optional module-level dicts that describe the shape of `ctx.input` and your return value. ```python theme={null} INPUT_SCHEMA = { "document_url": {"type": "string", "description": "URL of the PDF to process", "required": True}, "output_format": {"type": "string", "description": "markdown | html"}, "max_pages": {"type": "number", "description": "Cap on pages to read"}, } ``` * **Format.** A dict mapping each field name to `{type, description, required?}`. * **Types.** `"string"`, `"number"`, `"boolean"`, `"object"`, `"array"`. * **Required fields.** Add `"required": True`. Omitting the key (or `False`) marks the field optional. Schemas are **informational, not enforced at runtime.** They power the Portal, describing what `ctx.input` expects and what your output contains, and rendering the **Input** and **Output** nodes in the flow diagram. Validate inputs you actually depend on inside `execute`. The upload step extracts these dicts statically (via AST), so they must be **literal** dicts at module level, don't compute them at import time or build them inside a function. ## Recognized constants | Constant | Purpose | | --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `APPLICATION` | The application this workflow's LLM calls belong to. Used as the default `application` for `ctx.chat.completions.create(...)` when you pass it explicitly, and it documents intent. Binding is set at upload via `--application-name` / `--application-id` (see [Uploading](/build/workflows/data#uploading-a-workflow)). | | `INPUT_SCHEMA` | Input shape (above). | | `OUTPUT_SCHEMA` | Output shape (above). | Any other module-level constants (like `SYSTEM_PROMPT` above) are yours to use freely, they aren't special to Maitai. ## Streaming output If callers invoke your workflow with `stream=True`, you can push intermediate events to them as the script runs by calling [`ctx.emit(...)`](/build/workflows/context#ctx-emit). When not streaming, emitted chunks are buffered and returned with the final response, so the same code works either way. ```python theme={null} def execute(ctx: WorkflowContext): ctx.emit({"type": "status", "content": "fetching document"}) doc = ctx.get(ctx.input["document_url"]) ctx.emit({"type": "status", "content": "summarizing"}) resp = ctx.chat.completions.create( messages=[{"role": "user", "content": doc["text"]}], application="Docs", intent="summarize", ) return {"summary": resp["choices"][0]["message"]["content"]} ``` See [Invoking workflows, streaming responses](/build/workflows/overview#streaming-responses) for the consumer side. ## `transform_session` (optional) Define `transform_session(session)` if you want to convert recorded Maitai sessions into [test set](/test/test_sets/creation) items for this workflow. It takes a session and returns the `input`/`output` pair that `execute` would have seen: ```python theme={null} def transform_session(session): workflow_input, workflow_output = {}, {"content": ""} if session.timeline: first = session.timeline[0] if first.request and first.request.params: for msg in reversed(first.request.params.messages): if msg.role == "user" and msg.content: workflow_input = {"query": msg.content} break last = session.timeline[-1] if last.response and last.response.choices: workflow_output = {"content": last.response.choices[0].message.content or ""} return {"input": workflow_input, "output": workflow_output} ``` The return value must be a dict with exactly two keys, `"input"` and `"output"` (where `"output"` is itself a dict). When present, the Portal lets you import sessions into a test set; when absent, that option is simply hidden. ## Next * The full runtime API: [Workflow Context (`ctx`)](/build/workflows/context) * Attach reference data: [Datastores & Accessories](/build/workflows/data) * Call your workflow from code: [Invoking workflows](/build/workflows/overview) # Commands Source: https://docs.trymaitai.com/cli-reference/commands Full CLI command reference for resource commands and raw API. ## Auth commands | Command | Description | | ----------------------------------------------- | --------------------------------------------------- | | `maitai login [--api-key KEY] [--base-url URL]` | Authenticate and store API key in \~/.maitai/config | | `maitai logout` | Remove stored credentials | | `maitai whoami` | Show current authentication status | | `maitai docs` | Show API documentation references | ## Raw API Call any endpoint directly: ```bash theme={null} maitai api METHOD path [-b JSON_BODY] ``` Examples: ```bash theme={null} maitai api GET /applications maitai api GET /applications/42 maitai api POST /applications -b '{"application_name":"Test","application_ref_name":"test"}' maitai api PUT /applications/42 -b '{"application_name":"Updated"}' maitai api DELETE /applications/42 ``` ## Resource commands Each resource is a command group. Run `maitai --help` to see available subcommands, then `maitai --help` for scoped arguments. ### `maitai agents` | CLI command | Method | Path | Operation | | ----------------------------------------- | ------ | --------------------------------------------------------- | --------------------------- | | `maitai agents list` | GET | `/agents` | listAgents | | `maitai agents get` | GET | `/agents/{agent_id}` | getAgent | | `maitai agents create` | POST | `/agents` | createAgent | | `maitai agents update` | PUT | `/agents/{agent_id}` | updateAgent | | `maitai agents delete` | DELETE | `/agents/{agent_id}` | deleteAgent | | `maitai agents disable-action` | PUT | `/agents/actions/{action_id}/disable` | disableAction | | `maitai agents enable-action` | PUT | `/agents/actions/{action_id}/enable` | enableAction | | `maitai agents publish-action-version` | POST | `/agents/actions/{action_id}/publish` | publishActionVersion | | `maitai agents restore-action-version` | POST | `/agents/actions/{action_id}/versions/{version}/restore` | restoreActionVersion | | `maitai agents list-action-versions` | GET | `/agents/actions/{action_id}/versions` | listActionVersions | | `maitai agents delete-action` | DELETE | `/agents/actions/{action_id}` | deleteAgentAction | | `maitai agents get-action` | GET | `/agents/actions/{action_id}` | getAgentAction | | `maitai agents patch-action` | PATCH | `/agents/actions/{action_id}` | patchAgentAction | | `maitai agents update-action` | PUT | `/agents/actions/{action_id}` | updateAgentAction | | `maitai agents import` | POST | `/agents/import` | importAgents | | `maitai agents list-supplied-actions` | GET | `/agents/supplied-actions` | listSuppliedActions | | `maitai agents get-task-tree` | GET | `/agents/tasks/{task_id}/tree` | getAgentTaskTree | | `maitai agents add-supplied-action` | POST | `/agents/{agent_id}/actions/supplied/{catalog_action_id}` | addSuppliedAction | | `maitai agents list-actions` | GET | `/agents/{agent_id}/actions` | listAgentActions | | `maitai agents create-action` | POST | `/agents/{agent_id}/actions` | createAgentAction | | `maitai agents delete-form-fields` | DELETE | `/agents/{agent_id}/form-fields` | deleteFormFields | | `maitai agents get-form-fields` | GET | `/agents/{agent_id}/form-fields` | getFormFields | | `maitai agents update-form-fields` | PUT | `/agents/{agent_id}/form-fields` | updateFormFields | | `maitai agents get-invocation` | GET | `/agents/{agent_id}/invocations/{task_id}` | getAgentInvocation | | `maitai agents invoke` | POST | `/agents/{agent_id}/invoke` | invokeAgent | | `maitai agents publish-version` | POST | `/agents/{agent_id}/publish` | publishAgentVersion | | `maitai agents delete-release` | DELETE | `/agents/{agent_id}/releases/{name}` | deleteAgentRelease | | `maitai agents upsert-release` | PUT | `/agents/{agent_id}/releases/{name}` | upsertAgentRelease | | `maitai agents list-releases` | GET | `/agents/{agent_id}/releases` | listAgentReleases | | `maitai agents get-routing-config` | GET | `/agents/{agent_id}/routing/config` | getRoutingConfig | | `maitai agents update-routing-config` | PUT | `/agents/{agent_id}/routing/config` | updateRoutingConfig | | `maitai agents get-routing-result` | GET | `/agents/{agent_id}/routing/results/{routing_result_id}` | getRoutingResult | | `maitai agents reorder-routing-rule` | PUT | `/agents/{agent_id}/routing/rules/{rule_id}/reorder` | reorderRoutingRule | | `maitai agents test-routing-rule` | POST | `/agents/{agent_id}/routing/rules/{rule_id}/test` | testRoutingRule | | `maitai agents delete-routing-rule` | DELETE | `/agents/{agent_id}/routing/rules/{rule_id}` | deleteRoutingRule | | `maitai agents update-routing-rule` | PUT | `/agents/{agent_id}/routing/rules/{rule_id}` | updateRoutingRule | | `maitai agents create-routing-rule` | POST | `/agents/{agent_id}/routing/rules` | createRoutingRule | | `maitai agents get-session-timeline` | GET | `/agents/{agent_id}/sessions/{session_id}/timeline` | getAgentSessionTimeline | | `maitai agents get-session-detail` | GET | `/agents/{agent_id}/sessions/{session_id}` | getAgentSessionDetail | | `maitai agents list-sessions` | GET | `/agents/{agent_id}/sessions` | listAgentSessions | | `maitai agents get-status` | GET | `/agents/{agent_id}/status` | getAgentStatus | | `maitai agents disable-sub` | PUT | `/agents/{agent_id}/subagents/{child_agent_id}/disable` | disableSubAgent | | `maitai agents enable-sub` | PUT | `/agents/{agent_id}/subagents/{child_agent_id}/enable` | enableSubAgent | | `maitai agents remove-sub` | DELETE | `/agents/{agent_id}/subagents/{child_agent_id}` | removeSubAgent | | `maitai agents list-sub` | GET | `/agents/{agent_id}/subagents` | listSubAgents | | `maitai agents add-sub` | POST | `/agents/{agent_id}/subagents` | addSubAgent | | `maitai agents reset-system-step-config` | DELETE | `/agents/{agent_id}/system-steps/{step}/config` | resetAgentSystemStepConfig | | `maitai agents get-system-step-config` | GET | `/agents/{agent_id}/system-steps/{step}/config` | getAgentSystemStepConfig | | `maitai agents update-system-step-config` | PUT | `/agents/{agent_id}/system-steps/{step}/config` | updateAgentSystemStepConfig | | `maitai agents disable-system-step` | PUT | `/agents/{agent_id}/system-steps/{step}/disable` | disableAgentSystemStep | | `maitai agents enable-system-step` | PUT | `/agents/{agent_id}/system-steps/{step}/enable` | enableAgentSystemStep | | `maitai agents reset-system-step-prompt` | DELETE | `/agents/{agent_id}/system-steps/{step}/prompt` | resetAgentSystemStepPrompt | | `maitai agents get-system-step-prompt` | GET | `/agents/{agent_id}/system-steps/{step}/prompt` | getAgentSystemStepPrompt | | `maitai agents update-system-step-prompt` | PUT | `/agents/{agent_id}/system-steps/{step}/prompt` | updateAgentSystemStepPrompt | | `maitai agents list-system-steps` | GET | `/agents/{agent_id}/system-steps` | listAgentSystemSteps | | `maitai agents get-task-timeline` | GET | `/agents/{agent_id}/tasks/{task_id}/timeline` | getAgentTaskTimeline | | `maitai agents diff-versions` | GET | `/agents/{agent_id}/versions/diff` | diffAgentVersions | | `maitai agents restore-version` | POST | `/agents/{agent_id}/versions/{version}/restore` | restoreAgentVersion | | `maitai agents get-version` | GET | `/agents/{agent_id}/versions/{version}` | getAgentVersion | | `maitai agents list-versions` | GET | `/agents/{agent_id}/versions` | listAgentVersions | | `maitai agents patch` | PATCH | `/agents/{agent_id}` | patchAgent | ### `maitai analytics` | CLI command | Method | Path | Operation | | ---------------------------------------------------- | ------ | ------------------------------------------------------------------------- | -------------------------------------- | | `maitai analytics get-agent-request` | GET | `/analytics/agents/{agent_id}/requests` | getAgentRequestAnalytics | | `maitai analytics get-application-fault-rate` | GET | `/analytics/applications/{application_id}/fault-rate` | getApplicationFaultRateAnalytics | | `maitai analytics get-application-action-fault-rate` | GET | `/analytics/applications/{application_id}/intents/{action_id}/fault-rate` | getApplicationActionFaultRateAnalytics | | `maitai analytics get-application-action-request` | GET | `/analytics/applications/{application_id}/intents/{action_id}/requests` | getApplicationActionRequestAnalytics | | `maitai analytics get-application-request` | GET | `/analytics/applications/{application_id}/requests` | getApplicationRequestAnalytics | | `maitai analytics get-company-application-activity` | GET | `/analytics/company/application-activity` | getCompanyApplicationActivityAnalytics | | `maitai analytics get-company-model-usage` | GET | `/analytics/company/model-usage` | getCompanyModelUsageAnalytics | | `maitai analytics get-company-request` | GET | `/analytics/company/requests` | getCompanyRequestAnalytics | | `maitai analytics get-intent-group-fault-rate` | GET | `/analytics/intent-groups/{intent_group_id}/fault-rate` | getIntentGroupFaultRateAnalytics | | `maitai analytics get-intent-group-request` | GET | `/analytics/intent-groups/{intent_group_id}/requests` | getIntentGroupRequestAnalytics | ### `maitai applications` | CLI command | Method | Path | Operation | | -------------------------------------------------- | ------ | -------------------------------------------------------------------------- | --------------------------- | | `maitai applications list` | GET | `/applications` | listApplications | | `maitai applications get` | GET | `/applications/{application_id}` | getApplication | | `maitai applications create` | POST | `/applications` | createApplication | | `maitai applications update` | PUT | `/applications/{application_id}` | updateApplication | | `maitai applications delete` | DELETE | `/applications/{application_id}` | deleteApplication | | `maitai applications get-config` | GET | `/applications/{application_id}/config` | getApplicationConfig | | `maitai applications update-config` | PUT | `/applications/{application_id}/config` | updateApplicationConfig | | `maitai applications reset-intent-config` | POST | `/applications/{application_id}/intents/{intent_id}/config/reset` | resetIntentConfig | | `maitai applications get-intent-config` | GET | `/applications/{application_id}/intents/{intent_id}/config` | getIntentConfig | | `maitai applications update-intent-config` | PUT | `/applications/{application_id}/intents/{intent_id}/config` | updateIntentConfig | | `maitai applications disable-intent-notifications` | PUT | `/applications/{application_id}/intents/{intent_id}/notifications/disable` | disableIntentNotifications | | `maitai applications enable-intent-notifications` | PUT | `/applications/{application_id}/intents/{intent_id}/notifications/enable` | enableIntentNotifications | | `maitai applications delete-intent` | DELETE | `/applications/{application_id}/intents/{intent_id}` | deleteApplicationIntent | | `maitai applications get-intent` | GET | `/applications/{application_id}/intents/{intent_id}` | getApplicationIntent | | `maitai applications list-intents` | GET | `/applications/{application_id}/intents` | listApplicationIntents | | `maitai applications create-intent` | POST | `/applications/{application_id}/intents` | createApplicationIntent | | `maitai applications list-models` | GET | `/applications/{application_id}/models` | listApplicationModels | | `maitai applications list-sessions` | GET | `/applications/{application_id}/sessions` | listApplicationSessions | | `maitai applications list-workflow-runs` | GET | `/applications/{application_id}/workflow-runs` | listApplicationWorkflowRuns | ### `maitai compositions` | CLI command | Method | Path | Operation | | ------------------------------------------------ | ------ | ----------------------------------------------------------------- | ----------------------------------- | | `maitai compositions list` | GET | `/compositions` | listCompositions | | `maitai compositions get` | GET | `/compositions/{composition_id}` | getComposition | | `maitai compositions create` | POST | `/compositions` | createComposition | | `maitai compositions update` | PUT | `/compositions/{composition_id}` | updateComposition | | `maitai compositions delete` | DELETE | `/compositions/{composition_id}` | deleteComposition | | `maitai compositions list-by-application-intent` | GET | `/compositions/applications/{application_id}/intents/{intent_id}` | listCompositionsByApplicationIntent | | `maitai compositions list-by-intent-group` | GET | `/compositions/intent-groups/{intent_group_id}` | listCompositionsByIntentGroup | | `maitai compositions validate-rl` | POST | `/compositions/validate-rl` | validateRlComposition | | `maitai compositions list-versions` | GET | `/compositions/{composition_id}/versions` | listCompositionVersions | ### `maitai conversation-trees` | CLI command | Method | Path | Operation | | ----------------------------------------- | ------ | ---------------------------------------- | ---------------------------- | | `maitai conversation-trees list` | GET | `/conversation-trees` | listConversationTrees | | `maitai conversation-trees get` | GET | `/conversation-trees/{tree_id}` | getConversationTree | | `maitai conversation-trees delete` | DELETE | `/conversation-trees/{tree_id}` | deleteConversationTree | | `maitai conversation-trees get-status` | GET | `/conversation-trees/{tree_id}/status` | getConversationTreeStatus | | `maitai conversation-trees list-versions` | GET | `/conversation-trees/{tree_id}/versions` | listConversationTreeVersions | ### `maitai dashboard` | CLI command | Method | Path | Operation | | ----------------------------------- | ------ | ------------------------- | -------------- | | `maitai dashboard get-action-items` | GET | `/dashboard/action-items` | getActionItems | | `maitai dashboard get-alerts` | GET | `/dashboard/alerts` | getAlerts | ### `maitai datasets` | CLI command | Method | Path | Operation | | --------------------------------------------------------- | ------ | ------------------------------------------------------------- | ------------------------------------------ | | `maitai datasets list` | GET | `/datasets` | listDatasets | | `maitai datasets get` | GET | `/datasets/{dataset_id}` | getDataset | | `maitai datasets create` | POST | `/datasets` | createDataset | | `maitai datasets update` | PUT | `/datasets/{dataset_id}` | updateDataset | | `maitai datasets delete` | DELETE | `/datasets/{dataset_id}` | deleteDataset | | `maitai datasets list-by-intent` | GET | `/datasets/applications/{application_id}/intents/{intent_id}` | listDatasetsByIntent | | `maitai datasets list-by-tags` | GET | `/datasets/by-tags` | listDatasetsByTags | | `maitai datasets get-estimated-request-count` | GET | `/datasets/estimated-request-count` | getEstimatedDatasetRequestCount | | `maitai datasets estimate-generation-cost` | POST | `/datasets/generate/estimate-cost` | estimateDatasetGenerationCost | | `maitai datasets preview-generation` | POST | `/datasets/generate/preview` | previewDatasetGeneration | | `maitai datasets start-generation` | POST | `/datasets/generate` | startDatasetGeneration | | `maitai datasets list-by-intent-group` | GET | `/datasets/intent-groups/{intent_group_id}` | listDatasetsByIntentGroup | | `maitai datasets get-3-syncer-info` | GET | `/datasets/s3/syncer-info` | getDatasetS3SyncerInfo | | `maitai datasets validate-3-source` | POST | `/datasets/s3/validate` | validateDatasetS3Source | | `maitai datasets list-valid-tags` | GET | `/datasets/tags/valid` | listValidDatasetTags | | `maitai datasets list-by-test-set` | GET | `/datasets/test-sets/{test_set_id}` | listDatasetsByTestSet | | `maitai datasets search-added-requests` | POST | `/datasets/{dataset_id}/added-requests/search` | searchAddedDatasetRequests | | `maitai datasets clone` | POST | `/datasets/{dataset_id}/clone` | cloneDataset | | `maitai datasets get-delete-impact` | GET | `/datasets/{dataset_id}/delete-impact` | getDatasetDeleteImpact | | `maitai datasets search-eligible-requests` | POST | `/datasets/{dataset_id}/eligible-requests/search` | searchEligibleDatasetRequests | | `maitai datasets list-eligible-requests` | GET | `/datasets/{dataset_id}/eligible-requests` | listEligibleDatasetRequests | | `maitai datasets cancel-generation` | POST | `/datasets/{dataset_id}/generate/cancel` | cancelDatasetGeneration | | `maitai datasets pause-generation` | POST | `/datasets/{dataset_id}/generate/pause` | pauseDatasetGeneration | | `maitai datasets resume-generation` | POST | `/datasets/{dataset_id}/generate/resume` | resumeDatasetGeneration | | `maitai datasets get-prompt` | GET | `/datasets/{dataset_id}/prompt` | getDatasetPrompt | | `maitai datasets apply-prune` | POST | `/datasets/{dataset_id}/prune/apply` | applyDatasetPrune | | `maitai datasets preview-prune` | POST | `/datasets/{dataset_id}/prune/preview` | previewDatasetPrune | | `maitai datasets augment-reasoning` | POST | `/datasets/{dataset_id}/reasoning/augment` | augmentDatasetReasoning | | `maitai datasets get-regex-replacements-applicable-count` | POST | `/datasets/{dataset_id}/regex-replacements/count` | getDatasetRegexReplacementsApplicableCount | | `maitai datasets apply-regex-replacements` | POST | `/datasets/{dataset_id}/regex-replacements` | applyDatasetRegexReplacements | | `maitai datasets get-request-distribution` | GET | `/datasets/{dataset_id}/request-distribution` | getDatasetRequestDistribution | | `maitai datasets list-request-ids` | GET | `/datasets/{dataset_id}/request-ids` | listDatasetRequestIds | | `maitai datasets add-requests-bulk` | PUT | `/datasets/{dataset_id}/requests/add` | addDatasetRequestsBulk | | `maitai datasets remove-requests-bulk` | PUT | `/datasets/{dataset_id}/requests/remove` | removeDatasetRequestsBulk | | `maitai datasets get-request-reasoning` | GET | `/datasets/{dataset_id}/requests/{request_id}/reasoning` | getDatasetRequestReasoning | | `maitai datasets remove-requests` | DELETE | `/datasets/{dataset_id}/requests` | removeRequestsFromDataset | | `maitai datasets list-requests` | GET | `/datasets/{dataset_id}/requests` | listDatasetRequests | | `maitai datasets add-requests` | POST | `/datasets/{dataset_id}/requests` | addRequestsToDataset | | `maitai datasets cancel-3-import` | POST | `/datasets/{dataset_id}/s3/cancel` | cancelDatasetS3Import | | `maitai datasets reimport-3-source` | POST | `/datasets/{dataset_id}/s3/reimport` | reimportDatasetS3Source | | `maitai datasets sample-clone` | POST | `/datasets/{dataset_id}/sample-clone` | sampleCloneDataset | | `maitai datasets get-status` | GET | `/datasets/{dataset_id}/status` | getDatasetStatus | | `maitai datasets remove-tag` | DELETE | `/datasets/{dataset_id}/tags/{tag}` | removeDatasetTag | | `maitai datasets list-tags` | GET | `/datasets/{dataset_id}/tags` | listDatasetTags | | `maitai datasets add-tag` | POST | `/datasets/{dataset_id}/tags` | addDatasetTag | | `maitai datasets set-tags` | PUT | `/datasets/{dataset_id}/tags` | setDatasetTags | | `maitai datasets list-versions` | GET | `/datasets/{dataset_id}/versions` | listDatasetVersions | ### `maitai evaluation-criteria` | CLI command | Method | Path | Operation | | ------------------------------------------- | ------ | ------------------------------------ | ------------------------------- | | `maitai evaluation-criteria list` | GET | `/evaluation-criteria` | listEvaluationCriteria | | `maitai evaluation-criteria create` | POST | `/evaluation-criteria` | createEvaluationCriteria | | `maitai evaluation-criteria update` | PUT | `/evaluation-criteria/{criteria_id}` | updateEvaluationCriteria | | `maitai evaluation-criteria delete` | DELETE | `/evaluation-criteria/{criteria_id}` | deleteEvaluationCriteria | | `maitai evaluation-criteria list-available` | GET | `/evaluation-criteria/available` | listAvailableEvaluationCriteria | ### `maitai evaluations` | CLI command | Method | Path | Operation | | -------------------------------- | ------ | ------------------------------------------ | -------------------- | | `maitai evaluations list` | GET | `/evaluations` | listEvaluations | | `maitai evaluations get` | GET | `/evaluations/{evaluation_run_id}` | getEvaluation | | `maitai evaluations create` | POST | `/evaluations` | createEvaluation | | `maitai evaluations get-results` | GET | `/evaluations/{evaluation_run_id}/results` | getEvaluationResults | ### `maitai external-test-runs` | CLI command | Method | Path | Operation | | ---------------------------------------------- | ------ | ------------------------------------------------------ | --------------------- | | `maitai external-test-runs create` | POST | `/external-test-runs` | createExternalTestRun | | `maitai external-test-runs lease-next-case` | GET | `/external-test-runs/{test_run_id}/next` | leaseNextCase | | `maitai external-test-runs submit-case-result` | POST | `/external-test-runs/{test_run_id}/{case_hash}/result` | submitCaseResult | ### `maitai finetune-runs` | CLI command | Method | Path | Operation | | ------------------------------------ | ------ | --------------------------------------- | --------------------- | | `maitai finetune-runs list` | GET | `/finetune-runs` | listFinetuneRuns | | `maitai finetune-runs get` | GET | `/finetune-runs/{run_id}` | getFinetuneRun | | `maitai finetune-runs create` | POST | `/finetune-runs` | createFinetuneRun | | `maitai finetune-runs estimate-vram` | POST | `/finetune-runs/estimate` | estimateFinetuneVram | | `maitai finetune-runs cancel` | POST | `/finetune-runs/{run_id}/cancel` | cancelFinetuneRun | | `maitai finetune-runs cancel-deploy` | POST | `/finetune-runs/{run_id}/deploy/cancel` | cancelFinetuneDeploy | | `maitai finetune-runs deploy` | POST | `/finetune-runs/{run_id}/deploy` | deployFinetuneRun | | `maitai finetune-runs get-metrics` | GET | `/finetune-runs/{run_id}/metrics` | getFinetuneRunMetrics | | `maitai finetune-runs get-status` | GET | `/finetune-runs/{run_id}/status` | getFinetuneStatus | ### `maitai help` | CLI command | Method | Path | Operation | | -------------------- | ------ | ------- | --------- | | `maitai help create` | POST | `/help` | helpQuery | ### `maitai intent-groups` | CLI command | Method | Path | Operation | | ---------------------------------------- | ------ | ----------------------------------------------- | --------------------------- | | `maitai intent-groups list` | GET | `/intent-groups` | listIntentGroups | | `maitai intent-groups get` | GET | `/intent-groups/{intent_group_id}` | getIntentGroup | | `maitai intent-groups list-compositions` | GET | `/intent-groups/{intent_group_id}/compositions` | listIntentGroupCompositions | | `maitai intent-groups get-config` | GET | `/intent-groups/{intent_group_id}/config` | getIntentGroupConfig | | `maitai intent-groups update-config` | PUT | `/intent-groups/{intent_group_id}/config` | updateIntentGroupConfig | | `maitai intent-groups list-datasets` | GET | `/intent-groups/{intent_group_id}/datasets` | listIntentGroupDatasets | | `maitai intent-groups list-intents` | GET | `/intent-groups/{intent_group_id}/intents` | listIntentsByGroup | | `maitai intent-groups list-models` | GET | `/intent-groups/{intent_group_id}/models` | listIntentGroupModels | | `maitai intent-groups list-requests` | GET | `/intent-groups/{intent_group_id}/requests` | listIntentGroupRequests | | `maitai intent-groups list-sentinels` | GET | `/intent-groups/{intent_group_id}/sentinels` | listIntentGroupSentinels | | `maitai intent-groups list-test-sets` | GET | `/intent-groups/{intent_group_id}/test-sets` | listIntentGroupTestSets | ### `maitai models` | CLI command | Method | Path | Operation | | ----------------------------------------------------- | ------ | ----------------------------------------------------- | -------------------------------------- | | `maitai models list` | GET | `/models` | listModels | | `maitai models get` | GET | `/models/{model_id}` | getModel | | `maitai models list-by-application` | GET | `/models/applications/{application_id}` | listModelsByApplication | | `maitai models list-available` | GET | `/models/available` | listAvailableModels | | `maitai models create-base` | POST | `/models/base-model` | createBaseModel | | `maitai models list-by-intent-group` | GET | `/models/intent-groups/{intent_group_id}` | listModelsByIntentGroup | | `maitai models list-by-intent` | GET | `/models/intents/{intent_id}` | listModelsByIntent | | `maitai models disable` | POST | `/models/{model_id}/disable` | disableModel | | `maitai models enable` | POST | `/models/{model_id}/enable` | enableModel | | `maitai models list-inference-pool-attach-candidates` | GET | `/models/{model_id}/inference-pool/attach-candidates` | listModelInferencePoolAttachCandidates | | `maitai models attach-inference-pool` | POST | `/models/{model_id}/inference-pool/attach` | attachModelInferencePool | | `maitai models get-inference-pool-debug` | GET | `/models/{model_id}/inference-pool/debug` | getModelInferencePoolDebug | | `maitai models detach-inference-pool` | POST | `/models/{model_id}/inference-pool/detach` | detachModelInferencePool | | `maitai models get-inference-pool` | GET | `/models/{model_id}/inference-pool` | getModelInferencePool | | `maitai models patch-registry` | PATCH | `/models/{model_id}/registry` | patchModelRegistry | | `maitai models get-repoint-options` | GET | `/models/{model_id}/repoint-options` | getModelRepointOptions | | `maitai models repoint` | POST | `/models/{model_id}/repoint` | repointModel | | `maitai models execute-swap-ref` | POST | `/models/{model_id}/swap-model-ref/execute` | executeSwapModelRef | | `maitai models preview-swap-ref` | POST | `/models/{model_id}/swap-model-ref/preview` | previewSwapModelRef | ### `maitai monitors` | CLI command | Method | Path | Operation | | -------------------------------------------------------------------------- | ------ | -------------------------------------------------------------- | ------------------------- | | `maitai monitors list` | GET | `/monitors` | listMonitors | | `maitai monitors get` | GET | `/monitors/{monitor_id}` | getMonitor | | `maitai monitors create` | POST | `/monitors` | createMonitor | | `maitai monitors update` | PUT | `/monitors/{monitor_id}` | updateMonitor | | `maitai monitors delete` | DELETE | `/monitors/{monitor_id}` | deleteMonitor | | `maitai monitors list-by-target` | GET | `/monitors/by-target` | listMonitorsByTarget | | `maitai monitors list-runs-by-source` | GET | `/monitors/runs/by-source` | listRunsBySource | | `maitai monitors get-run-global` | GET | `/monitors/runs/{run_id}` | getRunGlobal | | `maitai monitors list-runs-global` | GET | `/monitors/runs` | listRunsGlobal | | `maitai monitors get-target-sample` | GET | `/monitors/sample-target` | getTargetSample | | `maitai monitors activate` | POST | `/monitors/{monitor_id}/activate` | activateMonitor | | `maitai monitors get-activity` | GET | `/monitors/{monitor_id}/activity` | getMonitorActivity | | `maitai monitors get-metrics-by-target` | GET | `/monitors/{monitor_id}/metrics/by-target` | getMonitorMetricsByTarget | | `maitai monitors get-metrics` | GET | `/monitors/{monitor_id}/metrics` | getMonitorMetrics | | `maitai monitors pause` | POST | `/monitors/{monitor_id}/pause` | pauseMonitor | | `maitai monitors preview` | POST | `/monitors/{monitor_id}/preview` | previewMonitor | | `maitai monitors get-default-version` | GET | `/monitors/{monitor_id}/releases/default` | getDefaultMonitorVersion | | `maitai monitors promote-release` | POST | `/monitors/{monitor_id}/releases/{release_name}/promote` | promoteMonitorRelease | | `maitai monitors delete-release` | DELETE | `/monitors/{monitor_id}/releases/{release_name}` | deleteMonitorRelease | | `maitai monitors list-releases` | GET | `/monitors/{monitor_id}/releases` | listMonitorReleases | | `maitai monitors upsert-release` | POST | `/monitors/{monitor_id}/releases` | upsertMonitorRelease | | `maitai monitors upsert-release-put-monitor-id-releases` | PUT | `/monitors/{monitor_id}/releases` | upsertMonitorRelease | | `maitai monitors get-run-scoped` | GET | `/monitors/{monitor_id}/runs/{run_id}` | getRunScoped | | `maitai monitors list-runs` | GET | `/monitors/{monitor_id}/runs` | listRunsForMonitor | | `maitai monitors disable-target` | POST | `/monitors/{monitor_id}/targets/{monitor_target_id}/disable` | disableMonitorTarget | | `maitai monitors enable-target` | POST | `/monitors/{monitor_id}/targets/{monitor_target_id}/enable` | enableMonitorTarget | | `maitai monitors get-target-metrics` | GET | `/monitors/{monitor_id}/targets/{monitor_target_id}/metrics` | getMonitorTargetMetrics | | `maitai monitors get-target-sparkline` | GET | `/monitors/{monitor_id}/targets/{monitor_target_id}/sparkline` | getMonitorTargetSparkline | | `maitai monitors delete-target` | DELETE | `/monitors/{monitor_id}/targets/{monitor_target_id}` | deleteMonitorTarget | | `maitai monitors update-target-patch-monitor-id-targets-monitor-target-id` | PATCH | `/monitors/{monitor_id}/targets/{monitor_target_id}` | updateMonitorTarget | | `maitai monitors update-target` | PUT | `/monitors/{monitor_id}/targets/{monitor_target_id}` | updateMonitorTarget | | `maitai monitors list-targets` | GET | `/monitors/{monitor_id}/targets` | listMonitorTargets | | `maitai monitors create-target` | POST | `/monitors/{monitor_id}/targets` | createMonitorTarget | | `maitai monitors get-version` | GET | `/monitors/{monitor_id}/versions/{version_id}` | getMonitorVersion | | `maitai monitors list-versions` | GET | `/monitors/{monitor_id}/versions` | listMonitorVersions | | `maitai monitors publish-version` | POST | `/monitors/{monitor_id}/versions` | publishMonitorVersion | | `maitai monitors patch` | PATCH | `/monitors/{monitor_id}` | updateMonitor | ### `maitai reports` | CLI command | Method | Path | Operation | | ----------------------------------------- | ------ | ---------------------------------------- | --------------------- | | `maitai reports list-fallbacks` | GET | `/reports/fallbacks` | listFallbacks | | `maitai reports list-faults` | GET | `/reports/faults` | listFaults | | `maitai reports list-fallbacks-by-intent` | GET | `/reports/intents/{intent_id}/fallbacks` | listFallbacksByIntent | | `maitai reports list-faults-by-intent` | GET | `/reports/intents/{intent_id}/faults` | listFaultsByIntent | ### `maitai requests` | CLI command | Method | Path | Operation | | --------------------------------- | ------ | --------------------------------- | --------------------- | | `maitai requests list` | GET | `/requests` | listRequests | | `maitai requests get` | GET | `/requests/{request_id}` | getRequest | | `maitai requests update-response` | PUT | `/requests/{request_id}/response` | updateRequestResponse | ### `maitai search` | CLI command | Method | Path | Operation | | ---------------------- | ------ | --------- | --------- | | `maitai search create` | POST | `/search` | search | ### `maitai sentinels` | CLI command | Method | Path | Operation | | --------------------------------------------------------- | ------ | ------------------------------------------------------ | ------------------------------------ | | `maitai sentinels list` | GET | `/sentinels` | listSentinels | | `maitai sentinels get` | GET | `/sentinels/{sentinel_id}` | getSentinel | | `maitai sentinels create` | POST | `/sentinels` | createSentinel | | `maitai sentinels update` | PUT | `/sentinels/{sentinel_id}` | updateSentinel | | `maitai sentinels delete` | DELETE | `/sentinels/{sentinel_id}` | deleteSentinel | | `maitai sentinels list-by-application-human-needs` | GET | `/sentinels/applications/{application_id}/human-needs` | listSentinelsByApplicationHumanNeeds | | `maitai sentinels list-by-application` | GET | `/sentinels/applications/{application_id}` | listSentinelsByApplication | | `maitai sentinels generate-new` | POST | `/sentinels/generate/new` | generateNewSentinel | | `maitai sentinels generate` | POST | `/sentinels/generate` | generateSentinels | | `maitai sentinels list-by-intent-name` | GET | `/sentinels/intent-groups/by-name/{intent_name}` | listSentinelsByIntentName | | `maitai sentinels list-by-intent-group` | GET | `/sentinels/intent-groups/{intent_group_id}` | listSentinelsByIntentGroup | | `maitai sentinels regenerate` | POST | `/sentinels/regenerate` | regenerateSentinels | | `maitai sentinels list-simple` | GET | `/sentinels/simple` | listSimpleSentinels | | `maitai sentinels update-directive` | PUT | `/sentinels/{sentinel_id}/directive` | updateSentinelDirective | | `maitai sentinels regenerate-correction` | POST | `/sentinels/{sentinel_id}/regenerate/correction` | regenerateSentinelCorrection | | `maitai sentinels regenerate-evaluation` | POST | `/sentinels/{sentinel_id}/regenerate/evaluation` | regenerateSentinelEvaluation | | `maitai sentinels regenerate-qualification` | POST | `/sentinels/{sentinel_id}/regenerate/qualification` | regenerateSentinelQualification | | `maitai sentinels regenerate-post-sentinel-id-regenerate` | POST | `/sentinels/{sentinel_id}/regenerate` | regenerateSentinel | ### `maitai sessions` | CLI command | Method | Path | Operation | | ------------------------------ | ------ | --------------------------------- | ------------------ | | `maitai sessions list` | GET | `/sessions` | listSessions | | `maitai sessions get` | GET | `/sessions/{session_id}` | getSession | | `maitai sessions set-feedback` | POST | `/sessions/{session_id}/feedback` | setSessionFeedback | | `maitai sessions get-timeline` | GET | `/sessions/{session_id}/timeline` | getSessionTimeline | ### `maitai test-runs` | CLI command | Method | Path | Operation | | --------------------------------------------- | ------ | -------------------------------------------------------------------- | ------------------------------ | | `maitai test-runs list` | GET | `/test-runs` | listTestRuns | | `maitai test-runs get` | GET | `/test-runs/{test_run_id}` | getTestRun | | `maitai test-runs create` | POST | `/test-runs` | createTestRun | | `maitai test-runs delete` | DELETE | `/test-runs/{test_run_id}` | deleteTestRun | | `maitai test-runs compare` | POST | `/test-runs/compare` | compareTestRuns | | `maitai test-runs list-by-intent-group` | GET | `/test-runs/intent-groups/{intent_group_id}` | listTestRunsByIntentGroup | | `maitai test-runs get-request-comparison` | GET | `/test-runs/requests/compare/{base_request_id}/{compare_request_id}` | getTestRunRequestComparison | | `maitai test-runs get-progress-by-set` | GET | `/test-runs/test-sets/{test_set_id}/progress` | getTestRunProgressByTestSet | | `maitai test-runs list-by-set` | GET | `/test-runs/test-sets/{test_set_id}` | listTestRunsByTestSet | | `maitai test-runs evaluate` | POST | `/test-runs/{test_run_id}/evaluate` | evaluateTestRun | | `maitai test-runs update-item` | PUT | `/test-runs/{test_run_id}/items/{item_id}` | updateTestRunItem | | `maitai test-runs list-items` | GET | `/test-runs/{test_run_id}/items` | listTestRunItems | | `maitai test-runs get-progress` | GET | `/test-runs/{test_run_id}/progress` | getTestRunProgress | | `maitai test-runs re-evaluate` | POST | `/test-runs/{test_run_id}/re-evaluate` | reEvaluateTestRun | | `maitai test-runs list-requests-with-filters` | POST | `/test-runs/{test_run_id}/requests/{offset}/{limit}` | listTestRunRequestsWithFilters | | `maitai test-runs evaluate-request` | POST | `/test-runs/{test_run_id}/requests/{test_run_request_id}/evaluate` | evaluateTestRunRequest | | `maitai test-runs list-requests` | GET | `/test-runs/{test_run_id}/requests` | listTestRunRequests | | `maitai test-runs get-results` | GET | `/test-runs/{test_run_id}/results` | getTestRunResults | | `maitai test-runs resume` | POST | `/test-runs/{test_run_id}/resume` | resumeTestRun | | `maitai test-runs retry` | POST | `/test-runs/{test_run_id}/retry` | retryTestRun | | `maitai test-runs list-review-segments` | GET | `/test-runs/{test_run_id}/review-segments` | listTestRunReviewSegments | | `maitai test-runs update-status` | PUT | `/test-runs/{test_run_id}/status/{status}` | updateTestRunStatus | | `maitai test-runs stop` | POST | `/test-runs/{test_run_id}/stop` | stopTestRun | ### `maitai test-set-jobs` | CLI command | Method | Path | Operation | | -------------------------- | ------ | ------------------------- | ------------- | | `maitai test-set-jobs get` | GET | `/test-set-jobs/{job_id}` | getTestSetJob | ### `maitai test-sets` | CLI command | Method | Path | Operation | | ----------------------------------------------- | ------ | --------------------------------------------------------------------- | -------------------------------- | | `maitai test-sets list` | GET | `/test-sets` | listTestSets | | `maitai test-sets get` | GET | `/test-sets/{test_set_id}` | getTestSet | | `maitai test-sets create` | POST | `/test-sets` | createTestSet | | `maitai test-sets update` | PUT | `/test-sets/{test_set_id}` | updateTestSet | | `maitai test-sets delete` | DELETE | `/test-sets/{test_set_id}` | deleteTestSet | | `maitai test-sets list-by-application-intent` | GET | `/test-sets/applications/{application_id}/intents/{intent_id}` | listTestSetsByApplicationIntent | | `maitai test-sets convert-to-workflow` | POST | `/test-sets/convert-to-workflow` | convertTestSetToWorkflow | | `maitai test-sets convert-workflow-to-requests` | POST | `/test-sets/convert-workflow-to-requests` | convertWorkflowTestSetToRequests | | `maitai test-sets list-created-from-dataset` | GET | `/test-sets/datasets/{dataset_id}/source` | listTestSetsCreatedFromDataset | | `maitai test-sets list-by-dataset` | GET | `/test-sets/datasets/{dataset_id}` | listTestSetsByDataset | | `maitai test-sets search-eligible-requests` | POST | `/test-sets/eligible-requests/{intent_group_id}/{test_set_id}/search` | searchEligibleTestSetRequests | | `maitai test-sets list-eligible-requests` | GET | `/test-sets/eligible-requests/{intent_group_id}/{test_set_id}` | listEligibleTestSetRequests | | `maitai test-sets list-by-intent-group-name` | GET | `/test-sets/intent-groups/by-name/{intent_group_name}` | listTestSetsByIntentGroupName | | `maitai test-sets list-by-intent-group` | GET | `/test-sets/intent-groups/{intent_group_id}` | listTestSetsByIntentGroup | | `maitai test-sets add-single-request` | PUT | `/test-sets/request/add` | addSingleRequestToTestSet | | `maitai test-sets remove-single-request` | DELETE | `/test-sets/request/remove` | removeSingleRequestFromTestSet | | `maitai test-sets add-requests-bulk` | PUT | `/test-sets/requests/add` | addTestSetRequestsBulk | | `maitai test-sets remove-requests-bulk` | PUT | `/test-sets/requests/remove` | removeTestSetRequestsBulk | | `maitai test-sets clone` | POST | `/test-sets/{test_set_id}/clone` | cloneTestSet | | `maitai test-sets list-imported-session-ids` | GET | `/test-sets/{test_set_id}/imported-session-ids` | listTestSetImportedSessionIds | | `maitai test-sets delete-item-response-sub` | DELETE | `/test-sets/{test_set_id}/items/{item_id}/response-sub` | deleteTestSetItemResponseSub | | `maitai test-sets get-item-response-sub` | GET | `/test-sets/{test_set_id}/items/{item_id}/response-sub` | getTestSetItemResponseSub | | `maitai test-sets upsert-item-response-sub` | PUT | `/test-sets/{test_set_id}/items/{item_id}/response-sub` | upsertTestSetItemResponseSub | | `maitai test-sets delete-item` | DELETE | `/test-sets/{test_set_id}/items/{item_id}` | deleteTestSetItem | | `maitai test-sets remove-items` | DELETE | `/test-sets/{test_set_id}/items` | removeTestSetItems | | `maitai test-sets list-items` | GET | `/test-sets/{test_set_id}/items` | listTestSetItems | | `maitai test-sets add-items` | POST | `/test-sets/{test_set_id}/items` | addTestSetItems | | `maitai test-sets get-prompt-template` | GET | `/test-sets/{test_set_id}/prompt-template` | getTestSetPromptTemplate | | `maitai test-sets get-request-distribution` | GET | `/test-sets/{test_set_id}/request-distribution` | getTestSetRequestDistribution | | `maitai test-sets update-request-response` | PUT | `/test-sets/{test_set_id}/requests/{request_id}/response` | updateTestSetRequestResponse | | `maitai test-sets update-request-tags` | PUT | `/test-sets/{test_set_id}/requests/{request_id}/tags` | updateTestSetRequestTags | | `maitai test-sets remove-requests` | DELETE | `/test-sets/{test_set_id}/requests` | removeRequestsFromTestSet | | `maitai test-sets list-requests` | GET | `/test-sets/{test_set_id}/requests` | listTestSetRequests | | `maitai test-sets add-requests` | POST | `/test-sets/{test_set_id}/requests` | addRequestsToTestSet | | `maitai test-sets list-session-facets` | GET | `/test-sets/{test_set_id}/sessions` | listTestSetSessionFacets | ### `maitai unified-test-runs` | CLI command | Method | Path | Operation | | ------------------------------------------------ | ------ | ----------------------------------------------- | ------------------------------ | | `maitai unified-test-runs list` | GET | `/unified-test-runs` | listUnifiedTestRuns | | `maitai unified-test-runs get` | GET | `/unified-test-runs/{test_run_id}` | getUnifiedTestRun | | `maitai unified-test-runs create` | POST | `/unified-test-runs` | createUnifiedTestRun | | `maitai unified-test-runs delete` | DELETE | `/unified-test-runs/{test_run_id}` | deleteUnifiedTestRun | | `maitai unified-test-runs preview-adapted-input` | POST | `/unified-test-runs/preview-adapted-input` | previewAdaptedInput | | `maitai unified-test-runs cancel` | POST | `/unified-test-runs/{test_run_id}/cancel` | cancelUnifiedTestRun | | `maitai unified-test-runs execute` | POST | `/unified-test-runs/{test_run_id}/execute` | executeUnifiedTestRun | | `maitai unified-test-runs list-items` | GET | `/unified-test-runs/{test_run_id}/items` | listUnifiedTestRunItems | | `maitai unified-test-runs prepare` | POST | `/unified-test-runs/{test_run_id}/prepare` | prepareUnifiedTestRun | | `maitai unified-test-runs get-progress` | GET | `/unified-test-runs/{test_run_id}/progress` | getUnifiedTestRunProgress | | `maitai unified-test-runs reset-failed-items` | POST | `/unified-test-runs/{test_run_id}/reset-failed` | resetFailedUnifiedTestRunItems | | `maitai unified-test-runs score` | POST | `/unified-test-runs/{test_run_id}/score` | scoreUnifiedTestRun | ### `maitai unified-test-sets` | CLI command | Method | Path | Operation | | --------------------------------------------------------------- | ------ | --------------------------------------------------------------------------- | ---------------------------------------------- | | `maitai unified-test-sets list` | GET | `/unified-test-sets` | listUnifiedTestSets | | `maitai unified-test-sets get` | GET | `/unified-test-sets/{test_set_id}` | getUnifiedTestSet | | `maitai unified-test-sets create` | POST | `/unified-test-sets` | createUnifiedTestSet | | `maitai unified-test-sets upload-jsonl` | POST | `/unified-test-sets/upload` | uploadUnifiedTestSetJsonl | | `maitai unified-test-sets clone` | POST | `/unified-test-sets/{test_set_id}/clone` | cloneUnifiedTestSet | | `maitai unified-test-sets import-agent-task-into` | POST | `/unified-test-sets/{test_set_id}/import-agent-task` | importAgentTaskIntoUnifiedTestSet | | `maitai unified-test-sets import-request-into` | POST | `/unified-test-sets/{test_set_id}/import-request` | importRequestIntoUnifiedTestSet | | `maitai unified-test-sets import-workflow-run-into` | POST | `/unified-test-sets/{test_set_id}/import-workflow-run` | importWorkflowRunIntoUnifiedTestSet | | `maitai unified-test-sets upsert-item-expected-output-override` | PUT | `/unified-test-sets/{test_set_id}/items/{item_id}/expected-output-override` | upsertUnifiedTestSetItemExpectedOutputOverride | | `maitai unified-test-sets delete-item` | DELETE | `/unified-test-sets/{test_set_id}/items/{item_id}` | deleteUnifiedTestSetItem | | `maitai unified-test-sets get-item` | GET | `/unified-test-sets/{test_set_id}/items/{item_id}` | getUnifiedTestSetItem | | `maitai unified-test-sets list-items` | GET | `/unified-test-sets/{test_set_id}/items` | listUnifiedTestSetItems | | `maitai unified-test-sets create-manual-item` | POST | `/unified-test-sets/{test_set_id}/items` | createUnifiedManualTestSetItem | | `maitai unified-test-sets patch` | PATCH | `/unified-test-sets/{test_set_id}` | patchUnifiedTestSet | ### `maitai workflows` | CLI command | Method | Path | Operation | | --------------------------------------------- | ------ | ------------------------------------------------------------------- | ------------------------------- | | `maitai workflows list` | GET | `/workflows` | listWorkflows | | `maitai workflows get` | GET | `/workflows/{workflow_id}` | getWorkflow | | `maitai workflows create` | POST | `/workflows` | createWorkflow | | `maitai workflows update` | PUT | `/workflows/{workflow_id}` | updateWorkflow | | `maitai workflows delete` | DELETE | `/workflows/{workflow_id}` | deleteWorkflow | | `maitai workflows create-chat-completion` | POST | `/workflows/chat/completions` | createWorkflowChatCompletion | | `maitai workflows get-default-script` | GET | `/workflows/default-script` | getDefaultWorkflowScript | | `maitai workflows log-run` | PUT | `/workflows/runs/log` | logWorkflowRun | | `maitai workflows get-run` | GET | `/workflows/runs/{session_id}` | getWorkflowRun | | `maitai workflows list-runs` | GET | `/workflows/runs` | listWorkflowRuns | | `maitai workflows upload-bundle` | POST | `/workflows/upload` | uploadWorkflowBundle | | `maitai workflows validate-script` | POST | `/workflows/validate` | validateWorkflowScript | | `maitai workflows delete-accessory` | DELETE | `/workflows/{workflow_id}/accessories` | deleteWorkflowAccessory | | `maitai workflows upload-accessory` | POST | `/workflows/{workflow_id}/accessories` | uploadWorkflowAccessory | | `maitai workflows get-artifact-files` | GET | `/workflows/{workflow_id}/artifact-files` | getWorkflowArtifactFiles | | `maitai workflows upload-datastore` | POST | `/workflows/{workflow_id}/datastores/{name}/upload` | uploadWorkflowDatastore | | `maitai workflows delete-datastore` | DELETE | `/workflows/{workflow_id}/datastores/{name}` | deleteWorkflowDatastore | | `maitai workflows upsert-datastore` | POST | `/workflows/{workflow_id}/datastores/{name}` | upsertWorkflowDatastore | | `maitai workflows list-datastores` | GET | `/workflows/{workflow_id}/datastores` | listWorkflowDatastores | | `maitai workflows check-transforms` | GET | `/workflows/{workflow_id}/has-transforms` | checkWorkflowTransforms | | `maitai workflows clear-default-release` | POST | `/workflows/{workflow_id}/releases/clear-default` | clearWorkflowDefaultRelease | | `maitai workflows delete-release` | DELETE | `/workflows/{workflow_id}/releases/{name}` | deleteWorkflowRelease | | `maitai workflows update-release` | PUT | `/workflows/{workflow_id}/releases/{name}` | updateWorkflowRelease | | `maitai workflows list-releases` | GET | `/workflows/{workflow_id}/releases` | listWorkflowReleases | | `maitai workflows create-release` | POST | `/workflows/{workflow_id}/releases` | createWorkflowRelease | | `maitai workflows get-schema` | GET | `/workflows/{workflow_id}/schema` | getWorkflowSchema | | `maitai workflows get-script` | GET | `/workflows/{workflow_id}/script` | getWorkflowScript | | `maitai workflows get-transform-info` | GET | `/workflows/{workflow_id}/transform-info` | getWorkflowTransformInfo | | `maitai workflows preview-transform` | POST | `/workflows/{workflow_id}/transform-preview` | previewWorkflowTransform | | `maitai workflows get-version-artifact-files` | GET | `/workflows/{workflow_id}/versions/{version_number}/artifact-files` | getWorkflowVersionArtifactFiles | | `maitai workflows list-versions` | GET | `/workflows/{workflow_id}/versions` | listWorkflowVersions | | `maitai workflows publish-version` | POST | `/workflows/{workflow_id}/versions` | publishWorkflowVersion | ## All endpoints The CLI wraps all api/v1 endpoints. Use `maitai api METHOD path` for any endpoint: | Method | Path | Operation | | ------ | --------------------------------------------------------------------------- | ---------------------------------------------- | | GET | `/agents` | listAgents | | POST | `/agents` | createAgent | | DELETE | `/agents/actions/{action_id}` | deleteAgentAction | | GET | `/agents/actions/{action_id}` | getAgentAction | | PATCH | `/agents/actions/{action_id}` | patchAgentAction | | PUT | `/agents/actions/{action_id}` | updateAgentAction | | PUT | `/agents/actions/{action_id}/disable` | disableAction | | PUT | `/agents/actions/{action_id}/enable` | enableAction | | POST | `/agents/actions/{action_id}/publish` | publishActionVersion | | GET | `/agents/actions/{action_id}/versions` | listActionVersions | | POST | `/agents/actions/{action_id}/versions/{version}/restore` | restoreActionVersion | | POST | `/agents/import` | importAgents | | GET | `/agents/supplied-actions` | listSuppliedActions | | GET | `/agents/tasks/{task_id}/tree` | getAgentTaskTree | | DELETE | `/agents/{agent_id}` | deleteAgent | | GET | `/agents/{agent_id}` | getAgent | | PATCH | `/agents/{agent_id}` | patchAgent | | PUT | `/agents/{agent_id}` | updateAgent | | GET | `/agents/{agent_id}/actions` | listAgentActions | | POST | `/agents/{agent_id}/actions` | createAgentAction | | POST | `/agents/{agent_id}/actions/supplied/{catalog_action_id}` | addSuppliedAction | | DELETE | `/agents/{agent_id}/form-fields` | deleteFormFields | | GET | `/agents/{agent_id}/form-fields` | getFormFields | | PUT | `/agents/{agent_id}/form-fields` | updateFormFields | | GET | `/agents/{agent_id}/invocations/{task_id}` | getAgentInvocation | | POST | `/agents/{agent_id}/invoke` | invokeAgent | | POST | `/agents/{agent_id}/publish` | publishAgentVersion | | GET | `/agents/{agent_id}/releases` | listAgentReleases | | DELETE | `/agents/{agent_id}/releases/{name}` | deleteAgentRelease | | PUT | `/agents/{agent_id}/releases/{name}` | upsertAgentRelease | | GET | `/agents/{agent_id}/routing/config` | getRoutingConfig | | PUT | `/agents/{agent_id}/routing/config` | updateRoutingConfig | | GET | `/agents/{agent_id}/routing/results/{routing_result_id}` | getRoutingResult | | POST | `/agents/{agent_id}/routing/rules` | createRoutingRule | | DELETE | `/agents/{agent_id}/routing/rules/{rule_id}` | deleteRoutingRule | | PUT | `/agents/{agent_id}/routing/rules/{rule_id}` | updateRoutingRule | | PUT | `/agents/{agent_id}/routing/rules/{rule_id}/reorder` | reorderRoutingRule | | POST | `/agents/{agent_id}/routing/rules/{rule_id}/test` | testRoutingRule | | GET | `/agents/{agent_id}/sessions` | listAgentSessions | | GET | `/agents/{agent_id}/sessions/{session_id}` | getAgentSessionDetail | | GET | `/agents/{agent_id}/sessions/{session_id}/timeline` | getAgentSessionTimeline | | GET | `/agents/{agent_id}/status` | getAgentStatus | | GET | `/agents/{agent_id}/subagents` | listSubAgents | | POST | `/agents/{agent_id}/subagents` | addSubAgent | | DELETE | `/agents/{agent_id}/subagents/{child_agent_id}` | removeSubAgent | | PUT | `/agents/{agent_id}/subagents/{child_agent_id}/disable` | disableSubAgent | | PUT | `/agents/{agent_id}/subagents/{child_agent_id}/enable` | enableSubAgent | | GET | `/agents/{agent_id}/system-steps` | listAgentSystemSteps | | DELETE | `/agents/{agent_id}/system-steps/{step}/config` | resetAgentSystemStepConfig | | GET | `/agents/{agent_id}/system-steps/{step}/config` | getAgentSystemStepConfig | | PUT | `/agents/{agent_id}/system-steps/{step}/config` | updateAgentSystemStepConfig | | PUT | `/agents/{agent_id}/system-steps/{step}/disable` | disableAgentSystemStep | | PUT | `/agents/{agent_id}/system-steps/{step}/enable` | enableAgentSystemStep | | DELETE | `/agents/{agent_id}/system-steps/{step}/prompt` | resetAgentSystemStepPrompt | | GET | `/agents/{agent_id}/system-steps/{step}/prompt` | getAgentSystemStepPrompt | | PUT | `/agents/{agent_id}/system-steps/{step}/prompt` | updateAgentSystemStepPrompt | | GET | `/agents/{agent_id}/tasks/{task_id}/timeline` | getAgentTaskTimeline | | GET | `/agents/{agent_id}/versions` | listAgentVersions | | GET | `/agents/{agent_id}/versions/diff` | diffAgentVersions | | GET | `/agents/{agent_id}/versions/{version}` | getAgentVersion | | POST | `/agents/{agent_id}/versions/{version}/restore` | restoreAgentVersion | | GET | `/analytics/agents/{agent_id}/requests` | getAgentRequestAnalytics | | GET | `/analytics/applications/{application_id}/fault-rate` | getApplicationFaultRateAnalytics | | GET | `/analytics/applications/{application_id}/intents/{action_id}/fault-rate` | getApplicationActionFaultRateAnalytics | | GET | `/analytics/applications/{application_id}/intents/{action_id}/requests` | getApplicationActionRequestAnalytics | | GET | `/analytics/applications/{application_id}/requests` | getApplicationRequestAnalytics | | GET | `/analytics/company/application-activity` | getCompanyApplicationActivityAnalytics | | GET | `/analytics/company/model-usage` | getCompanyModelUsageAnalytics | | GET | `/analytics/company/requests` | getCompanyRequestAnalytics | | GET | `/analytics/intent-groups/{intent_group_id}/fault-rate` | getIntentGroupFaultRateAnalytics | | GET | `/analytics/intent-groups/{intent_group_id}/requests` | getIntentGroupRequestAnalytics | | GET | `/applications` | listApplications | | POST | `/applications` | createApplication | | DELETE | `/applications/{application_id}` | deleteApplication | | GET | `/applications/{application_id}` | getApplication | | PUT | `/applications/{application_id}` | updateApplication | | GET | `/applications/{application_id}/config` | getApplicationConfig | | PUT | `/applications/{application_id}/config` | updateApplicationConfig | | GET | `/applications/{application_id}/intents` | listApplicationIntents | | POST | `/applications/{application_id}/intents` | createApplicationIntent | | DELETE | `/applications/{application_id}/intents/{intent_id}` | deleteApplicationIntent | | GET | `/applications/{application_id}/intents/{intent_id}` | getApplicationIntent | | GET | `/applications/{application_id}/intents/{intent_id}/config` | getIntentConfig | | PUT | `/applications/{application_id}/intents/{intent_id}/config` | updateIntentConfig | | POST | `/applications/{application_id}/intents/{intent_id}/config/reset` | resetIntentConfig | | PUT | `/applications/{application_id}/intents/{intent_id}/notifications/disable` | disableIntentNotifications | | PUT | `/applications/{application_id}/intents/{intent_id}/notifications/enable` | enableIntentNotifications | | GET | `/applications/{application_id}/models` | listApplicationModels | | GET | `/applications/{application_id}/sessions` | listApplicationSessions | | GET | `/applications/{application_id}/workflow-runs` | listApplicationWorkflowRuns | | GET | `/compositions` | listCompositions | | POST | `/compositions` | createComposition | | GET | `/compositions/applications/{application_id}/intents/{intent_id}` | listCompositionsByApplicationIntent | | GET | `/compositions/intent-groups/{intent_group_id}` | listCompositionsByIntentGroup | | POST | `/compositions/validate-rl` | validateRlComposition | | DELETE | `/compositions/{composition_id}` | deleteComposition | | GET | `/compositions/{composition_id}` | getComposition | | PUT | `/compositions/{composition_id}` | updateComposition | | GET | `/compositions/{composition_id}/versions` | listCompositionVersions | | GET | `/conversation-trees` | listConversationTrees | | DELETE | `/conversation-trees/{tree_id}` | deleteConversationTree | | GET | `/conversation-trees/{tree_id}` | getConversationTree | | GET | `/conversation-trees/{tree_id}/status` | getConversationTreeStatus | | GET | `/conversation-trees/{tree_id}/versions` | listConversationTreeVersions | | GET | `/dashboard/action-items` | getActionItems | | GET | `/dashboard/alerts` | getAlerts | | GET | `/datasets` | listDatasets | | POST | `/datasets` | createDataset | | GET | `/datasets/applications/{application_id}/intents/{intent_id}` | listDatasetsByIntent | | GET | `/datasets/by-tags` | listDatasetsByTags | | GET | `/datasets/estimated-request-count` | getEstimatedDatasetRequestCount | | POST | `/datasets/generate` | startDatasetGeneration | | POST | `/datasets/generate/estimate-cost` | estimateDatasetGenerationCost | | POST | `/datasets/generate/preview` | previewDatasetGeneration | | GET | `/datasets/intent-groups/{intent_group_id}` | listDatasetsByIntentGroup | | GET | `/datasets/s3/syncer-info` | getDatasetS3SyncerInfo | | POST | `/datasets/s3/validate` | validateDatasetS3Source | | GET | `/datasets/tags/valid` | listValidDatasetTags | | GET | `/datasets/test-sets/{test_set_id}` | listDatasetsByTestSet | | DELETE | `/datasets/{dataset_id}` | deleteDataset | | GET | `/datasets/{dataset_id}` | getDataset | | PUT | `/datasets/{dataset_id}` | updateDataset | | POST | `/datasets/{dataset_id}/added-requests/search` | searchAddedDatasetRequests | | POST | `/datasets/{dataset_id}/clone` | cloneDataset | | GET | `/datasets/{dataset_id}/delete-impact` | getDatasetDeleteImpact | | GET | `/datasets/{dataset_id}/eligible-requests` | listEligibleDatasetRequests | | POST | `/datasets/{dataset_id}/eligible-requests/search` | searchEligibleDatasetRequests | | POST | `/datasets/{dataset_id}/generate/cancel` | cancelDatasetGeneration | | POST | `/datasets/{dataset_id}/generate/pause` | pauseDatasetGeneration | | POST | `/datasets/{dataset_id}/generate/resume` | resumeDatasetGeneration | | GET | `/datasets/{dataset_id}/prompt` | getDatasetPrompt | | POST | `/datasets/{dataset_id}/prune/apply` | applyDatasetPrune | | POST | `/datasets/{dataset_id}/prune/preview` | previewDatasetPrune | | POST | `/datasets/{dataset_id}/reasoning/augment` | augmentDatasetReasoning | | POST | `/datasets/{dataset_id}/regex-replacements` | applyDatasetRegexReplacements | | POST | `/datasets/{dataset_id}/regex-replacements/count` | getDatasetRegexReplacementsApplicableCount | | GET | `/datasets/{dataset_id}/request-distribution` | getDatasetRequestDistribution | | GET | `/datasets/{dataset_id}/request-ids` | listDatasetRequestIds | | DELETE | `/datasets/{dataset_id}/requests` | removeRequestsFromDataset | | GET | `/datasets/{dataset_id}/requests` | listDatasetRequests | | POST | `/datasets/{dataset_id}/requests` | addRequestsToDataset | | PUT | `/datasets/{dataset_id}/requests/add` | addDatasetRequestsBulk | | PUT | `/datasets/{dataset_id}/requests/remove` | removeDatasetRequestsBulk | | GET | `/datasets/{dataset_id}/requests/{request_id}/reasoning` | getDatasetRequestReasoning | | POST | `/datasets/{dataset_id}/s3/cancel` | cancelDatasetS3Import | | POST | `/datasets/{dataset_id}/s3/reimport` | reimportDatasetS3Source | | POST | `/datasets/{dataset_id}/sample-clone` | sampleCloneDataset | | GET | `/datasets/{dataset_id}/status` | getDatasetStatus | | GET | `/datasets/{dataset_id}/tags` | listDatasetTags | | POST | `/datasets/{dataset_id}/tags` | addDatasetTag | | PUT | `/datasets/{dataset_id}/tags` | setDatasetTags | | DELETE | `/datasets/{dataset_id}/tags/{tag}` | removeDatasetTag | | GET | `/datasets/{dataset_id}/versions` | listDatasetVersions | | GET | `/evaluation-criteria` | listEvaluationCriteria | | POST | `/evaluation-criteria` | createEvaluationCriteria | | GET | `/evaluation-criteria/available` | listAvailableEvaluationCriteria | | DELETE | `/evaluation-criteria/{criteria_id}` | deleteEvaluationCriteria | | PUT | `/evaluation-criteria/{criteria_id}` | updateEvaluationCriteria | | GET | `/evaluations` | listEvaluations | | POST | `/evaluations` | createEvaluation | | GET | `/evaluations/{evaluation_run_id}` | getEvaluation | | GET | `/evaluations/{evaluation_run_id}/results` | getEvaluationResults | | POST | `/external-test-runs` | createExternalTestRun | | GET | `/external-test-runs/{test_run_id}/next` | leaseNextCase | | POST | `/external-test-runs/{test_run_id}/{case_hash}/result` | submitCaseResult | | GET | `/finetune-runs` | listFinetuneRuns | | POST | `/finetune-runs` | createFinetuneRun | | POST | `/finetune-runs/estimate` | estimateFinetuneVram | | GET | `/finetune-runs/{run_id}` | getFinetuneRun | | POST | `/finetune-runs/{run_id}/cancel` | cancelFinetuneRun | | POST | `/finetune-runs/{run_id}/deploy` | deployFinetuneRun | | POST | `/finetune-runs/{run_id}/deploy/cancel` | cancelFinetuneDeploy | | GET | `/finetune-runs/{run_id}/metrics` | getFinetuneRunMetrics | | GET | `/finetune-runs/{run_id}/status` | getFinetuneStatus | | POST | `/help` | helpQuery | | GET | `/intent-groups` | listIntentGroups | | GET | `/intent-groups/{intent_group_id}` | getIntentGroup | | GET | `/intent-groups/{intent_group_id}/compositions` | listIntentGroupCompositions | | GET | `/intent-groups/{intent_group_id}/config` | getIntentGroupConfig | | PUT | `/intent-groups/{intent_group_id}/config` | updateIntentGroupConfig | | GET | `/intent-groups/{intent_group_id}/datasets` | listIntentGroupDatasets | | GET | `/intent-groups/{intent_group_id}/intents` | listIntentsByGroup | | GET | `/intent-groups/{intent_group_id}/models` | listIntentGroupModels | | GET | `/intent-groups/{intent_group_id}/requests` | listIntentGroupRequests | | GET | `/intent-groups/{intent_group_id}/sentinels` | listIntentGroupSentinels | | GET | `/intent-groups/{intent_group_id}/test-sets` | listIntentGroupTestSets | | GET | `/models` | listModels | | GET | `/models/applications/{application_id}` | listModelsByApplication | | GET | `/models/available` | listAvailableModels | | POST | `/models/base-model` | createBaseModel | | GET | `/models/intent-groups/{intent_group_id}` | listModelsByIntentGroup | | GET | `/models/intents/{intent_id}` | listModelsByIntent | | GET | `/models/{model_id}` | getModel | | POST | `/models/{model_id}/disable` | disableModel | | POST | `/models/{model_id}/enable` | enableModel | | GET | `/models/{model_id}/inference-pool` | getModelInferencePool | | POST | `/models/{model_id}/inference-pool/attach` | attachModelInferencePool | | GET | `/models/{model_id}/inference-pool/attach-candidates` | listModelInferencePoolAttachCandidates | | GET | `/models/{model_id}/inference-pool/debug` | getModelInferencePoolDebug | | POST | `/models/{model_id}/inference-pool/detach` | detachModelInferencePool | | PATCH | `/models/{model_id}/registry` | patchModelRegistry | | POST | `/models/{model_id}/repoint` | repointModel | | GET | `/models/{model_id}/repoint-options` | getModelRepointOptions | | POST | `/models/{model_id}/swap-model-ref/execute` | executeSwapModelRef | | POST | `/models/{model_id}/swap-model-ref/preview` | previewSwapModelRef | | GET | `/monitors` | listMonitors | | POST | `/monitors` | createMonitor | | GET | `/monitors/by-target` | listMonitorsByTarget | | GET | `/monitors/runs` | listRunsGlobal | | GET | `/monitors/runs/by-source` | listRunsBySource | | GET | `/monitors/runs/{run_id}` | getRunGlobal | | GET | `/monitors/sample-target` | getTargetSample | | DELETE | `/monitors/{monitor_id}` | deleteMonitor | | GET | `/monitors/{monitor_id}` | getMonitor | | PATCH | `/monitors/{monitor_id}` | updateMonitor | | PUT | `/monitors/{monitor_id}` | updateMonitor | | POST | `/monitors/{monitor_id}/activate` | activateMonitor | | GET | `/monitors/{monitor_id}/activity` | getMonitorActivity | | GET | `/monitors/{monitor_id}/metrics` | getMonitorMetrics | | GET | `/monitors/{monitor_id}/metrics/by-target` | getMonitorMetricsByTarget | | POST | `/monitors/{monitor_id}/pause` | pauseMonitor | | POST | `/monitors/{monitor_id}/preview` | previewMonitor | | GET | `/monitors/{monitor_id}/releases` | listMonitorReleases | | POST | `/monitors/{monitor_id}/releases` | upsertMonitorRelease | | PUT | `/monitors/{monitor_id}/releases` | upsertMonitorRelease | | GET | `/monitors/{monitor_id}/releases/default` | getDefaultMonitorVersion | | DELETE | `/monitors/{monitor_id}/releases/{release_name}` | deleteMonitorRelease | | POST | `/monitors/{monitor_id}/releases/{release_name}/promote` | promoteMonitorRelease | | GET | `/monitors/{monitor_id}/runs` | listRunsForMonitor | | GET | `/monitors/{monitor_id}/runs/{run_id}` | getRunScoped | | GET | `/monitors/{monitor_id}/targets` | listMonitorTargets | | POST | `/monitors/{monitor_id}/targets` | createMonitorTarget | | DELETE | `/monitors/{monitor_id}/targets/{monitor_target_id}` | deleteMonitorTarget | | PATCH | `/monitors/{monitor_id}/targets/{monitor_target_id}` | updateMonitorTarget | | PUT | `/monitors/{monitor_id}/targets/{monitor_target_id}` | updateMonitorTarget | | POST | `/monitors/{monitor_id}/targets/{monitor_target_id}/disable` | disableMonitorTarget | | POST | `/monitors/{monitor_id}/targets/{monitor_target_id}/enable` | enableMonitorTarget | | GET | `/monitors/{monitor_id}/targets/{monitor_target_id}/metrics` | getMonitorTargetMetrics | | GET | `/monitors/{monitor_id}/targets/{monitor_target_id}/sparkline` | getMonitorTargetSparkline | | GET | `/monitors/{monitor_id}/versions` | listMonitorVersions | | POST | `/monitors/{monitor_id}/versions` | publishMonitorVersion | | GET | `/monitors/{monitor_id}/versions/{version_id}` | getMonitorVersion | | GET | `/reports/fallbacks` | listFallbacks | | GET | `/reports/faults` | listFaults | | GET | `/reports/intents/{intent_id}/fallbacks` | listFallbacksByIntent | | GET | `/reports/intents/{intent_id}/faults` | listFaultsByIntent | | GET | `/requests` | listRequests | | GET | `/requests/{request_id}` | getRequest | | PUT | `/requests/{request_id}/response` | updateRequestResponse | | POST | `/search` | search | | GET | `/sentinels` | listSentinels | | POST | `/sentinels` | createSentinel | | GET | `/sentinels/applications/{application_id}` | listSentinelsByApplication | | GET | `/sentinels/applications/{application_id}/human-needs` | listSentinelsByApplicationHumanNeeds | | POST | `/sentinels/generate` | generateSentinels | | POST | `/sentinels/generate/new` | generateNewSentinel | | GET | `/sentinels/intent-groups/by-name/{intent_name}` | listSentinelsByIntentName | | GET | `/sentinels/intent-groups/{intent_group_id}` | listSentinelsByIntentGroup | | POST | `/sentinels/regenerate` | regenerateSentinels | | GET | `/sentinels/simple` | listSimpleSentinels | | DELETE | `/sentinels/{sentinel_id}` | deleteSentinel | | GET | `/sentinels/{sentinel_id}` | getSentinel | | PUT | `/sentinels/{sentinel_id}` | updateSentinel | | PUT | `/sentinels/{sentinel_id}/directive` | updateSentinelDirective | | POST | `/sentinels/{sentinel_id}/regenerate` | regenerateSentinel | | POST | `/sentinels/{sentinel_id}/regenerate/correction` | regenerateSentinelCorrection | | POST | `/sentinels/{sentinel_id}/regenerate/evaluation` | regenerateSentinelEvaluation | | POST | `/sentinels/{sentinel_id}/regenerate/qualification` | regenerateSentinelQualification | | GET | `/sessions` | listSessions | | GET | `/sessions/{session_id}` | getSession | | POST | `/sessions/{session_id}/feedback` | setSessionFeedback | | GET | `/sessions/{session_id}/timeline` | getSessionTimeline | | GET | `/test-runs` | listTestRuns | | POST | `/test-runs` | createTestRun | | POST | `/test-runs/compare` | compareTestRuns | | GET | `/test-runs/intent-groups/{intent_group_id}` | listTestRunsByIntentGroup | | GET | `/test-runs/requests/compare/{base_request_id}/{compare_request_id}` | getTestRunRequestComparison | | GET | `/test-runs/test-sets/{test_set_id}` | listTestRunsByTestSet | | GET | `/test-runs/test-sets/{test_set_id}/progress` | getTestRunProgressByTestSet | | DELETE | `/test-runs/{test_run_id}` | deleteTestRun | | GET | `/test-runs/{test_run_id}` | getTestRun | | POST | `/test-runs/{test_run_id}/evaluate` | evaluateTestRun | | GET | `/test-runs/{test_run_id}/items` | listTestRunItems | | PUT | `/test-runs/{test_run_id}/items/{item_id}` | updateTestRunItem | | GET | `/test-runs/{test_run_id}/progress` | getTestRunProgress | | POST | `/test-runs/{test_run_id}/re-evaluate` | reEvaluateTestRun | | GET | `/test-runs/{test_run_id}/requests` | listTestRunRequests | | POST | `/test-runs/{test_run_id}/requests/{offset}/{limit}` | listTestRunRequestsWithFilters | | POST | `/test-runs/{test_run_id}/requests/{test_run_request_id}/evaluate` | evaluateTestRunRequest | | GET | `/test-runs/{test_run_id}/results` | getTestRunResults | | POST | `/test-runs/{test_run_id}/resume` | resumeTestRun | | POST | `/test-runs/{test_run_id}/retry` | retryTestRun | | GET | `/test-runs/{test_run_id}/review-segments` | listTestRunReviewSegments | | PUT | `/test-runs/{test_run_id}/status/{status}` | updateTestRunStatus | | POST | `/test-runs/{test_run_id}/stop` | stopTestRun | | GET | `/test-set-jobs/{job_id}` | getTestSetJob | | GET | `/test-sets` | listTestSets | | POST | `/test-sets` | createTestSet | | GET | `/test-sets/applications/{application_id}/intents/{intent_id}` | listTestSetsByApplicationIntent | | POST | `/test-sets/convert-to-workflow` | convertTestSetToWorkflow | | POST | `/test-sets/convert-workflow-to-requests` | convertWorkflowTestSetToRequests | | GET | `/test-sets/datasets/{dataset_id}` | listTestSetsByDataset | | GET | `/test-sets/datasets/{dataset_id}/source` | listTestSetsCreatedFromDataset | | GET | `/test-sets/eligible-requests/{intent_group_id}/{test_set_id}` | listEligibleTestSetRequests | | POST | `/test-sets/eligible-requests/{intent_group_id}/{test_set_id}/search` | searchEligibleTestSetRequests | | GET | `/test-sets/intent-groups/by-name/{intent_group_name}` | listTestSetsByIntentGroupName | | GET | `/test-sets/intent-groups/{intent_group_id}` | listTestSetsByIntentGroup | | PUT | `/test-sets/request/add` | addSingleRequestToTestSet | | DELETE | `/test-sets/request/remove` | removeSingleRequestFromTestSet | | PUT | `/test-sets/requests/add` | addTestSetRequestsBulk | | PUT | `/test-sets/requests/remove` | removeTestSetRequestsBulk | | DELETE | `/test-sets/{test_set_id}` | deleteTestSet | | GET | `/test-sets/{test_set_id}` | getTestSet | | PUT | `/test-sets/{test_set_id}` | updateTestSet | | POST | `/test-sets/{test_set_id}/clone` | cloneTestSet | | GET | `/test-sets/{test_set_id}/imported-session-ids` | listTestSetImportedSessionIds | | DELETE | `/test-sets/{test_set_id}/items` | removeTestSetItems | | GET | `/test-sets/{test_set_id}/items` | listTestSetItems | | POST | `/test-sets/{test_set_id}/items` | addTestSetItems | | DELETE | `/test-sets/{test_set_id}/items/{item_id}` | deleteTestSetItem | | DELETE | `/test-sets/{test_set_id}/items/{item_id}/response-sub` | deleteTestSetItemResponseSub | | GET | `/test-sets/{test_set_id}/items/{item_id}/response-sub` | getTestSetItemResponseSub | | PUT | `/test-sets/{test_set_id}/items/{item_id}/response-sub` | upsertTestSetItemResponseSub | | GET | `/test-sets/{test_set_id}/prompt-template` | getTestSetPromptTemplate | | GET | `/test-sets/{test_set_id}/request-distribution` | getTestSetRequestDistribution | | DELETE | `/test-sets/{test_set_id}/requests` | removeRequestsFromTestSet | | GET | `/test-sets/{test_set_id}/requests` | listTestSetRequests | | POST | `/test-sets/{test_set_id}/requests` | addRequestsToTestSet | | PUT | `/test-sets/{test_set_id}/requests/{request_id}/response` | updateTestSetRequestResponse | | PUT | `/test-sets/{test_set_id}/requests/{request_id}/tags` | updateTestSetRequestTags | | GET | `/test-sets/{test_set_id}/sessions` | listTestSetSessionFacets | | GET | `/unified-test-runs` | listUnifiedTestRuns | | POST | `/unified-test-runs` | createUnifiedTestRun | | POST | `/unified-test-runs/preview-adapted-input` | previewAdaptedInput | | DELETE | `/unified-test-runs/{test_run_id}` | deleteUnifiedTestRun | | GET | `/unified-test-runs/{test_run_id}` | getUnifiedTestRun | | POST | `/unified-test-runs/{test_run_id}/cancel` | cancelUnifiedTestRun | | POST | `/unified-test-runs/{test_run_id}/execute` | executeUnifiedTestRun | | GET | `/unified-test-runs/{test_run_id}/items` | listUnifiedTestRunItems | | POST | `/unified-test-runs/{test_run_id}/prepare` | prepareUnifiedTestRun | | GET | `/unified-test-runs/{test_run_id}/progress` | getUnifiedTestRunProgress | | POST | `/unified-test-runs/{test_run_id}/reset-failed` | resetFailedUnifiedTestRunItems | | POST | `/unified-test-runs/{test_run_id}/score` | scoreUnifiedTestRun | | GET | `/unified-test-sets` | listUnifiedTestSets | | POST | `/unified-test-sets` | createUnifiedTestSet | | POST | `/unified-test-sets/upload` | uploadUnifiedTestSetJsonl | | GET | `/unified-test-sets/{test_set_id}` | getUnifiedTestSet | | PATCH | `/unified-test-sets/{test_set_id}` | patchUnifiedTestSet | | POST | `/unified-test-sets/{test_set_id}/clone` | cloneUnifiedTestSet | | POST | `/unified-test-sets/{test_set_id}/import-agent-task` | importAgentTaskIntoUnifiedTestSet | | POST | `/unified-test-sets/{test_set_id}/import-request` | importRequestIntoUnifiedTestSet | | POST | `/unified-test-sets/{test_set_id}/import-workflow-run` | importWorkflowRunIntoUnifiedTestSet | | GET | `/unified-test-sets/{test_set_id}/items` | listUnifiedTestSetItems | | POST | `/unified-test-sets/{test_set_id}/items` | createUnifiedManualTestSetItem | | DELETE | `/unified-test-sets/{test_set_id}/items/{item_id}` | deleteUnifiedTestSetItem | | GET | `/unified-test-sets/{test_set_id}/items/{item_id}` | getUnifiedTestSetItem | | PUT | `/unified-test-sets/{test_set_id}/items/{item_id}/expected-output-override` | upsertUnifiedTestSetItemExpectedOutputOverride | | GET | `/workflows` | listWorkflows | | POST | `/workflows` | createWorkflow | | POST | `/workflows/chat/completions` | createWorkflowChatCompletion | | GET | `/workflows/default-script` | getDefaultWorkflowScript | | GET | `/workflows/runs` | listWorkflowRuns | | PUT | `/workflows/runs/log` | logWorkflowRun | | GET | `/workflows/runs/{session_id}` | getWorkflowRun | | POST | `/workflows/upload` | uploadWorkflowBundle | | POST | `/workflows/validate` | validateWorkflowScript | | DELETE | `/workflows/{workflow_id}` | deleteWorkflow | | GET | `/workflows/{workflow_id}` | getWorkflow | | PUT | `/workflows/{workflow_id}` | updateWorkflow | | DELETE | `/workflows/{workflow_id}/accessories` | deleteWorkflowAccessory | | POST | `/workflows/{workflow_id}/accessories` | uploadWorkflowAccessory | | GET | `/workflows/{workflow_id}/artifact-files` | getWorkflowArtifactFiles | | GET | `/workflows/{workflow_id}/datastores` | listWorkflowDatastores | | DELETE | `/workflows/{workflow_id}/datastores/{name}` | deleteWorkflowDatastore | | POST | `/workflows/{workflow_id}/datastores/{name}` | upsertWorkflowDatastore | | POST | `/workflows/{workflow_id}/datastores/{name}/upload` | uploadWorkflowDatastore | | GET | `/workflows/{workflow_id}/has-transforms` | checkWorkflowTransforms | | GET | `/workflows/{workflow_id}/releases` | listWorkflowReleases | | POST | `/workflows/{workflow_id}/releases` | createWorkflowRelease | | POST | `/workflows/{workflow_id}/releases/clear-default` | clearWorkflowDefaultRelease | | DELETE | `/workflows/{workflow_id}/releases/{name}` | deleteWorkflowRelease | | PUT | `/workflows/{workflow_id}/releases/{name}` | updateWorkflowRelease | | GET | `/workflows/{workflow_id}/schema` | getWorkflowSchema | | GET | `/workflows/{workflow_id}/script` | getWorkflowScript | | GET | `/workflows/{workflow_id}/transform-info` | getWorkflowTransformInfo | | POST | `/workflows/{workflow_id}/transform-preview` | previewWorkflowTransform | | GET | `/workflows/{workflow_id}/versions` | listWorkflowVersions | | POST | `/workflows/{workflow_id}/versions` | publishWorkflowVersion | | GET | `/workflows/{workflow_id}/versions/{version_number}/artifact-files` | getWorkflowVersionArtifactFiles | # CLI Reference Source: https://docs.trymaitai.com/cli-reference/overview Command-line interface for the Maitai Platform Developer API. The Maitai CLI lets you manage Maitai product resources from the terminal, applications, intents, agents, workflows, datasets, test sets, finetune runs, sentinels, analytics, and more. It wraps the [API Reference](/api-reference) (`/api/v1`). ## Install ```bash theme={null} pip install maitai-cli ``` ## Authenticate ```bash theme={null} # Browser login (default): opens Portal, then saves key to ~/.maitai/config maitai login # Or pass a pre-created key (CI / headless) maitai login --api-key YOUR_MAITAI_API_KEY # Or use environment variable (no login needed) export MAITAI_API_KEY="YOUR_MAITAI_API_KEY" ``` Create API keys in the [Portal](https://portal.trymaitai.com) under **Settings > API Keys**. ## Quick usage ```bash theme={null} # Check auth status maitai whoami # List resources maitai applications --list maitai agents --list maitai intent-groups --list maitai workflows --list maitai datasets --list maitai test-sets --list # Get by ID maitai applications --get 42 maitai agents --get 1 # Raw API calls (any endpoint) maitai api GET /applications maitai api POST /applications -b '{"application_name":"Test","application_ref_name":"test"}' maitai api POST /workflows/123/versions -b '{"bump_type":"minor"}' ``` ## Base URL * **Production**: `https://api.trymaitai.ai/api/v1` * Override with `MAITAI_BASE_URL` or `maitai login --base-url` ## Response format * **Success**: `{"data": {...}}` * **Paginated**: `{"data": [...], "pagination": {"total": N, "offset": 0, "limit": 25}}` * **Error**: `{"error": {"status": 401, "message": "..."}}` ## Command reference See [Commands](/cli-reference/commands) for the full list of resource commands and options. For coverage details and intentionally excluded Portal/admin surfaces, see [API Coverage](/api-reference/coverage). # API to Agent Source: https://docs.trymaitai.com/examples/api_to_agent Turn an API into a structured Agent by importing endpoints as API Call actions, organizing them into domain subagents, and encoding the API docs as instructions. This recipe shows how to take an existing API and build a **single Agent entrypoint** where: * **Each endpoint becomes a Capability** (an **API Call** action) * Endpoints are grouped into a **domain subagent tree** (Payments, Customers, Invoices, etc.) * Your “API docs” are encoded into the agent's **flow step prompts** so the agent can sequence calls reliably The outcome is an “integration without reading the docs”: users can point requests at the parent agent, and the agent knows which endpoints to call and in what order. ## 1) Choose a subagent tree (example: “Stripe Agent”) Before importing anything, decide how you want to divide the API into domains. For a Stripe-style API, a common tree is: * **Parent**: “Stripe Agent” * Routes requests to the right domain agent * Produces user-facing answers (and asks clarifying questions when inputs are missing) * **Children (subagents)**: * “Stripe Payments Agent” (payments + refunds) * “Stripe Customers Agent” (customer lookup + updates) * “Stripe Invoices Agent” (invoice lifecycle) Each subagent should “own” the endpoints for its domain. Example mapping (illustrative): | Subagent | Example endpoints that become API Actions | | ---------------------- | ------------------------------------------------------------------------------------------------------ | | Stripe Payments Agent | `POST /v1/payment_intents`, `GET /v1/payment_intents/{id}`, `POST /v1/refunds`, `GET /v1/charges/{id}` | | Stripe Customers Agent | `POST /v1/customers`, `GET /v1/customers/{id}`, `GET /v1/customers`, `POST /v1/customers/{id}` | | Stripe Invoices Agent | `GET /v1/invoices`, `GET /v1/invoices/{id}`, `POST /v1/invoices`, `POST /v1/invoices/{id}/finalize` | This keeps each subagent’s toolset smaller and makes routing more reliable. See: [Subagent](/build/agents/capabilities/subagent) ## 2) Create the parent agent + domain subagents ### Create the parent (“Stripe Agent”) 1. Go to **Forge > Agents > Create Agent**. 2. Set: * **Name**: “Stripe Agent” * **Description**: “Routes requests across Stripe domains and returns user-facing answers.” * **Execution Mode**: **Reasoning**: the parent needs to sequence calls, delegate to domain subagents, and wait on webhooks * **Base URL** (optional): default host used by API actions (often your Stripe proxy / gateway) See: [Create Agent](/build/agents/create_agent) · [Execution Mode](/build/agents/execution_mode) ### Create one agent per domain (Payments, Customers, Invoices) 1. Go to **Forge > Agents > Create Agent** (repeat for each domain). 2. Keep descriptions crisp (they are used for routing). ### Attach subagents under the parent 1. Open the parent agent. 2. Go to **Capabilities**. 3. Click **Add Capability**, then choose **Subagent**. 4. Add your domain agents (Payments, Customers, Invoices). Tip: keep each subagent’s **description** crisp, the parent router uses these descriptions to decide where to delegate. ## 3) Capabilities: convert endpoints into API Call actions (OpenAPI to actions) This is the “endpoint-to-action” step: each selected OpenAPI operation becomes one **API Call** action. ### Bulk Import (recommended for “every endpoint becomes an action”) Do this once per subagent, selecting only the endpoints for that domain: 1. Open a domain subagent (e.g., “Stripe Payments Agent”). 2. Go to **Capabilities**. 3. Click **Bulk Import**. 4. Paste your **OpenAPI spec** (JSON or YAML). 5. Deselect any endpoints that *don’t* belong to this subagent’s domain. 6. Optionally rename action names/descriptions (recommended for readability). 7. Click **Import**. This creates one **API Call** action per selected OpenAPI operation, including: * Path + query parameters * JSON request body schema (when present) * JSON response schemas (per status code, when present) ### Single-action Import (good for incremental additions) 1. Open the agent. 2. Go to **Capabilities > Add Capability > API Call**. 3. In the API action editor, click **Import** and paste an OpenAPI spec. 4. Select the operation to use and import it. See: [API Action](/build/agents/capabilities/api_action) ## 4) Capabilities: add webhook-based lifecycle handling (PaymentIntent example) Some APIs are inherently asynchronous. For Stripe PaymentIntents, “create” and “final status” are often separated by an event (`payment_intent.succeeded`, `payment_intent.payment_failed`, etc.). Maitai supports this using a **Webhook Creation** capability that: * registers a per-session webhook callback with Maitai's **hooks service** and returns a unique `webhook_url` (e.g., `https://hooks.trymaitai.ai/agent/`) * **automatically infers** where and how to register the webhook based on conversation context * sends the registration request to the appropriate external service (e.g., Stripe, Shopify) * returns **WAITING\_WEBHOOK** until an external system POSTs an event payload to that `webhook_url` When a payload is POSTed to the `webhook_url`, Maitai's hooks service looks up the webhook key and routes the payload to the correct **agent + session**, then resumes execution. See: [Webhook](/build/agents/capabilities/webhook) ### Add a "Webhook Creation" capability to the Payments subagent 1. Open **Stripe Payments Agent**. 2. Go to **Capabilities > Add Capability > Webhook Creation**. 3. **Invocation Mode** is automatically set to **Foreground** (webhooks must pause execution to wait for callbacks). 4. Configure authentication and optional settings: * **Authentication**: Add your Stripe API key (Bearer token, API key, or Basic auth) * **Headers** (optional): Additional headers needed for registration * **Timeout**: How long to wait for the webhook callback (default: 10 minutes) That's it! When the agent invokes this capability, Maitai will: 1. Create a unique webhook URL 2. Analyze the conversation context (user request, capability description, recent API calls) 3. **Intelligently infer** the registration endpoint and payload structure (e.g., Stripe's `/v1/webhook_endpoints`) 4. Add your configured authentication and headers 5. Make the registration HTTP call 6. Wait for the external service to call back The registration endpoint and payload structure are **automatically inferred** by the LLM based on context. You only need to provide authentication credentials. **Example scenario:** User: "Create a payment intent for \$50 and let me know when it's confirmed" Agent flow: 1. Calls `create_payment_intent` API action, which creates PaymentIntent `pi_123` 2. Invokes webhook capability 3. **LLM infers** from context: * Service: Stripe * Endpoint: `https://api.stripe.com/v1/webhook_endpoints` * Payload: `{"url": "https://hooks.trymaitai.ai/agent/abc123", "enabled_events": ["payment_intent.succeeded"]}` 4. **System adds** your configured Stripe API key to the request 5. Registration call succeeds 6. Agent waits for Stripe to call the webhook 7. Stripe calls webhook when payment succeeds 8. Agent resumes and responds to user If registration fails (e.g., invalid credentials), the agent returns an error immediately instead of waiting. ### What should call back to Maitai? Maitai’s agent webhook endpoint (the `webhook_url` on `hooks.trymaitai.ai`) does **not** verify provider signatures (like Stripe’s `Stripe-Signature`). * If you need signature verification, use: Stripe, then **your webhook handler** (verify), then Maitai `webhook_url` (forward) * If you accept the risk (not recommended), you can configure Stripe to call the `webhook_url` directly. For testing, you can POST a payload to the `webhook_url` the agent generated. Example: ```json theme={null} { "provider": "stripe", "event_type": "payment_intent.succeeded", "payment_intent_id": "pi_123", "customer_id": "cus_456", "amount": 2000, "currency": "usd" } ``` If you want to simulate the callback manually: ```bash theme={null} curl --request POST \ --url "$WEBHOOK_URL" \ --header "Content-Type: application/json" \ --data '{"provider":"stripe","event_type":"payment_intent.succeeded","payment_intent_id":"pi_123","customer_id":"cus_456","amount":2000,"currency":"usd"}' ``` ## 5) Make capabilities usable (naming, auth, foreground/background, schemas) After import, open actions and confirm the config is correct: * **Invocation Mode** * **Foreground**: results are user-facing (used in the agent’s response) * **Background**: results are stored in context only (not shown to the user) * **Response Type** (API Call actions only) * **Real**: call the live API * **Mock**: return predefined responses (useful to prototype before auth is ready) * **Auth / Headers** * bulk import defaults auth to “none”; set the auth scheme your API needs * **Request/Response schema** * keep request schemas tight so the agent doesn’t “invent” parameters Tip: For large APIs, naming consistency matters. Prefer action names like `stripe_create_payment_intent`, `stripe_get_customer`, `stripe_finalize_invoice`. ## 6) Agent State: define the state the agent should maintain When an API is involved, state is often the difference between a clean flow and a confusing one. 1. Open the agent. 2. Go to the **State** tab. 3. Add fields like: * `customer_id` * `payment_intent_id` * `invoice_id` * `last_refund_id` 4. Set **Read/Write scope** based on which subagents should be able to see/update the field. See: [Agent State](/build/agents/agent_state) ## 7) “Load the API docs” into the flow step prompts This is the step that makes the agent behave like it “understands the docs”. Open **Configuration** on the parent agent and customize the relevant [flow steps](/build/agents/agent_flow): ### Orchestrate step Owns delegation and sequencing. Add guidance like: * Route “payment”, “charge”, “refund” to **Stripe Payments Agent** * Route “customer”, “user”, “account” to **Stripe Customers Agent** * Route “invoice”, “bill”, “statement” to **Stripe Invoices Agent** * Required inputs per workflow (what IDs must exist before calling an endpoint) * Happy-path sequences (e.g., lookup customer, list invoices, then finalize invoice) * Guardrails (never create resources without explicit user confirmation; if required IDs are missing, ask a clarifying question instead of guessing) * Async lifecycles (e.g., “create PaymentIntent, wait for event, then fetch final status”) ### Process step Owns result interpretation and state updates. Add guidance like: * What matters in each API response (which fields are findings worth keeping) * What to store in state for later steps (IDs, statuses, amounts) * How to treat API errors (what a 4xx vs 5xx means for the task) ### Respond step Owns the user-facing answer. Add guidance like: * How to summarize responses for the user (what fields matter) * How to translate API errors into user-facing guidance * Tone and formatting policy See: [Agent Flow Steps](/build/agents/agent_flow) ## 8) Test in Studio (including webhook lifecycle) 1. Go to **Forge > Studio**. 2. Select the parent agent (e.g., “Stripe Agent”). 3. Try prompts that force cross-domain behavior: * “Refund the last payment for customer `cus_...`” * “Create an invoice for customer `cus_...` for \$200 and finalize it” * “Which invoices are past due for customer `cus_...`?” 4. Test an async PaymentIntent flow: * “Create a \$20 PaymentIntent for customer `cus_...` and tell me when it succeeds.” * You should see the agent return **WAITING\_WEBHOOK** plus a **Webhook URL**. * When your system (or a test script) POSTs the event payload to that webhook URL, Maitai will resume the agent and continue execution. 5. Iterate: * tighten subagent descriptions (better routing) * add more “API doc” guidance to the Orchestrate/Process step prompts (better sequencing + summaries) * narrow schemas so actions can’t be called with ambiguous inputs See: [Agent Flow Steps](/build/agents/agent_flow) # Build an Agent Source: https://docs.trymaitai.com/examples/build_agent Create a new Agent, add capabilities, and configure how it executes This recipe shows how to build an Agent from scratch: giving it tools (Capabilities), choosing how it executes (Execution Mode), and tuning its behavior. ## 1) Create the Agent 1. Go to **Forge > Agents**. 2. Click **Create Agent**. 3. Fill in the basics: * **Name**: e.g., "Support Triage Bot". * **Description**: What the agent does (used for routing if this agent is a sub-agent). * **Execution Mode**: **Reasoning** (multi-step planning loop, the default) or **Route** (each request goes straight to one capability). You can change this later from the Configuration tab, but not per-request. * **Application**: Where the agent's traffic is monitored. * **Base URL** (optional): A default API host for any API Actions this agent uses. See: [Create Agent](/build/agents/create_agent) · [Execution Mode](/build/agents/execution_mode) ## 2) Add Capabilities Capabilities are the "tools" your agent can use. 1. Open your new Agent. 2. Go to the **Capabilities** tab. 3. Click **Add Capability** and choose a type: * **LLM Action**: A sub-task handled by an LLM prompt (e.g., "Summarize Ticket"). * **API Action**: An external API call (e.g., "Get User Profile"). * **Webhook Creation**: Pause the session until an external system calls back. * **Maitai Workflow**: Run a hosted multi-step workflow. * **Sub-Agent**: Delegate to another existing agent. Give each capability a clear description plus **When to use / When not to use** guidance, both routing and orchestration rely on them. See: [Capabilities](/build/agents/capabilities/llm_action) ## 3) Configure execution Open the **Configuration** tab. What you see depends on the agent's execution mode: **Reasoning mode**: the Agent Flow pipeline (Orchestrate, Action, Process, Progress Probe, then Respond): * Toggle the optional **Process** and **Progress Probe** steps. * Click any step to customize its system prompt, model, and parameters. **Route mode**: the routing strategy: * **LLM Routing**: the model selects the best capability from the request, state, and capability descriptions. * **Rules-Based Routing**: ordered deterministic rules (e.g., "If message contains 'refund', route to RefundCapability"), with a default route as fallback. See: [Execution Mode](/build/agents/execution_mode) · [Agent Flow Steps](/build/agents/agent_flow) ## 4) Define State If your agent needs to remember specific information across the session (like a user's name or order ID), define it as state. 1. Go to the **State** tab. 2. Add fields (e.g., `order_id`, `user_email`) with a type and a clear description. 3. Set **Read/Write Scope** to control which sub-agents can see or update each field. See: [Agent State](/build/agents/agent_state) ## 5) Test in Studio Verify your agent works as expected. 1. Go to **Forge > Studio**. 2. Select your Agent. 3. Chat with it to verify routing, state collection, and capability execution. ## 6) Publish Portal edits land on the agent's draft. When the behavior looks right in Studio, click **Publish** to create an immutable version, and optionally label it with a release name like `prod` to pin production traffic to it. ## 7) Call the agent from your application Invoke the agent from code: * [Agent Call](/sdk/agent_call) ```python theme={null} request = AgentRequest( agent="Support Triage Bot", application="YOUR_APPLICATION", session_id="session-123", params=ChatCompletionParams( messages=[{"role": "user", "content": "Handle refund for order 12345"}], ), ) ``` For per-request control (pinning versions, seeding state, disabling capabilities, supplying secrets) see [Agent Call](/sdk/agent_call). For performance flags like `high_performance`, see [Execution Mode, performance options](/build/agents/execution_mode#performance-options). # Configuration Source: https://docs.trymaitai.com/examples/configuration Common Portal configuration patterns (application / intent / intent group) Maitai’s Portal exposes a **Configuration** panel at multiple scopes so you can tune behavior without shipping code changes. ## Where to configure In the Portal, configuration appears in these places: * **Application**: `/application/:applicationId`, then **Configuration** * **Intent Group**: `/intent-group/:intentGroupId`, then **Configuration** * **Intent (ApplicationAction)**: `/application/:applicationId/intent/:actionId`, then **Configuration** * **Test Run**: the Test Run wizard includes a configuration step used for that run config portal ## Common patterns ### Enable evaluations + corrections * Turn on **Evaluations** * Optionally turn on **Apply Corrections** * This toggle is only available when evaluations are enabled. * Enabling corrections unlocks **Safe Mode**. ### Add a secondary model + fallback behavior * Select a **Secondary Model** * Choose a **Fallback Strategy** * Optionally set a **Fallback Timeout** (only appears for timeout strategy) ### Make output more deterministic * Lower **Temperature** * Use a **Stop Sequence** * Set **Max Tokens** ## Parameters available in the Portal Configuration panel Controls where inference is executed. Controls the model’s reasoning depth when supported by the selected model. Enables Sentinel evaluation for requests at this scope. Corrections are only available for Server inference, and require Evaluations to be turned on. Enables automatic correction behavior when evaluations identify faults. Safe mode prioritizes accuracy over latency. Only available when Apply Corrections is enabled. Primary AI model to be used for inference. Models are grouped by provider and sorted alphabetically, with Maitai models (recommended) appearing first. Optional fallback model to use if the primary model is not available or has degraded performance. Strategy to use when falling back to secondary model. Options: * `reactive`: Falls back on primary model failure * If a timeout is specified, it will initiate the fallback request after the timeout period * `timeout`: Falls back after specified timeout period * `first_response`: Uses whichever model responds first * If a timeout is specified, it will only use the fallback model after the timeout period Timeout in seconds before falling back to secondary model. Only applicable when using the "timeout" fallback strategy. Controls randomness in the model's output. Range 0-2, where: * Lower values (e.g., 0): More deterministic output * Higher values (e.g., 2): More random output Sequences where the API will stop generating further tokens. Include the log probabilities of the output tokens. How many completions to generate for each prompt. The maximum number of tokens to generate in the completion. Range -2 to 2. Increases the model's likelihood to talk about new topics. Range -2 to 2. Decreases the model's likelihood to repeat the same line verbatim. # Debug a Request with Reruns + Evaluations Source: https://docs.trymaitai.com/examples/debug_request_reruns Test variations of a single request (model/messages) and evaluate them with Sentinels This recipe is for “I have one problematic request, I want to try changes quickly and see what improves.” ## 1) Open the Request Overview page You can reach a Request detail page from: * an Intent Group’s **Requests** tab, or * `Test > Evaluations` (open a request from a row), or * Test Set / Test Run request tables (open a request from a row) See: [Request Overview](/observe/request_overview) ## 2) Create a rerun version On the right side of the Request page, open the **Versions** panel and click **New Rerun**. Then: 1. Select the new rerun version from the Versions list. 2. Adjust the rerun inputs (for example, model selection and messages as needed). 3. Click **Run Request** to execute the rerun. The rerun will capture a new response and show its basic metrics (time / tokens / cost) in the UI. ## 3) Evaluate the rerun with Sentinels On the same Request page, use the **Evaluation** section: 1. Click **Add Evaluation** 2. Select the sentinel(s) you want to run 3. Click **Run Evaluation** This makes it easy to compare sentinel outcomes between the **original request** and your **rerun**. See: [Evaluating a Request](/observe/sentinels/request_evaluations) ## 4) Promote the original case into a Test Set (optional) If this is a case you don’t want to regress on: 1. Switch back to **Original** in the Versions panel. 2. Click **Add to Test Set**. 3. Add tags and submit. Then run a Test Run on that Test Set to verify improvements at scale. See: * [Test Set Creation](/test/test_sets/creation) * [Test Run Execution](/test/test_sets/execution) # Fine-tune a Model Source: https://docs.trymaitai.com/examples/fine_tuning End-to-end workflow: Dataset, Composition, Run, then Validation This recipe walks through the complete lifecycle of creating a fine-tuned model in Maitai, from raw data to a deployed model. ## 1) Create a Dataset The first step is gathering the data you want the model to learn from. 1. Go to **Forge > Finetuning**. 2. Click **New Dataset**. 3. Choose a source (e.g., **Production Traffic** to filter real requests, or **File Upload**). 4. Save the dataset. See: [Dataset Creation](/build/models/dataset_creation) ## 2) Prepare & Refine Data Once the dataset is created, you can improve its quality. 1. Open the Dataset. 2. Use **AI Review** to spot inconsistencies. 3. Edit individual requests to fix errors or improve the "ideal" response. 4. Apply **Modifications** (regex) or **Augmentations** (synthetic variations) if needed. See: [Dataset Preparation](/build/models/dataset_preparation) ## 3) Create a Composition A Composition is a "recipe" that combines one or more datasets. This lets you mix and match data (e.g., "80% production data + 20% golden test set") without duplicating the underlying requests. 1. Go to **Forge > Compositions**. 2. Click **New Composition**. 3. Select the datasets to include. 4. Adjust sampling weights if desired. See: [Compositions](/build/models/compositions) ## 4) Start a Fine-tune Run Now you're ready to train. 1. Go to **Forge > Finetuning** (or start directly from a Composition). 2. Click **New Finetune Run**. 3. Select your **Composition**. 4. Choose the **Base Model** (e.g., `llama-3.1-8b-instruct`). 5. Configure hyperparameters (epochs, learning rate) or stick to defaults. 6. Start the run. See: [Fine-tune Runs](/build/models/fine_tune_runs) ## 5) Validate the Model After training completes, Maitai automatically runs a validation step (if configured) or you can run one manually. 1. Open the **Fine-tune Run**. 2. Go to the **Validation** tab. 3. Review the **Validation Run** results to see how the new model performed against a holdout set or specific Test Set. 4. Check metrics like accuracy and pass rate. See: [Validation](/build/models/validation) ## 6) Benchmarking against Golden Test Sets Once you have a candidate model, the final hurdle before deployment is ensuring it actually outperforms your baseline on high-signal cases. 1. Go to **Test > Test Sets**. 2. Select your **Golden Test Set** for this intent. 3. Start a **New Test Run** using your newly fine-tuned model. 4. Use the **Compare Runs** feature to side-by-side the results of the new model against your current production model. 5. Verify that the new model maintains or improves performance on critical "golden" examples. See: [Test Set Creation](/test/test_sets/creation), [Test Run Execution](/test/test_sets/execution) ## 7) Deploy If the model performs well and passes your golden benchmarks: 1. Click **Deploy** on the Fine-tune Run page. 2. The model becomes available as a `ft:...` ID that you can use in your Applications or Intents configuration. See: [Deployment](/build/models/deployment) # Import Data Source: https://docs.trymaitai.com/examples/import_data Upload your data to the Portal to build custom models specific to your application Importing your data is the first step toward finetuning models and building agents. The Portal makes it easy and quick to upload, examine, and augment your data. The Import Data feature can be accessed from two places: * **Global view**: Forge > Compositions > Datasets * **Application view**: select an existing Application, drill into the Intents tab, and select an Intent. From there, click on the Datasets tab From either view, click on the **New Dataset** button. If you are coming from the Global view, you will have to choose an Application and Intent to assign the dataset to. You can also create a new Application and Intent using the dropdown if necessary. New Dataset ## Manually Upload a Dataset 1. Select **Manual Upload** under the Data Source options. 2. Under Upload Dataset File, select a JSONL dataset file from your computer. Hit **Continue**. 3. Under Basic Information, enter a Name and Description for the dataset. Hit **Continue**. 4. Review the Dataset Summary, and click **Create Dataset**.
## Import from S3 1. Select **Import from S3** under the Data Source options. 2. Enter the S3 URL (Add the AWS Region and File glob if necessary). 3. Copy and paste the Bucket Policy script into the Permissions section in your AWS account. Replace `YOUR_PREFIX` with the folder path WITHOUT a leading slash (example: `data/exports`). If you want to grant access to the bucket root, use an empty prefix and remove the prefix condition. 4. Click **Validate Access** in the Portal. Hit **Continue**. 5. Under Basic Information enter a Name and Description for the dataset. Hit **Continue**. 6. Review the Dataset Summary, and click **Create Dataset**.

* false: run inference with your own client. To send those request/response pairs to Maitai for monitoring and indexing, use maitai.log() (bring your own inference).

## Fallbacks (optional) Optional fallback configuration used when the primary model fails or times out. Must be one of `reactive`, `first_response`, or `timeout`. The fallback model name. Timeout (seconds) for strategies that use a timeout. ## OpenAI parameters Maitai accepts the standard OpenAI `chat.completions.create` parameters (for example: `model`, `temperature`, `tools`, `tool_choice`, `response_format`, `stream`, etc.). If you set them in the request, they override Portal defaults. Related guides: * [Structured Output](/sdk/structured_output) * [Tool Calling](/sdk/tool_calling) ## Response fields Maitai returns an OpenAI-compatible `chat.completion` response, plus a few extra fields: The Maitai request identifier. Evaluation results (when evaluations are enabled). Whether a correction was applied. Optional input safety score (when enabled). A number between 0 and 1 where lower values indicate a potential jailbreak attempt. Time to first token (milliseconds), when available. Total response time (milliseconds), when available. Present when a fallback model was used (explains why). ## Streaming usage # Evaluate Source: https://docs.trymaitai.com/sdk/evaluate Detect faults in LLM output ## Create evaluations Evaluations are created when you call [Chat](/sdk/chat) with evaluations enabled (via Portal configuration or SDK parameters). The model output is what gets evaluated, with the request used as context. If a `callback` is provided, the evaluation is passed to that function asynchronously. If no `callback` is provided, and `stream` is `true`, then the evaluation is available on the last chunk. If no `callback` is provided and `stream` is `false`, then the evaluation can be found on the completion response. ## Example (callback) ```python Python (async) theme={null} import maitai async def on_eval(eval_response): # handle evaluate response (EvaluateResponse) print(eval_response) messages = [ {"role": "system", "content": "You are a helpful assistant"}, {"role": "user", "content": "Generate numbers 1-10"}, ] client = maitai.MaitaiAsync() response = await client.chat.completions.create( messages=messages, model="gpt-4o", session_id="YOUR_SESSION_ID", intent="NUMBER_GENERATOR", application="demo_app", callback=on_eval, ) ``` ```python Python (sync) theme={null} import maitai def on_eval(eval_response): # handle evaluate response (EvaluateResponse) print(eval_response) messages = [ {"role": "system", "content": "You are a helpful assistant"}, {"role": "user", "content": "Generate numbers 1-10"}, ] client = maitai.Maitai() response = client.chat.completions.create( messages=messages, model="gpt-4o", session_id="YOUR_SESSION_ID", intent="NUMBER_GENERATOR", application="demo_app", callback=on_eval, ) ``` ```javascript Node theme={null} import Maitai from "maitai"; const maitai = new Maitai(); const onEval = (evalResponse) => { // handle evaluate response (EvaluateResponse) console.log(evalResponse); }; const messages = [ { role: "system", content: "You are a helpful assistant" }, { role: "user", content: "Generate numbers 1-10" }, ]; const response = await maitai.chat.completions.create({ messages: messages, model: "gpt-4o", session_id: "YOUR_SESSION_ID", intent: "NUMBER_GENERATOR", application: "demo_app", callback: onEval, }); ``` ```json theme={null} { "application_id": 16, "session_id": "xxx", "evaluation_results": [ { "status": "FAULT", "description": "Test with random number", "confidence": 100.0, "correction": "Test with random number 42.", "sentinel_id": 41, "eval_time": 489, "date_created": 1716682520088, "usage": { "prompt_tokens": 380, "completion_tokens": 54, "total_tokens": 434 } } ], "evaluation_request_id": "b84f0d55-7c7e-4af4-8d84-106385682250" } ``` The Maitai identifier for the application. A unique identifier for the session, passed in from the [Chat](/sdk/chat) endpoint. The identifier of the evaluated chat completion request. A list of individual Sentinel results. A unique identifier for the evaluation result. The status of the evaluation. `FAULT` means a fault was detected. `PASS` means the LLM output passed testing. `NA` means an evaluation wasn't performed. A detailed description of the evaluation outcome. The confidence level of the evaluation result. Optional suggested correction text when a `FAULT` is detected. Additional metadata associated with the evaluation result. An identifier for the Sentinel associated with this result. The evaluation time in milliseconds. The Unix timestamp marking the creation of the evaluation result. Usage statistics for the Sentinel Number of tokens in the generated completion. Number of tokens in the prompt. Total number of tokens used in the request (prompt + completion). The unique identifier for the evaluation request. # File Inputs Source: https://docs.trymaitai.com/sdk/file_inputs Send images, video, and documents to multimodal models as part of a chat completion. Maitai lets you attach files, **images, video, and documents (PDFs)**: to a chat completion request. Maitai stores the file, hands it to the model in the format that provider expects, and runs inference. There are two ways to send a file, and you can mix them freely: 1. **Inline with the request**: attach the file directly in the message. Best for one-off requests; nothing to pre-upload. 2. **Upload first, then reference by `file_id`**, call `files.upload(...)` once to get a `file_id`, then reference it on any number of later requests without re-sending the bytes. Best when you ask multiple questions about the same (especially large) file. ## Quickstart Attach a local file inline. Maitai reads the bytes, validates that the target model accepts that file type, and includes it in the request. ```python Python theme={null} import maitai client = maitai.Maitai() messages = [ { "role": "user", "content": [ {"type": "text", "text": "Describe what happens in this video."}, maitai.file_content_part_from_path("clip.mp4"), ], } ] response = client.chat.completions.create( messages=messages, application="demo_app", intent="VIDEO_ANALYSIS", session_id="YOUR_SESSION_ID", model="gemini-3.5-flash", server_side_inference=True, ) print(response.choices[0].message.content) ``` ```javascript Node theme={null} import fs from "node:fs"; import Maitai, { fileContentPartFromBytes } from "maitai"; const maitai = new Maitai(); const bytes = fs.readFileSync("clip.mp4"); const messages = [ { role: "user", content: [ { type: "text", text: "Describe what happens in this video." }, fileContentPartFromBytes(bytes, "video/mp4", "clip.mp4"), ], }, ]; const response = await maitai.chat.completions.create({ messages, application: "demo_app", intent: "VIDEO_ANALYSIS", session_id: "YOUR_SESSION_ID", model: "gemini-3.5-flash", server_side_inference: true, }); console.log(response.choices[0].message.content); ``` File inputs require server-side inference (`server_side_inference=True`), which is the default. Maitai needs to run the request to upload and convert the file for the provider. ## What's supported File support depends on the model. Maitai validates the file against the target model and returns a `400` if the model does not accept that file type. | Provider | Image | Video | Document (PDF) | | ------------------------------------ | :---: | :---: | :------------: | | **Gemini** (e.g. `gemini-3.5-flash`) | ✓ | ✓ | ✓ | | **OpenAI** (e.g. `gpt-4o`) | ✓ | ✗ | ✓ | **Video is currently supported on Gemini models only.** Sending a video to a model that does not support it (for example an OpenAI model) returns a `400` error, Maitai will not silently route it elsewhere. Choose a Gemini model for video. Supported file types: * **Images**: PNG, JPEG, GIF, WebP * **Video**: MP4, WebM, MOV (Gemini) * **Documents**: PDF ## The `file` content part A file is just another entry in a message's `content` array. It has the shape: ```json theme={null} { "type": "file", "file": { "file_data": "data:video/mp4;base64,<...>", "filename": "clip.mp4" } } ``` You normally don't write this by hand, use the helpers below. ### Python Build an inline file part from a local file path. Reads the file and base64-encodes it. The MIME type is inferred from the file extension when not provided. ```python theme={null} import maitai # Image maitai.file_content_part_from_path("diagram.png") # Document maitai.file_content_part_from_path("report.pdf") # Explicit MIME type maitai.file_content_part_from_path("recording", mime_type="video/mp4") ``` ### Node Build an inline file part from raw bytes (`Uint8Array` or `Buffer`). The MIME type is required; `filename` is optional but recommended for documents. ```javascript theme={null} import fs from "node:fs"; import { fileContentPartFromBytes } from "maitai"; const bytes = fs.readFileSync("diagram.png"); fileContentPartFromBytes(bytes, "image/png", "diagram.png"); ``` ## Multiple files in one message Combine text and several files in a single message. Order is preserved. ```python Python theme={null} messages = [ { "role": "user", "content": [ {"type": "text", "text": "Compare these two screenshots."}, maitai.file_content_part_from_path("before.png"), maitai.file_content_part_from_path("after.png"), ], } ] ``` ```javascript Node theme={null} import fs from "node:fs"; import Maitai, { fileContentPartFromBytes } from "maitai"; const messages = [ { role: "user", content: [ { type: "text", text: "Compare these two screenshots." }, fileContentPartFromBytes(fs.readFileSync("before.png"), "image/png", "before.png"), fileContentPartFromBytes(fs.readFileSync("after.png"), "image/png", "after.png"), ], }, ]; ``` ## Reuse a file across requests When you'll reference the same file in more than one request, upload it once with `files.upload(...)` to get a `file_id`, then pass that `file_id` with `file_content_part` (Python) / `fileContentPart` (Node). This avoids re-sending the bytes (and for video, avoids re-preparing it for the provider) on every call. ```python Python theme={null} import maitai client = maitai.Maitai() # Upload once file_id = client.files.upload("clip.mp4") # Reference it across as many requests as you like for question in ["What happens first?", "Who is speaking?", "Summarize it."]: response = client.chat.completions.create( messages=[ { "role": "user", "content": [ {"type": "text", "text": question}, maitai.file_content_part(file_id), ], } ], application="demo_app", intent="VIDEO_ANALYSIS", model="gemini-3.5-flash", server_side_inference=True, ) print(response.choices[0].message.content) ``` ```javascript Node theme={null} import fs from "node:fs"; import Maitai, { fileContentPart } from "maitai"; const maitai = new Maitai(); // Upload once (from a path, or use uploadBytes for in-memory data) const fileId = await maitai.files.upload("clip.mp4"); // Reference it across as many requests as you like for (const question of ["What happens first?", "Who is speaking?", "Summarize it."]) { const response = await maitai.chat.completions.create({ messages: [ { role: "user", content: [ { type: "text", text: question }, fileContentPart(fileId), ], }, ], application: "demo_app", intent: "VIDEO_ANALYSIS", model: "gemini-3.5-flash", server_side_inference: true, }); console.log(response.choices[0].message.content); } ``` ### Upload methods Upload a file from a local path; returns the `file_id`. MIME type is inferred from the extension when not provided. Upload raw bytes; returns the `file_id`. Upload a file from a local path; resolves to the `file_id`. Upload raw bytes (`Uint8Array` or `Buffer`); resolves to the `file_id`. `maitai.file_content_part(file_id)` (Python) / `fileContentPart(fileId)` (Node) builds the content part that references an uploaded file. ## Image URLs For images already hosted at a public URL, you can use the standard OpenAI `image_url` content part instead of uploading bytes: ```python Python theme={null} messages = [ { "role": "user", "content": [ {"type": "text", "text": "What's in this image?"}, { "type": "image_url", "image_url": {"url": "https://example.com/photo.jpg"}, }, ], } ] ``` ```javascript Node theme={null} const messages = [ { role: "user", content: [ { type: "text", text: "What's in this image?" }, { type: "image_url", image_url: { url: "https://example.com/photo.jpg" } }, ], }, ]; ``` ## Errors The target model does not support that file type. For example, sending a video to an OpenAI model. Switch to a model that supports it (a Gemini model for video), or remove the file. ## Notes * **Video is uploaded once and prepared for the model**, which can take a few seconds for larger files; this happens on the first request that references it. * Files are scoped to your company and are not shared across accounts. * File inputs work with the rest of the request as usual, combine them with [Structured Output](/sdk/structured_output), [Tool Calling](/sdk/tool_calling), and streaming. # Logging (Bring Your Own Inference) Source: https://docs.trymaitai.com/sdk/indexing Run inference with your own provider client, then send Maitai the raw request/response with maitai.log() for monitoring, debugging, and Test Sets, without routing inference through Maitai. Already running inference with your own provider client (OpenAI, Anthropic, Gemini, or any OpenAI-compatible API) and don't want to move it behind Maitai's inference service? Use **`maitai.log()`** to hand Maitai the request/response pair after each call. Maitai normalizes the native payloads by `provider` and runs them through the **same pipeline as proxied traffic**, so your existing client keeps doing inference while you still get Maitai observability: [Sentinels](/observe/sentinels/overview), [Sessions](/observe/agent_sessions), [request history](/observe/request_overview), and data for [Test Sets](/test/test_sets/creation). Maitai never calls the model on this path, you do. That means **no provider keys are needed** by the Maitai client (only your `MAITAI_API_KEY`), and inference latency is entirely yours. Logging is fire-and-forget and is engineered to **never raise into your application**. ## How it works Call OpenAI / Anthropic / Gemini (or any OpenAI-compatible API) exactly as you do today, with your own keys. Pass the native `request` and `response` along with the `intent`, `application`, and `provider`. The call returns immediately; the send happens on a background thread (sync) or task (async). The native payloads are converted into Maitai's OpenAI-compatible schema based on `provider` and stored as **PROD** traffic (inference location `CLIENT`). The logged pair flows through the same pipeline as Maitai-routed requests, so Sentinels evaluate it, Sessions thread it, and it's available for debugging and building Test Sets. ## Quickstart Run inference with your own client, then log the pair. Reusing the same `request` dict you pass to your provider keeps the logged inputs perfectly in sync with what you sent. ```python Python (sync) theme={null} import maitai from openai import OpenAI maitai_client = maitai.Maitai() # only needs MAITAI_API_KEY openai_client = OpenAI() # your own client + OPENAI_API_KEY request = { "model": "gpt-4o", "messages": [{"role": "user", "content": "Summarize this text..."}], } # You run inference, not Maitai. response = openai_client.chat.completions.create(**request) # Fire-and-forget: returns immediately, never raises. maitai_client.log( intent="SUMMARIZATION", application="demo_app", provider="openai", request=request, response=response, # SDK objects are serialized automatically session_id="YOUR_SESSION_ID", metadata={"source": "byo_inference"}, ) ``` ```python Python (async) theme={null} import maitai from openai import AsyncOpenAI maitai_client = maitai.MaitaiAsync() # only needs MAITAI_API_KEY openai_client = AsyncOpenAI() # your own client + OPENAI_API_KEY request = { "model": "gpt-4o", "messages": [{"role": "user", "content": "Summarize this text..."}], } response = await openai_client.chat.completions.create(**request) await maitai_client.log( intent="SUMMARIZATION", application="demo_app", provider="openai", request=request, response=response, session_id="YOUR_SESSION_ID", metadata={"source": "byo_inference"}, ) ``` ## Provider examples Send the provider's **native** payloads, Maitai converts them server-side. `request` and `response` accept either plain dicts or the provider SDK's objects (they're serialized for you). ```python OpenAI-compatible theme={null} # provider="openai" covers OpenAI and any OpenAI-compatible API # (Groq, Together, vLLM, ...). Point your own client wherever you like. import maitai from openai import OpenAI maitai_client = maitai.Maitai() openai_client = OpenAI() request = { "model": "gpt-4o", "messages": [{"role": "user", "content": "Summarize this text..."}], } response = openai_client.chat.completions.create(**request) maitai_client.log( intent="SUMMARIZATION", application="demo_app", provider="openai", request=request, response=response, session_id="YOUR_SESSION_ID", ) ``` ```python Anthropic theme={null} import maitai import anthropic maitai_client = maitai.Maitai() anthropic_client = anthropic.Anthropic() request = { "model": "claude-sonnet-4-20250514", "max_tokens": 1024, "system": "You are a helpful assistant.", "messages": [{"role": "user", "content": "Summarize this text..."}], } message = anthropic_client.messages.create(**request) maitai_client.log( intent="SUMMARIZATION", application="demo_app", provider="anthropic", request=request, response=message, # native Anthropic Message; tool_use + thinking are mapped session_id="YOUR_SESSION_ID", ) ``` ```python Gemini theme={null} import maitai from google import genai maitai_client = maitai.Maitai() genai_client = genai.Client() request = { "model": "gemini-2.0-flash", # Structured contents normalize cleanly into messages. "contents": [ {"role": "user", "parts": [{"text": "Summarize this text..."}]} ], } response = genai_client.models.generate_content(**request) maitai_client.log( intent="SUMMARIZATION", application="demo_app", provider="gemini", request=request, response=response, # native GenerateContentResponse session_id="YOUR_SESSION_ID", ) ``` ## Parameters The Intent / action type for the request (also called `action_type`). Scopes Sentinels, configuration, and quality tooling. The Application reference name (shown in the Portal). Groups your traffic. Source provider for the pair, used to normalize the native payloads. One of `openai`, `anthropic`, or `gemini`. Use `openai` for any **OpenAI-compatible** API (Groq, Together, vLLM, etc.) where payloads are already in OpenAI shape. The provider's native request payload, the dict you sent to your client, or the SDK request object. Serialized automatically. The provider's native response, the SDK response object (e.g. OpenAI `ChatCompletion`, Anthropic `Message`, Gemini `GenerateContentResponse`) or its dict form. Serialized automatically. Optional but recommended. Groups related requests into a [Session](/observe/agent_sessions). If omitted, Maitai generates one server-side. Optional caller-supplied identifier for correlating the logged request with your own systems. Optional end-user identifier stored with the request. Optional metadata tags stored with the request for filtering and debugging. Optional per-call client-side sampling rate between `0.0` and `1.0`. Overrides the default (see [Sampling and kill-switch](#sampling-and-kill-switch)). The surviving rate is recorded so server-side aggregates can correct for sampling. Optional latency metrics. Provide a `RequestTimingMetric` (`from maitai.models.metric import RequestTimingMetric`) to record response time. Epoch seconds when you issued the request. Required for latency to be recorded. Epoch seconds when the response completed. Required for latency to be recorded. Epoch seconds of the first streamed chunk, used to derive time-to-first-token. Timing is only recorded when **both** `time_request_start` and `time_request_end` are set; partial metrics are ignored. ## Behavior and guarantees * **Non-blocking.** `maitai_client.log(...)` returns immediately. Sync clients send on a background thread; async clients schedule a task. * **Never raises into your code.** Bad inputs, network errors, and serialization failures are swallowed and reported to Maitai's internal error metric. Instrumentation can't take down your request path. * **No inference, no inline corrections.** Maitai does not call the model and does not apply corrections here (corrections require server-side inference, see [Model Request](/sdk/chat)). [Sentinels](/observe/sentinels/overview) still evaluate the stored traffic asynchronously. * **Stored as PROD / CLIENT.** Logged pairs are recorded as production traffic with the inference location marked `CLIENT`, so they're indistinguishable downstream from Maitai-routed requests for monitoring and Test Sets. ## Sampling and kill-switch Dial volume without touching call sites using environment variables: | Variable | Default | Effect | | ------------------------ | ------- | ------------------------------------------------------------------------------------------ | | `MAITAI_LOGGING_ENABLED` | `true` | Set to a falsey value (`0`, `false`, `no`, `off`, empty) to disable all logging. | | `MAITAI_LOG_SAMPLE_RATE` | `1.0` | Fraction of events to send (`0.0`–`1.0`). A per-call `sample_rate` overrides this default. | Short-lived processes (scripts, lambdas) don't need to do anything special, queued logs are drained automatically at interpreter exit, so in-flight events aren't dropped. ## Raw HTTP (any language) The `maitai.log()` helper ships in the Python SDK. From Node, Go, or any other stack, `PUT` the same payload to the endpoint directly: ``` PUT https://api.trymaitai.ai/chat/completions/log ``` The HTTP body uses the underlying field names: `intent` maps to `action_type` and `application` maps to `application_ref_name`. `request` and `response` are the provider's native payloads as JSON. ```bash cURL theme={null} curl --request PUT \ --url https://api.trymaitai.ai/chat/completions/log \ --header "Content-Type: application/json" \ --header "x-api-key: $MAITAI_API_KEY" \ --data '{ "provider": "openai", "action_type": "SUMMARIZATION", "application_ref_name": "demo_app", "session_id": "YOUR_SESSION_ID", "metadata": { "source": "byo_inference" }, "request": { "model": "gpt-4o", "messages": [ { "role": "user", "content": "Summarize this text..." } ] }, "response": { "id": "chatcmpl_example", "object": "chat.completion", "created": 1730000000, "model": "gpt-4o", "choices": [ { "index": 0, "message": { "role": "assistant", "content": "Here is the summary..." }, "finish_reason": "stop" } ], "usage": { "prompt_tokens": 42, "completion_tokens": 18, "total_tokens": 60 } } }' ``` ```javascript Node (fetch) theme={null} const request = { model: "gpt-4o", messages: [{ role: "user", content: "Summarize this text..." }], }; // You run inference with your own client. const response = await openai.chat.completions.create(request); // Fire-and-forget: don't await if you don't want to block. fetch("https://api.trymaitai.ai/chat/completions/log", { method: "PUT", headers: { "Content-Type": "application/json", "x-api-key": process.env.MAITAI_API_KEY, }, body: JSON.stringify({ provider: "openai", action_type: "SUMMARIZATION", application_ref_name: "demo_app", session_id: "YOUR_SESSION_ID", request, response, metadata: { source: "byo_inference" }, }), }).catch(() => {}); // logging should never break your request path ``` On success the endpoint returns `{ "success": true }`. It returns `400` for caller mistakes, an unsupported `provider`, or a missing `action_type` / `application_ref_name`. For **anthropic** and **gemini**, send each provider's native request/response shape (Anthropic Messages API, google-genai `GenerateContent`). Maitai maps finish reasons, `tool_use`/`function_call` to tool calls, usage, and Anthropic `thinking` into its OpenAI-compatible schema. ## When to use this vs. routing through Maitai Logging is the right fit when you must keep inference on your own infrastructure or client but still want full observability. If you can route inference through Maitai instead (`server_side_inference=true`, the default), you additionally unlock server-side features like automatic corrections and `input_safety_score`, see [Model Request](/sdk/chat). # Observe (Bring Your Own Agent) Source: https://docs.trymaitai.com/sdk/observe Instrument existing agent frameworks with one call — optional agent name, intent, and session stamps so multi-agent apps stay distinct in the Portal. Already running LangChain, LangGraph, CrewAI, OpenAI Agents, or another supported framework? Call **`maitai.observe()`** once at process start. Maitai instruments installed frameworks and exports OpenInference/OTel spans to `POST /api/v1/traces`. Observed runs show up as agents in the Portal (same Runs / session / cost surfaces as native agents). Requires the optional extra: `pip install 'maitai-python[observe]'`. Capture is fire-and-forget — never raises into your application. Use `MAITAI_LOGGING_ENABLED` / sample-rate env vars the same way as `maitai.log()`. ## One-line setup ```python theme={null} import maitai maitai.observe(application="support-app") # your existing agent.invoke() / graph.invoke() / crew.kickoff() … ``` First trace with an unknown agent name auto-creates an observed agent — no pre-registration. ## Agent name (multi-agent processes) When frameworks emit generic span names, two agents in one process can collapse into a single observed agent. Stamp an explicit name: ```python theme={null} import maitai maitai.observe(application="support-app") maitai.set_agent("triage") # sticky until cleared # … with maitai.agent("billing"): # scoped (callable module — same attr as hosted agent client) billing_graph.invoke(payload) ``` Stamps OpenInference `agent.name` on spans (does not override a name the framework already set). The backend already prefers `agent.name` when resolving the observed agent. `maitai.agent` is also the hosted-agent client package (`Completions`). Calling it as `maitai.agent("name")` enters the observe naming context; importing `from maitai.agent import Completions` is unchanged. ## Intent override Observed LLM steps default `action_type` to the span name. Override when you want a stable Maitai intent: ```python theme={null} with maitai.intent("classify_ticket"): graph.invoke(ticket) ``` Also available as sticky `maitai.set_intent("classify_ticket")`. Backend reads `maitai.intent` when re-logging LLM steps through the BYO chat-log path. ## Session contract | Situation | `session_id` | | -------------------------------------------------------------------------------------------- | ---------------------------------------------------------- | | You call `maitai.set_session(...)` / `with maitai.session(...):` (or `observe(session=...)`) | Your id — groups many invocations into one Portal session | | Framework sets `session.id` / conversation id attrs | That value | | Nothing set | **`session_id = trace_id`** (one invocation = one session) | ```python theme={null} maitai.set_session("user-42-thread-7") agent.invoke(...) # same session agent.invoke(...) ``` ## Workflows vs agents A fixed pipeline should be a **workflow**, not an observed agent. Declare it: ```python theme={null} maitai.observe( application="checkout-app", workflows=["order-graph"], # stamps maitai.run_type=workflow when names match ) # or: maitai.observe(application="checkout-app", run_type="workflow") ``` When a workflow graph also contains nested agent boundaries (for example a LangGraph sub-agent inside a fixed pipeline), ingestion **dual-persists**: a `workflow_run` for the container plus observed agents for nested boundaries. Profile spans carry `refs` / `nav_path` so the Portal can drill into those agents. The workflow root is not double-counted as an agent. See [Workflow logging (BYOW)](/sdk/workflow_logging) for code-native and no-code workflow capture without `observe()`. # SDK Reference Source: https://docs.trymaitai.com/sdk/overview The Maitai SDK, from chat completions to agents, tools, files, and logging. The Maitai SDK is a drop-in for the OpenAI client with Maitai's routing, evaluation, and observability built in. These pages document each capability. ## Core Create chat completions with Maitai metadata (Application, Intent, Session). Get typed, schema-validated JSON back from a model. Let the model call your functions and tools. Use chain-of-thought and extended thinking with one provider-agnostic API. ## Inputs & modalities Send images, video, and documents to multimodal models. Transcribe audio through Maitai. ## Agents & evaluation Invoke an agent from code and pin versions or releases. Run evaluations against a request from the SDK. Run inference with your own client, then send Maitai the request/response. # Reasoning Source: https://docs.trymaitai.com/sdk/reasoning Use chain-of-thought and extended thinking with a single, provider-agnostic API. ## Overview **Reasoning** lets the model spend more compute on internal chain-of-thought or extended thinking before producing the final answer. Maitai exposes one **`reasoning`** object on chat requests. The platform maps it to each provider’s native interface (for example reasoning controls on OpenAI reasoning models, thinking blocks on Anthropic, or extracted reasoning content where the model exposes it). You do not need to branch your integration by vendor. ## Request parameters Pass **`reasoning`** inside the same argument list as your chat completion (SDK) or inside the request body’s **`params`** object (REST). It has two fields: How much the model should think before answering. Typical values: `"low"`, `"medium"`, `"high"`. Exact behavior depends on the model; Maitai forwards this through the unified config. When `true`, the response can include the model’s reasoning text in the top-level **`reasoning`** field (see [Response fields](#response-fields)). ## SDK examples ### Basic usage (effort only) ```python Python (async) theme={null} import maitai client = maitai.MaitaiAsync() messages = [ {"role": "user", "content": "Explain briefly how you would solve 17 * 24."}, ] response = await client.chat.completions.create( messages=messages, application="demo_app", intent="REASONING_DEMO", session_id="YOUR_SESSION_ID", model="gpt-5", reasoning={"effort": "high"}, ) print(response.choices[0].message.content) ``` ```python Python (sync) theme={null} import maitai client = maitai.Maitai() messages = [ {"role": "user", "content": "Explain briefly how you would solve 17 * 24."}, ] response = client.chat.completions.create( messages=messages, application="demo_app", intent="REASONING_DEMO", session_id="YOUR_SESSION_ID", model="gpt-5", reasoning={"effort": "high"}, ) print(response.choices[0].message.content) ``` ```javascript Node theme={null} import Maitai from "maitai"; const maitai = new Maitai(); const messages = [ { role: "user", content: "Explain briefly how you would solve 17 * 24." }, ]; const response = await maitai.chat.completions.create({ messages, application: "demo_app", intent: "REASONING_DEMO", session_id: "YOUR_SESSION_ID", model: "gpt-5", reasoning: { effort: "high" }, }); console.log(response.choices[0].message.content); ``` ### Returning reasoning in the response Set **`summary: true`** (Python: `True`) so the completion may include the model’s reasoning text on the response object. ```python Python (async) theme={null} response = await client.chat.completions.create( messages=messages, application="demo_app", intent="REASONING_DEMO", session_id="YOUR_SESSION_ID", model="gpt-5", reasoning={"effort": "high", "summary": True}, ) print(response.reasoning) print(response.choices[0].message.content) ``` ```python Python (sync) theme={null} response = client.chat.completions.create( messages=messages, application="demo_app", intent="REASONING_DEMO", session_id="YOUR_SESSION_ID", model="gpt-5", reasoning={"effort": "high", "summary": True}, ) print(response.reasoning) print(response.choices[0].message.content) ``` ```javascript Node theme={null} const response = await maitai.chat.completions.create({ messages, application: "demo_app", intent: "REASONING_DEMO", session_id: "YOUR_SESSION_ID", model: "gpt-5", reasoning: { effort: "high", summary: true }, }); console.log(response.reasoning); console.log(response.choices[0].message.content); ``` ## API example Server-side chat completions are sent to Maitai with `application_ref_name`, `session_id`, `action_type` (intent), and a `params` object that mirrors SDK parameters. Include **`reasoning`** inside **`params`**: ```bash theme={null} curl -sS -X POST "https://api.trymaitai.ai/api/v1/chat/completions/serialized" \ -H "Content-Type: application/json" \ -H "X-Maitai-Api-Key: $MAITAI_API_KEY" \ -d '{ "application_ref_name": "demo_app", "session_id": "YOUR_SESSION_ID", "action_type": "REASONING_DEMO", "evaluation_enabled": true, "params": { "messages": [ { "role": "user", "content": "Short plan to boil an egg." } ], "model": "gpt-5", "stream": false, "reasoning": { "effort": "high", "summary": true } }, "auth_keys": {} }' ``` Adjust `model` and message content to match your environment. ## Response fields When **`reasoning.summary`** is enabled and the model returns thinking content, the completion may include: The model’s chain-of-thought or thinking text. Omitted or `null` when summaries are off or the model does not expose reasoning in the response. Usage may break out reasoning token counts: Number of completion tokens attributed to reasoning, when the provider supplies this breakdown. Standard `choices[0].message.content` still holds the final assistant message. ## Portal configuration In the Maitai Portal, you can set a default **reasoning effort** at the **intent** or **intent group** level as part of model / request configuration (the UI uses the legacy **`reasoning_effort`** field for that default). Per-request **`reasoning`** from the SDK or API overrides those defaults when both are present. ## Related * [Model Request](/sdk/chat), full chat parameters and Maitai-specific fields # Structured Output Source: https://docs.trymaitai.com/sdk/structured_output Generate consistent JSON using JSON Schema (or Pydantic models in Python). Maitai supports JSON Schema-based structured output. In Python, you can also pass a Pydantic model and parse the result via `message.parsed`. ```python Python (Pydantic) theme={null} import maitai from pydantic import BaseModel class CalendarEvent(BaseModel): name: str date: str participants: list[str] maitai_client = maitai.Maitai() completion = maitai_client.chat.completions.create( application="DEMO_APP", intent="CALENDAR", model="llama3.1-8b", messages=[ {"role": "system", "content": "Extract the event information."}, {"role": "user", "content": "AI learning seminar, October 20."}, ], response_format=CalendarEvent, ) event: CalendarEvent = completion.choices[0].message.parsed ``` ## JSON Schema example ```python Python theme={null} import maitai schema = { "name": "athlete_schema", "strict": True, "schema": { "type": "object", "properties": { "athletes": { "type": "array", "items": { "type": "object", "properties": { "sport": {"type": "string"}, "name": {"type": "string"}, }, "required": ["sport", "name"], "additionalProperties": False, }, } }, "required": ["athletes"], "additionalProperties": False, }, } maitai_client = maitai.Maitai() response = maitai_client.chat.completions.create( session_id="YOUR_SESSION_ID", intent="FACTS", application="DEMO_APP", model="claude-3-5-sonnet-20240620", messages=[{"role": "user", "content": "Give me a list of 5 famous athletes and their sport"}], response_format={"type": "json_schema", "json_schema": schema} ) data = response.choices[0].message.parsed ``` ```javascript Node theme={null} import Maitai from "maitai"; const maitai = new Maitai(); const schema = { name: "athlete_schema", strict: true, schema: { type: "object", properties: { athletes: { type: "array", items: { type: "object", properties: { sport: { type: "string" }, name: { type: "string" }, }, required: ["sport", "name"], additionalProperties: false, }, }, }, required: ["athletes"], additionalProperties: false, }, }; const response = await maitai.chat.completions.create({ session_id: "YOUR_SESSION_ID", intent: "FACTS", application: "DEMO_APP", model: "claude-3-5-sonnet-20240620", messages: [ { role: "user", content: "Give me a list of 5 famous athletes and their sport", }, ], response_format: { type: "json_schema", json_schema: schema }, }); const data = response.choices[0].message.parsed; ``` # Tool Calling Source: https://docs.trymaitai.com/sdk/tool_calling Define tools and let the model call them (OpenAI-style tool calling). Maitai supports OpenAI-style tool calling (`tools` + `tool_choice`). In Python, you can define tools either as raw OpenAI tool definitions, or using Maitai’s `@tool` decorator. ## Option A: Raw OpenAI tool definitions ```python theme={null} import json import maitai def get_current_weather(location: str, unit: str = "fahrenheit") -> str: return json.dumps({"location": location, "unit": unit, "temperature": "72"}) weather_tool = { "type": "function", "function": { "name": "get_current_weather", "description": "Get the current weather in a given location", "parameters": { "type": "object", "properties": { "location": {"type": "string", "description": "The city and state"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}, }, "required": ["location"], }, }, } available_functions = { "get_current_weather": get_current_weather, } client = maitai.MaitaiAsync() messages = [{"role": "user", "content": "What's the weather in San Francisco?"}] response = await client.chat.completions.create( messages=messages, application="demo_app", intent="WEATHER", tool_choice="auto", tools=[weather_tool], ) for call in response.choices[0].message.tool_calls or []: fn = available_functions[call.function.name] args = json.loads(call.function.arguments) tool_result = fn(**args) ``` ## Option B (recommended): `@tool` + `Tools` wrapper ```python theme={null} import maitai from maitai.tools import Tools, tool @tool(strict=True) def get_current_weather(location: str, unit: str = "fahrenheit") -> str: """ Get the current weather in a given location. Args: location: The city and state, e.g. San Francisco, CA unit: The unit of measurement (default is "fahrenheit") """ return f"{location}: 72° ({unit})" tools = Tools(get_current_weather) client = maitai.MaitaiAsync() messages = [{"role": "user", "content": "What's the weather in San Francisco?"}] response = await client.chat.completions.create( messages=messages, application="demo_app", intent="WEATHER", tool_choice="auto", tools=tools, ) for call in response.choices[0].message.tool_calls or []: tool_result = tools.invoke(call) ``` # Speech to text Source: https://docs.trymaitai.com/sdk/transcription Convert audio files to text using Maitai's speech-to-text API. ```python Python (async) theme={null} import maitai maitai_client = maitai.MaitaiAsync() with open("/path/to/file/audio.mp3", "rb") as audio_file: transcription = await maitai_client.audio.transcriptions.create( file=audio_file, model="breeze", response_format="json", application="demo_app", intent="TRANSCRIPTION", session_id="YOUR_SESSION_ID" ) print(transcription.text) ``` ```python Python (sync) theme={null} import maitai maitai_client = maitai.Maitai() with open("/path/to/file/audio.mp3", "rb") as audio_file: transcription = maitai_client.audio.transcriptions.create( file=audio_file, model="breeze", response_format="json", application="demo_app", intent="TRANSCRIPTION", session_id="YOUR_SESSION_ID" ) print(transcription.text) ``` ```bash cURL theme={null} curl --request POST \ --url https://api.trymaitai.ai/audio/transcriptions \ --header "Authorization: Bearer $MAITAI_API_KEY" \ --header 'Content-Type: multipart/form-data' \ --form file=@/path/to/file/audio.mp3 \ --form model=breeze \ --form application=demo_app \ --form intent=TRANSCRIPTION ``` ```json theme={null} { "text": "Imagine the wildest idea that you've ever had, and you're curious about how it might scale to something that's a 100, a 1,000 times bigger.", "language": "en" } ``` The Audio API provides transcription capabilities with support for multiple providers and models. You can transcribe audio into whatever language the audio is in, or translate and transcribe the audio into English. The following input file types are supported: `mp3`, `mp4`, `mpeg`, `mpga`, `m4a`, `wav`, and `webm`. ## Supported Models Maitai supports a variety of speech-to-text models across different providers: ### Maitai Models * `breeze` - Custom transcription model by Maitai ### Groq Models * `whisper-large-v3` - High-quality transcription model * `whisper-large-v3-turbo` - Fast transcription model * `distil-whisper-large-v3-en` - English-optimized transcription model ### OpenAI Models * `whisper-1` - OpenAI's Whisper model * `gpt-4o-mini-transcribe` - GPT-4o mini transcription model * `gpt-4o-transcribe` - GPT-4o transcription model ### Deepgram Models * `nova-3` - Latest model from Deepgram * `nova-2` - Deepgram's Nova 2 model ## Quickstart ### Using the Maitai SDK `application` is required for transcription requests. We strongly recommend also passing `intent` and `session_id` so the traffic is organized cleanly in the Portal (similar to chat completions). ```python Python (async) theme={null} import maitai maitai_client = maitai.MaitaiAsync() with open("/path/to/file/audio.mp3", "rb") as audio_file: transcription = await maitai_client.audio.transcriptions.create( file=audio_file, model="breeze", response_format="json", application="demo_app", intent="TRANSCRIPTION", session_id="YOUR_SESSION_ID" ) print(transcription.text) ``` ```python Python (sync) theme={null} import maitai maitai_client = maitai.Maitai() with open("/path/to/file/audio.mp3", "rb") as audio_file: transcription = maitai_client.audio.transcriptions.create( file=audio_file, model="breeze", response_format="json", application="demo_app", intent="TRANSCRIPTION", session_id="YOUR_SESSION_ID" ) print(transcription.text) ``` ### Using OpenAI SDK with Maitai You can also use the OpenAI SDK with Maitai by setting the base URL and adding the required Maitai parameters as headers: ```python theme={null} import openai client = openai.OpenAI( base_url="https://api.trymaitai.ai", api_key="your_maitai_api_key" ) # Add Maitai-specific headers headers = { "X-Maitai-Application": "demo_app", "X-Maitai-Intent": "TRANSCRIPTION", "X-Maitai-Session-Id": "YOUR_SESSION_ID", "X-Maitai-Metadata-custom_tag": "value" } with open("/path/to/file/audio.mp3", "rb") as audio_file: transcription = client.audio.transcriptions.create( file=audio_file, model="breeze", response_format="json", extra_headers=headers ) print(transcription.text) ``` ## Create transcription Creates a transcription of the given audio file. ### Maitai Specific Parameters The reference to the application the request is created from. Optional. Used to organize traffic in the Portal. **Examples**: `TRANSCRIPTION`, `TRANSLATION` Optional but recommended. Used to group related requests into a session in the Portal. ### Model Provider Parameters The audio file to transcribe, in one of the supported formats: mp3, mp4, mpeg, mpga, m4a, wav, or webm. ID of the model to use. See [Supported Models](#supported-models) for available options. The format of the transcript output, in one of these options: json, text, srt, verbose\_json, or vtt. The language of the input audio. Supplying the input language in ISO-639-1 format will improve accuracy and latency. An optional text to guide the model's style or continue a previous audio segment. The prompt should match the audio language. The sampling temperature, between 0 and 1. Higher values like 0.8 will make the output more random, while lower values like 0.2 will make it more focused and deterministic. If set to 0, the model will use log probability to automatically increase the temperature until certain thresholds are hit. The timestamp granularities to populate for this transcription. response\_format must be set verbose\_json to use timestamp granularities. One or more of the following: word, or segment. If set, partial message deltas will be sent, like in ChatGPT. Tokens will be sent as data-only server-sent events as they become available, with the stream terminated by a `data: [DONE]` message. ### Returns Returns a transcription object, or a streamed sequence of transcription chunk objects if the request is streamed. ## The transcription object Represents a transcription response returned by model, based on the provided input. The transcribed text. The language of the input audio. The duration of the input audio. Extracted words and their corresponding timestamps. Segmented transcript content, with start and end timestamps. ## Streaming transcriptions You can stream transcriptions to receive partial results as they become available. This is useful for real-time applications or when processing long audio files. ### Using Maitai SDK for streaming ```python theme={null} import maitai maitai_client = maitai.Maitai() with open("/path/to/file/audio.mp3", "rb") as audio_file: stream = maitai_client.audio.transcriptions.create( file=audio_file, model="breeze", response_format="json", stream=True, application="demo_app", intent="TRANSCRIPTION", session_id="YOUR_SESSION_ID" ) for chunk in stream: print(f"Chunk: {chunk}") if hasattr(chunk, 'delta') and chunk.delta: print(f"Delta: {chunk.delta}") ``` ### Using OpenAI SDK for streaming ```python theme={null} import openai client = openai.OpenAI( base_url="https://api.trymaitai.ai", api_key="your_maitai_api_key" ) headers = { "X-Maitai-Application": "demo_app", "X-Maitai-Intent": "TRANSCRIPTION", "X-Maitai-Session-Id": "YOUR_SESSION_ID" } with open("/path/to/file/audio.mp3", "rb") as audio_file: stream = client.audio.transcriptions.create( file=audio_file, model="breeze", response_format="json", stream=True, extra_headers=headers ) for chunk in stream: print(f"Chunk: {chunk}") ``` ## Supported languages Maitai supports transcription in the following languages through our supported models: Afrikaans, Arabic, Armenian, Azerbaijani, Belarusian, Bosnian, Bulgarian, Catalan, Chinese, Croatian, Czech, Danish, Dutch, English, Estonian, Finnish, French, Galician, German, Greek, Hebrew, Hindi, Hungarian, Icelandic, Indonesian, Italian, Japanese, Kannada, Kazakh, Korean, Latvian, Lithuanian, Macedonian, Malay, Marathi, Maori, Nepali, Norwegian, Persian, Polish, Portuguese, Romanian, Russian, Serbian, Slovak, Slovenian, Spanish, Swahili, Swedish, Tagalog, Tamil, Thai, Turkish, Ukrainian, Urdu, Vietnamese, and Welsh. While the underlying models were trained on 98 languages, we only list the languages that exceeded 50% [word error rate](https://en.wikipedia.org/wiki/Word_error_rate) (WER) which is an industry standard benchmark for speech to text model accuracy. The models will return results for languages not listed above but the quality will be low. ## Timestamps By default, the Transcriptions API will output a transcript of the provided audio in text. The `timestamp_granularities[]` parameter enables a more structured and timestamped json output format, with timestamps at the segment, word level, or both. This enables word-level precision for transcripts and video edits, which allows for the removal of specific frames tied to individual words. ```python theme={null} import maitai maitai_client = maitai.Maitai() with open("/path/to/file/audio.mp3", "rb") as audio_file: transcription = maitai_client.audio.transcriptions.create( file=audio_file, model="whisper-large-v3", response_format="verbose_json", timestamp_granularities=["word"], application="demo_app", intent="TRANSCRIPTION", session_id="YOUR_SESSION_ID" ) print(transcription.words) ``` ## Longer inputs For very long audio files, you may want to break them up into smaller chunks to get the best performance. We suggest that you avoid breaking the audio up mid-sentence as this may cause some context to be lost. One way to handle this is to use the [PyDub open source Python package](https://github.com/jiaaro/pydub) to split the audio: ```python theme={null} from pydub import AudioSegment song = AudioSegment.from_mp3("good_morning.mp3") # PyDub handles time in milliseconds ten_minutes = 10 * 60 * 1000 first_10_minutes = song[:ten_minutes] first_10_minutes.export("good_morning_10.mp3", format="mp3") ``` ## Prompting You can use a `prompt` parameter to improve the quality of the transcripts generated by the Transcriptions API. ```python theme={null} import maitai maitai_client = maitai.Maitai() with open("/path/to/file/speech.mp3", "rb") as audio_file: transcription = maitai_client.audio.transcriptions.create( file=audio_file, model="breeze", response_format="text", prompt="The following conversation is a lecture about the recent developments around AI and machine learning.", application="demo_app", intent="TRANSCRIPTION", session_id="YOUR_SESSION_ID" ) print(transcription.text) ``` Here are some examples of how prompting can help in different scenarios: * **Correcting specific words or acronyms**: Prompts can help correct specific words or acronyms that the model misrecognizes in the audio. * **Preserving context**: To preserve the context of a file that was split into segments, prompt the model with the transcript of the preceding segment. * **Punctuation**: Sometimes the model skips punctuation in the transcript. To prevent this, use a simple prompt that includes punctuation. * **Filler words**: The model may also leave out common filler words in the audio. If you want to keep the filler words in your transcript, use a prompt that contains them. * **Writing style**: Some languages can be written in different ways, such as simplified or traditional Chinese. You can improve this by using a prompt in your preferred writing style. ## Notes * **SDK support**: Maitai provides both Python and TypeScript SDKs. You can also use the OpenAI SDK with the [Base URL approach](/get_started/base_url). * **Model availability**: Not all models support all features. For example, timestamp granularities are only available with certain models and response formats. * **Streaming support**: Streaming is not supported by all models. Check the model documentation for specific capabilities. # Workflow logging (Bring Your Own Workflow) Source: https://docs.trymaitai.com/sdk/workflow_logging Log externally executed workflows into Maitai with a 1–2 line Python SDK, raw HTTP, or n8n — same Portal Runs experience as native workflows. Already running workflows outside Maitai (Python pipelines, n8n, Zapier, Make) and want them indexed in the Portal? Use **Bring Your Own Workflow (BYOW)** to ship completed runs to Maitai with minimal integration cost. Capture is fire-and-forget and engineered to **never raise into your application**. Respects `MAITAI_LOGGING_ENABLED` and `MAITAI_LOG_SAMPLE_RATE`. First log with an unknown ref auto-creates a shell workflow — no pre-registration. ## Python SDK ### Context manager ```python theme={null} import maitai with maitai.workflow_run( "order-pipeline", application="checkout-app", input={"order_id": 1}, ) as run: # your workflow logic… result = {"ok": True} run.output = result ``` Nested `maitai.log(...)` / proxied LLM calls automatically inherit the run's `session_id`, so the cost card and View Session populate without extra wiring. ### Decorator ```python theme={null} import maitai @maitai.workflow("order-pipeline", application="checkout-app") def handle(order_id: int): return {"ok": True, "order_id": order_id} ``` ### One-shot ```python theme={null} import maitai maitai.log_workflow_run( workflow_ref="order-pipeline", application="checkout-app", session_id="run-123", status="COMPLETED", input={"order_id": 1}, output={"ok": True}, ) ``` ### LangGraph / observe as workflow When instrumenting with `maitai.observe()`, declare which graphs are workflows so they persist as `workflow_run` rows instead of observed agents: ```python theme={null} import maitai maitai.observe( application="checkout-app", workflows=["order-graph"], # stamps maitai.run_type=workflow ) # or force every span: # maitai.observe(application="checkout-app", run_type="workflow") ``` ## Raw HTTP ```bash theme={null} curl -X PUT "https://api.trymaitai.ai/workflow/runs/log" \ -H "Content-Type: application/json" \ -H "x-api-key: $MAITAI_API_KEY" \ -d '{ "workflow_ref": "order-pipeline", "application": "checkout-app", "session_id": "run-123", "status": "COMPLETED", "input": {"order_id": 1}, "output": {"ok": true}, "source_tool": "http" }' ``` Idempotent on `session_id`. Prefer a **write-only ingest token** (`WORKFLOW_INGEST`) from Portal → Forge → Workflows → **Connect external** when pasting into third-party SaaS. ## n8n 1. Open Portal → Forge → Workflows → **Connect external**. 2. Pick **n8n**, choose application + workflow ref, mint an ingest token. 3. Add an HTTP Request node as the **last** step, or import the starter workflow from the repo (`docs/external/sdk/n8n/maitai-byow-log.json`) and replace placeholders: * Method: `PUT` * URL: `https://api.trymaitai.ai/workflow/runs/log` (or your local `http://host.docker.internal:5000/workflow/runs/log`) * Header `x-api-key`: your ingest token * Body (JSON). In the Portal wizard artifact, n8n uses expression fields with an `=` prefix. **Never** send bare `$execution.id` as `session_id` (values like `"1"` collide with other traffic and cost backfill). Namespace it: ```json theme={null} { "workflow_ref": "order-pipeline", "application": "checkout-app", "session_id": "={{ 'n8n-' + $workflow.id + '-' + $execution.id }}", "status": "COMPLETED", "input": "={{$json}}", "output": "={{$json}}", "source_tool": "n8n" } ``` The API rejects numeric-only / short (`< 8` char) external `session_id` values. 4. Optional: Error Workflow HTTP node with `"status": "FAILED"` and `error_message`. 5. Use **Wait for first run** in the Portal wizard — it navigates to the rendered run when it lands. ### Local verify (Docker) ```bash theme={null} docker run -it --rm \ -p 5678:5678 \ -e WEBHOOK_URL=http://localhost:5678/ \ -e GENERIC_TIMEZONE=UTC \ n8nio/n8n ``` Open [http://localhost:5678](http://localhost:5678) → **Import from File** → `docs/external/sdk/n8n/maitai-byow-log.json` → set `x-api-key`, `workflow_ref`, and `application` → Execute. Confirm the run appears under the workflow in Portal (or via the wizard listener). From Docker Desktop on Mac/Windows, `host.docker.internal` reaches a backend on the host at `:5000`. ## TypeScript SDK ```ts theme={null} import { logWorkflowRun } from "maitai"; await logWorkflowRun({ workflow_ref: "order-pipeline", application: "checkout-app", session_id: "run-123", status: "COMPLETED", input: { order_id: 1 }, output: { ok: true }, source_tool: "node", }); ``` ## What appears in the Portal External runs show under the linked application and workflow with `source=EXTERNAL`. Input/output panels, cost (when LLM calls share the session), and execution timeline (when a profile/spans are sent) match native runs. Flow-tab graph import for external definitions is a follow-up. # Test Source: https://docs.trymaitai.com/test/overview Curate test sets and run them against a model, workflow, or agent (scored by evaluation criteria) before you ship. **Test** is where you prove a change is good *before* it reaches production. You curate a **test set**, run it against a model, workflow, or agent, and score the results against **evaluation criteria**: so you can compare a candidate against a baseline with confidence. Come here when you want to answer: *"Did my last change make things better or worse?"* Watching **live** production quality (Sentinels and Monitors) lives under [Observe](/observe/overview). Test is for curated, offline evaluation; Observe is for real traffic. ## The building blocks Type-less sets of `input + expected output` items, built from requests, workflow runs, agent tasks, manual entries, or a file. Execute a set against a model, workflow, or agent, then read the results and track regressions. Reusable checks (deterministic, LLM-judge, or agent-trajectory) that score your Test Runs. ## How they fit together * A **Test Set** is a type-less batch of items. * A **Test Run** executes that set against a chosen **consumer** (model, workflow, or agent). * **Evaluation Criteria** score each run, so you can compare baseline vs. candidate and catch regressions. ## Where to go next * Build your first regression suite: [Test Set Creation](/test/test_sets/creation) * Fix a failure and prevent it from returning: [Quality Loop](/examples/quality_loop) * Watch live quality instead: [Sentinels](/observe/sentinels/overview) and [Monitors](/observe/monitors/overview) # Test Set Creation Source: https://docs.trymaitai.com/test/test_sets/creation Build a test set from any mix of production requests, workflow runs, agent tasks, manual entries, or a file upload. A **Test Set** is a reusable collection of test items you benchmark against, repeatedly, as your prompts, models, agents, and workflows change. Each item is a canonical **input + expected output** pair. A test set has **no type**. One set can hold items sourced from production requests, workflow runs, agent tasks, manual entries, or an uploaded file, mixed freely. You choose *how* to run it (against a model, a workflow, or an agent) later, at run time, see [Test Run Execution](/test/test_sets/execution). ## Where to find Test Sets In the Portal, go to `Test > Test Sets`. The list shows every test set in your company; each item carries a badge showing where it came from. ## Create a test set (wizard) Click **New Test Set**. The wizard has three steps: * **1) Details** * Pick the **Application** the set belongs to. * Give it a **Name** (required) and optional **Description**. * Optionally mark it **Golden**: a small, high-signal set you trust as a quality bar (for example, to gate a fine-tuned model). * **2) Items**: add items now, or skip and add them later. Two modes: * **Pick items**: add from any of four sources (see below). You can queue items from *different* sources into the same set. * **Upload JSONL**: bulk-import a file where each line declares its own `type` (`request`, `workflow`, `agent`, or `manual`). * **3) Review**: confirm the details and the queued items, then create. A set is just a named bucket of items scoped to an Application, there is no "request set" vs. "workflow set" anymore. What an item *was* (its source) is provenance; how you *run* the set is a separate choice made per test run. ## Item sources When picking items, you'll see four tabs: Promote a real production **chat completion** you've observed. The request messages become the input; the assistant response becomes the expected output. Import a completed **workflow run**. Its input and final output are flattened into the canonical input/expected shape. Import a completed **agent task**. The task's opening message becomes the input; the agent's final response becomes the expected output. Author an item by hand, type the input messages and the expected assistant output yourself. Useful before you have production traffic. Imports are **idempotent**: importing the same request, workflow run, or agent task into the same set twice won't create a duplicate. ## Add items from Observe pages The fastest way to grow a set is to promote interesting traffic you've already found: * **From a Session or Request page**: use **Add to Test Set** to drop that request into an existing set (or create one inline). ## Correct the expected output An imported item's expected output is the *original* production response, which may itself have been wrong. To encode "what *should* have happened," edit an item's **expected output override**. This stores your corrected answer for scoring **without** discarding the original imported ground truth, so you can always revert. This is what makes a test set a true regression bar: the expected output reflects the correct behavior, not just whatever the model happened to say at capture time. ## Manage an existing test set Open any test set to: * Browse its **items** (each badged by source). * **Add** more items or **remove** items. * Edit **name**, **description**, and **Golden** flag. * Create and review **Test Runs** (see [Test Run Execution](/test/test_sets/execution)). * **Clone** the set (runs as a background job for large sets). ## Practical guidance * **Prefer real traffic.** Test sets are strongest when they mirror what your users actually do. * **Keep Golden sets small.** A focused, high-signal set is easier to trust and maintain. * **Keep sets evergreen.** Add newly discovered failure modes from Sessions and Requests; retire items that no longer reflect expected behavior. # Evaluation Criteria Source: https://docs.trymaitai.com/test/test_sets/criteria The reusable checks that score Test Runs: deterministic, LLM-judge, and agent-trajectory. **Evaluation Criteria** are the reusable checks that score your [Test Runs](/test/test_sets/execution). You define them once in your company catalog, then select which ones apply when you create a run. ## Where to manage criteria In the Portal, go to `Test > Test Criteria`. You'll see: * **System criteria**: built-in checks (like **Exact Match**). Read-only, but you can clone one to customize it. * **Custom criteria**: checks you create for your own quality bar. ## Criteria types ### Deterministic Computed directly from the output. No LLM involved, fully repeatable: * **Exact Match**: output equals expected output * **Contains**: output contains an expected substring * **Regex Match**: output matches a pattern * **BLEU / ROUGE / Cosine Similarity / Levenshtein**: text-similarity scores against the expected output ### LLM judge A model grades the output against your instructions. Three scoring modes: * **Boolean**: PASS / FAIL against a yes/no question * **Numeric**: an integer rubric over a configurable range (default **1–5**), optionally with named levels (e.g. `1 = Poor` … `5 = Perfect`) * **Tags**: classify each output into one of N categories LLM-judge criteria let you pick the evaluation model and, where relevant, map which fields to compare. ### Agent trajectory (agent runs) For [agent](/build/agents/create_agent) runs, criteria can score *how* the agent worked, not just its final answer: * **Tool checks**: required tools were called, only whitelisted tools were used, tools ran in the expected sequence * **Shape checks**: step count, sub-agent depth, failure rate within bounds * **Budget checks**: latency and cost ceilings respected * **Outcome checks**: an LLM judge or a custom **Python predicate** over the final result ## How criteria attach to a run You select criteria in the **Evaluations** step of the Test Run wizard: * **Request** and **Workflow** runs default to **Exact Match**; add more as needed. * **Agent** runs require at least one criterion (trajectory, outcome, or custom). Each item is then scored against every selected criterion. How those results roll up into a single per-item percent (and the run's match/pass rate) is covered in [Scoring & Criteria](/test/test_sets/rubrics). # Test Run Execution Source: https://docs.trymaitai.com/test/test_sets/execution Run a test set against a model, workflow, or agent, then score it with evaluation criteria. A **Test Run** executes every item in a [test set](/test/test_sets/creation) against a **consumer** you choose, then scores the results. Because the set is type-less, the *same* set can be run against a model one time and a workflow or agent the next, which makes runs directly comparable through the set as the shared axis. ## Create a Test Run 1. Open a test set. 2. Click **New Run**. The wizard walks you through: * **1) Consumer**: choose what runs each item: * **Request**: run each item through a single **model** with generation params you tune (temperature, top-p, max tokens, …). * **Workflow**: send each item through a **workflow's** full pipeline. * **Agent**: replay each item as an **agent** session and score the final response. * **2) Description**: a human-readable label so you remember why you ran it (e.g. "prompt v3", "gpt-5.2 temp 0.2", "post-finetune"). Required for Request and Workflow runs. * **3) Config**: consumer-specific: * *Request*: pick the model and generation params. * *Workflow*: pick the workflow. * *Agent*: pick the agent (and optionally a max-iterations cap). * **4) Evaluations**: pick the [evaluation criteria](/test/test_sets/rubrics) that score the run. * Request/Workflow runs default to **Exact Match**; add more criteria as needed. * Agent runs **require** at least one criterion (trajectory, outcome, or custom). * **5) Review**: confirm and launch. ## What happens after you launch A run moves through a short lifecycle: ``` CREATED → PREPARING INPUTS → RUNNING → SCORING → COMPLETED ``` * **Preparing inputs**: each canonical item is adapted into the consumer's native input (identity for a model; the workflow's own transform for a workflow; a task payload for an agent). You can preview this before launch. * **Running**: the consumer executes every item. * **Scoring**: this runs as a **separate pass** after execution. Until scoring completes, an item's score shows as a dash (`-`). You can also re-score a completed run after editing criteria. While a run is preparing or running, its detail page reflects live progress. ## Comparing runs Because every run shares the set as its axis, you can compare runs against the same set, including runs against *different* consumers (e.g. a model baseline vs. an agent). Open a run to see per-item results, and use the comparison views to see how a specific item shifted across runs. ## Deleting or resetting a run From a run you can **cancel** an in-progress run, **reset failed items** to retry just the failures, or **delete** a run you no longer need. Next: [Scoring & Criteria](/test/test_sets/rubrics) · [Interpreting Results](/test/test_sets/results). # Regression Monitoring Source: https://docs.trymaitai.com/test/test_sets/regression Use Test Runs and comparisons to detect regressions over time Regression monitoring is the practice of running Test Runs repeatedly as your configuration and models evolve, then using comparisons to catch quality regressions early. ## A practical regression workflow * **Build one “golden” set per Intent Group** * Keep it focused: a smaller, high-signal set is easier to maintain and reason about. * Mark it as **Golden** when appropriate. * **Create a Test Run for every meaningful change** * Examples: a new system prompt revision, a model swap, a temperature change, or after a fine-tune. * Use descriptions that make comparisons easy later (e.g. “prompt v4”, “temp 0.0”, “model X”, “post-finetune run”). * **Compare runs instead of trusting a single metric** * Compare runs against the same set to see item-by-item score shifts, including runs against *different* consumers (e.g. a model baseline vs. an agent). * Compare a single item across runs to track how that specific scenario changed over time. * **Focus attention where it matters** * Watch the run's **match rate** (or, for agent runs, **pass rate**) between runs. * Use the per-criterion breakdown to understand *which* [criterion](/test/test_sets/rubrics) degraded, not just that the score dropped, for numeric criteria, drill into the low buckets first. ## Common “gotchas” when monitoring regressions * **Error % matters**: a run with a higher match rate but a higher error rate can still be a regression. * **Coverage drift**: as your product evolves, add new real-world failure cases to the test set so regressions don’t hide in untested corners. ## Keeping Test Sets healthy * **Continuously add new failure modes** discovered in Sessions/Requests via “Add to Test Set”. * **Tag requests consistently** so you can quickly spot which categories are regressing. # Interpreting Results Source: https://docs.trymaitai.com/test/test_sets/results Understand Test Run match rates, criteria breakdowns, and comparisons in the Portal. Once a Test Run completes, its detail page gives you three layers: run-level summary metrics, per-item scores, and side-by-side comparisons. ## Run summary (macro view) For **Request** and **Workflow** runs, the summary tiles are: * **# Items**: total items in the run * **Completed %**: how many items finished executing * **Match Rate %**: matched / (matched + mismatched), over items that have expected output * **Error Rate %**: items that errored during execution A banner breaks the match rate down into matched / mismatched / without-expected / errored. For **Agent** runs, the summary instead shows a **Pass Rate**: the average pass rate across the run's [criteria](/test/test_sets/rubrics), since agent runs are scored by criteria rather than structural matching. Scoring runs as a separate pass after execution. Before scoring finishes, item scores show as a dash (`-`). You can re-score a completed run after changing its criteria. ## Requests table (per-item view) The items table is where you debug individual cases: * **Status** for each item execution * **Score**: a normalized `0–100%` pill; hover it for the **criteria breakdown** (each criterion's pass/fail, numeric value, or tag) * **Actions**: inspect the item, and compare the expected output against the run's actual output ## Criteria breakdown The per-criterion breakdown adapts to the criterion type: * **Boolean** → a pass rate, a failure count, and a pass/fail bar * **Numeric** → a score distribution (a table + bar over the score buckets, colored low→high) * **Tag** → a categorical distribution across the tag values Use this to see *which* criterion is driving a low score, then jump into those items. ## Comparing runs Because runs share the test set as their axis, you can compare them directly: * **Across runs**: see item-by-item score shifts between two or more runs, including runs against *different* consumers (e.g. a model baseline vs. an agent). * **A single item across runs**: track how one specific scenario changed over time. Next: [Regression Monitoring](/test/test_sets/regression). # Scoring & Criteria Source: https://docs.trymaitai.com/test/test_sets/rubrics How Test Runs are scored: structural match rate plus evaluation criteria. A Test Run is scored on two complementary layers: a **structural match rate** (does the output match the expected output?) and **evaluation criteria** (richer, configurable checks). Scoring runs as a separate pass after execution, see [Test Run Execution](/test/test_sets/execution). ## Layer 1: Structural match rate For **Request** (model) and **Workflow** runs, each item's actual output is compared against its expected output. The item is `matched`, `mismatched`, or `no expected` (when the item has no ground truth). The run's headline **Match Rate** is: ``` match_rate = matched / (matched + mismatched) ``` Items without an expected output are excluded from the denominator. **Agent** runs don't use structural matching, they're scored entirely by criteria (below). ## Layer 2: Evaluation criteria **Evaluation criteria** are the configurable checks you attach to a run. Each item is scored against every selected criterion. Criteria come from your company's catalog on the `Test > Test Criteria` page (see [Evaluation Criteria](/test/test_sets/criteria)); the built-in **Exact Match** is selected by default for Request and Workflow runs. Criteria fall into a few families, and they don't all score the same way: | Family | How it scores | | --------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------- | | **Deterministic** (Exact Match, Contains, Regex, BLEU, ROUGE, Cosine Similarity, Levenshtein) | Pass/fail or a computed similarity value | | **LLM judge, boolean** | PASS / FAIL | | **LLM judge, numeric** | An integer rubric over a configurable range (default **1–5**), optionally with named levels | | **LLM judge, tags** | Classifies each output into one of N categories | | **Agent trajectory** (agent runs only) | Deterministic checks on *how* the agent worked, e.g. tools used, tool sequence, step count, latency, cost | The 1–5 scale is just the default range for a **numeric** LLM-judge criterion, it is one option among several, not the scoring model for the whole run. Many criteria are simple pass/fail. ## The per-item score (0–100%) In the requests table, each item shows a single normalized **percent** score, aggregated across its criteria: * **Boolean / pass-fail** criteria contribute `1` (pass) or `0` (fail). * **Numeric** criteria contribute `(score − min) / (max − min)` using that criterion's range. * **Tag** criteria are excluded from the score. ``` normalized_score = round( (sum of per-criterion contributions) / (criteria counted) × 100 ) ``` Hover an item's score to see the **criteria breakdown**: each criterion's individual result (pass/fail pill, numeric bar, or tag). ## Agent trajectory scoring Agent runs can be scored on the *trajectory*, not just the final answer. Trajectory criteria are deterministic checks such as: required tools were called, tools were called in the expected order, step count / sub-agent depth stayed within bounds, and latency or cost ceilings were respected. Agent runs can also use LLM-judge outcome criteria and custom Python-predicate criteria. ## Using scores when debugging * **Start with the distribution.** For numeric criteria, the breakdown shows a score distribution, drill into the low buckets first. * **Read the per-criterion breakdown.** It tells you *which* criterion degraded, not just that the score dropped. * **Compare responses.** For any item, compare the expected output against the run's actual output side by side (see [Interpreting Results](/test/test_sets/results)).