> ## Documentation Index
> Fetch the complete documentation index at: https://docs.trymaitai.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Evaluate

> Detect faults in LLM output

## Create evaluations

Evaluations are created when you call [Chat](/sdk/chat) with evaluations enabled (via Portal configuration or SDK parameters). The model output is what gets evaluated, with the request used as context.

If a `callback` is provided, the evaluation is passed to that function asynchronously.

If no `callback` is provided, and `stream` is `true`, then the evaluation is available on the last chunk.

If no `callback` is provided and `stream` is `false`, then the evaluation can be found on the completion response.

## Example (callback)

<RequestExample>
  ```python Python (async) theme={null}
  import maitai

  async def on_eval(eval_response):
      # handle evaluate response (EvaluateResponse)
      print(eval_response)

  messages = [
      {"role": "system", "content": "You are a helpful assistant"},
      {"role": "user", "content": "Generate numbers 1-10"},
  ]

  client = maitai.MaitaiAsync()

  response = await client.chat.completions.create(
      messages=messages,
      model="gpt-4o",
      session_id="YOUR_SESSION_ID",
      intent="NUMBER_GENERATOR",
      application="demo_app",
      callback=on_eval,
  )
  ```

  ```python Python (sync) theme={null}
  import maitai

  def on_eval(eval_response):
      # handle evaluate response (EvaluateResponse)
      print(eval_response)

  messages = [
      {"role": "system", "content": "You are a helpful assistant"},
      {"role": "user", "content": "Generate numbers 1-10"},
  ]

  client = maitai.Maitai()

  response = client.chat.completions.create(
      messages=messages,
      model="gpt-4o",
      session_id="YOUR_SESSION_ID",
      intent="NUMBER_GENERATOR",
      application="demo_app",
      callback=on_eval,
  )
  ```

  ```javascript Node theme={null}
  import Maitai from "maitai";

  const maitai = new Maitai();

  const onEval = (evalResponse) => {
    // handle evaluate response (EvaluateResponse)
    console.log(evalResponse);
  };

  const messages = [
    { role: "system", content: "You are a helpful assistant" },
    { role: "user", content: "Generate numbers 1-10" },
  ];

  const response = await maitai.chat.completions.create({
    messages: messages,
    model: "gpt-4o",
    session_id: "YOUR_SESSION_ID",
    intent: "NUMBER_GENERATOR",
    application: "demo_app",
    callback: onEval,
  });
  ```
</RequestExample>

<ResponseExample>
  ```json theme={null}
  {
    "application_id": 16,
    "session_id": "xxx",
    "evaluation_results": [
      {
        "status": "FAULT",
        "description": "Test with random number",
        "confidence": 100.0,
        "correction": "Test with random number 42.",
        "sentinel_id": 41,
        "eval_time": 489,
        "date_created": 1716682520088,
        "usage": {
          "prompt_tokens": 380,
          "completion_tokens": 54,
          "total_tokens": 434
        }
      }
    ],
    "evaluation_request_id": "b84f0d55-7c7e-4af4-8d84-106385682250"
  }
  ```
</ResponseExample>

<ResponseField name="application_id" type="integer">
  The Maitai identifier for the application.
</ResponseField>

<ResponseField name="session_id" type="string">
  A unique identifier for the session, passed in from the [Chat](/sdk/chat) endpoint.
</ResponseField>

<ResponseField name="request_id" type="string">
  The identifier of the evaluated chat completion request.
</ResponseField>

<ResponseField name="evaluation_results" type="array">
  A list of individual Sentinel results.

  <Expandable title="properties">
    <ResponseField name="id" type="integer">
      A unique identifier for the evaluation result.
    </ResponseField>

    <ResponseField name="status" type="string">
      The status of the evaluation. `FAULT` means a fault was detected. `PASS` means the LLM output passed testing. `NA` means an evaluation wasn't performed.
    </ResponseField>

    <ResponseField name="description" type="string">
      A detailed description of the evaluation outcome.
    </ResponseField>

    <ResponseField name="confidence" type="float">
      The confidence level of the evaluation result.
    </ResponseField>

    <ResponseField name="correction" type="string">
      Optional suggested correction text when a `FAULT` is detected.
    </ResponseField>

    <ResponseField name="meta" type="object">
      Additional metadata associated with the evaluation result.
    </ResponseField>

    <ResponseField name="sentinel_id" type="integer">
      An identifier for the Sentinel associated with this result.
    </ResponseField>

    <ResponseField name="eval_time" type="integer">
      The evaluation time in milliseconds.
    </ResponseField>

    <ResponseField name="date_created" type="integer">
      The Unix timestamp marking the creation of the evaluation result.
    </ResponseField>

    <ResponseField name="usage" type="object">
      Usage statistics for the Sentinel

      <Expandable title="properties">
        <ResponseField name="completion_tokens" type="integer">
          Number of tokens in the generated completion.
        </ResponseField>

        <ResponseField name="prompt_tokens" type="integer">
          Number of tokens in the prompt.
        </ResponseField>

        <ResponseField name="total_tokens" type="integer">
          Total number of tokens used in the request (prompt + completion).
        </ResponseField>
      </Expandable>
    </ResponseField>
  </Expandable>
</ResponseField>

<ResponseField name="evaluation_request_id" type="string">
  The unique identifier for the evaluation request.
</ResponseField>