Evaluation

Measure agent quality with automated evaluations — tool call accuracy, response relevance, output size, and more.

Overview

The docker agent eval command runs your agent against a set of recorded sessions and scores the results. Each eval session captures a user question, the expected tool calls, and criteria the response must satisfy. Docker Agent replays the question, compares the agent's behavior to expectations, and produces a report.

Quick Start

# Run evaluations for an agent
$ docker agent eval agent.yaml

# Specify a custom evals directory
$ docker agent eval agent.yaml ./my-evals

# Run with 8 concurrent evaluations
$ docker agent eval agent.yaml -c 8

# Only run evals matching a pattern
$ docker agent eval agent.yaml --only "auth*"

# Repeat each eval 5 times to compute a baseline
$ docker agent eval agent.yaml --repeat 5

# Repeat a specific eval 5 times
$ docker agent eval agent.yaml --only "auth*" --repeat 5

# Use a Docker-compatible runtime such as Podman
$ docker agent eval agent.yaml --container-runtime podman

Eval Directory Structure

By default, Docker Agent looks for eval sessions in an evals/ directory next to your agent config:

my-agent/
├── agent.yaml
└── evals/
    ├── 41b179a2-....json          # Eval session 1
    ├── 5d83e247-....json          # Eval session 2
    └── results/                   # Output (auto-created)
        ├── adjective-noun-1234.json
        ├── adjective-noun-1234.log
        ├── adjective-noun-1234.db
        └── adjective-noun-1234-sessions.json

Eval Session Format

Each eval file is a JSON session that captures a complete conversation. The key fields for evaluation are the user message, the expected tool calls (recorded from a real session), and optional eval criteria:

{
  "id": "41b179a2-ed19-4ae2-a45d-95775aaa90f7",
  "title": "Counting Files in Local Folder",
  "messages": [
    {
      "message": {
        "message": {
          "role": "user",
          "content": "How many files in the local folder?"
        }
      }
    },
    {
      "message": {
        "agent_name": "root",
        "message": {
          "role": "assistant",
          "tool_calls": [
            {
              "id": "call_abc123",
              "type": "function",
              "function": {
                "name": "list_directory",
                "arguments": "{\"path\":\"./\"}"
              }
            }
          ]
        }
      }
    },
    {
      "message": {
        "agent_name": "root",
        "message": {
          "role": "assistant",
          "content": "There are 2 files in the local folder..."
        }
      }
    }
  ],
  "evals": {
    "relevance": [
      "The response mentions exactly 2 files",
      "The response lists README.md and agent.yaml"
    ],
    "size": "S",
    "working_dir": "my-project",
    "setup": "echo 'hello' > test.txt"
  }
}

Eval Criteria

The evals object inside each session controls what gets scored:

Scoring Metrics

Docker Agent evaluates agents across three dimensions:

Serve-safety verification and rollback

When changing an agent served over MCP HTTP, chat, or A2A, add an evaluation that attempts an approval-requiring tool call and verifies the resolved safety policy and authentication behavior. Run the evaluation with the same explicit --safety setting used in deployment. If a rollout must be reversed, stop the affected listener, restore the prior agent configuration and explicit safety flag, then restart only after confirming non-loopback listeners still require authentication. Do not restore an unauthenticated network listener as a rollback shortcut.

Creating Eval Sessions

The easiest way to create eval sessions is from real conversations:

1. Run your agent interactively: docker agent run agent.yaml
2. Have a conversation that tests the behavior you care about
3. Use the /eval slash command in the TUI to save the session as an eval file
4. Edit the generated JSON to add evals criteria (relevance, size, etc.)

CLI Flags

$ docker agent eval <agent-file>|<registry-ref> [<eval-dir>|./evals]

Regression gate

--baseline compares the run against a previous one and exits non-zero when quality regressed, so an eval suite can gate CI:

$ docker agent eval ./agent.yaml --baseline results/2026-08-01-run.json

The baseline is the run JSON written by a previous invocation — <output>/<run-name>.json — so there is no separate artifact to produce.

Four rules decide the verdict, and they are worth knowing before wiring this into CI:

• The tolerance governs aggregate rates only. An LLM judge does not return the same score twice, so without a tolerance the gate flaps. --regression-tolerance 0.05 lets an aggregate rate fall five points before it counts.
• An evaluation that passed and now fails always gates, regardless of the tolerance. That transition is the signal the gate exists to catch, so it is never absorbed.
• Cost is reported but never gates. A provider price change is not a quality regression.
• An added failing evaluation gates via the aggregate rate, even though no existing evaluation regressed. A suite that got worse should say so — but it means committing a known-failing eval needs a tolerance bump or a fix.

A baseline that carries no evaluations, or a run that produced none (an --only pattern that matched nothing), is rejected rather than reported as passing: a gate that cannot fail is worse than no gate.

Provider Credentials

Eval containers are isolated from your host environment. Dedicated model provider API keys (for example ANTHROPIC_API_KEY or OPENAI_API_KEY) are forwarded into eval containers automatically, so most provider setups work without extra flags.

docker agent eval agent.yaml ./evals -e GITHUB_TOKEN

docker agent eval agent.yaml ./evals \
  --env-from-file /path/to/secrets.env \
  -e GITHUB_TOKEN

Note that the LLM judge runs on the host, not inside the eval container. If the token is not forwarded, judge validation can succeed while every evaluated agent run fails to authenticate.

Custom Base Images

When --base-image is set, the eval harness builds a derived image on top of your base image at evaluation time. Two things happen automatically:

1. The docker-agent binary is injected — it is copied from docker/docker-agent:edge into the derived image at build time, so you don't need to include it in your base image.
2. The entrypoint is overridden — Docker Agent replaces your base image's entrypoint with its own /run.sh wrapper.

Your base image therefore only needs to provide the runtime environment: language runtimes, installed dependencies, test fixtures, the appropriate working directory, and so on. Any ENTRYPOINT or CMD defined in your base image is ignored.

Output

After a run completes, Docker Agent produces:

• Console summary — Pass/fail status per eval with metric breakdowns
• JSON results — Full structured results for programmatic analysis
• SQLite database — Complete sessions for detailed investigation and debugging
• Sessions JSON — Exported session data for analysis
• Log file — Debug-level log of the entire evaluation run

$ docker agent eval demo.yaml ./evals

  ✓ Counting Files in Local Folder
    ✓ tool calls  ✓ relevance 2/2
  ✓ Checking the Content of README.md File
    ✓ tool calls  ✓ relevance 1/1

✅     Tool Calls: 100.0% avg F1 (2 evals)
✅      Relevance: 3/3 passed (100.0%)

Total Cost: $0.012345
Total Time: 12s

Sessions DB: ./evals/results/happy-panda-1234.db
Sessions JSON: ./evals/results/happy-panda-1234-sessions.json
Log: ./evals/results/happy-panda-1234.log

Example

Here's a minimal evaluation setup:

# agent.yaml
agents:
  root:
    model: openai/gpt-4o
    description: Test agent
    instruction: You know how to read/write and list files.
    toolsets:
      - type: filesystem

# Create evals from interactive sessions
$ docker agent run agent.yaml
# ... have conversations, then use /eval to save them

# Run the evaluations
$ docker agent eval agent.yaml ./evals