Evaluation

Measure agent quality with automated evaluations — tool call accuracy, response relevance, output size, and more.

Overview

The docker agent eval command runs your agent against a set of recorded sessions and scores the results. Each eval session captures a user question, the expected tool calls, and criteria the response must satisfy. Docker Agent replays the question, compares the agent's behavior to expectations, and produces a report.

Quick Start

# Run evaluations for an agent
$ docker agent eval agent.yaml

# Specify a custom evals directory
$ docker agent eval agent.yaml ./my-evals

# Run with 8 concurrent evaluations
$ docker agent eval agent.yaml -c 8

# Only run evals matching a pattern
$ docker agent eval agent.yaml --only "auth*"

# Repeat each eval 5 times to compute a baseline
$ docker agent eval agent.yaml --repeat 5

# Repeat a specific eval 5 times
$ docker agent eval agent.yaml --only "auth*" --repeat 5

# Use a Docker-compatible runtime such as Podman
$ docker agent eval agent.yaml --container-runtime podman

Eval Directory Structure

By default, Docker Agent looks for eval sessions in an evals/ directory next to your agent config:

my-agent/
├── agent.yaml
└── evals/
    ├── 41b179a2-....json          # Eval session 1
    ├── 5d83e247-....json          # Eval session 2
    └── results/                   # Output (auto-created)
        ├── adjective-noun-1234.json
        ├── adjective-noun-1234.log
        ├── adjective-noun-1234.db
        └── adjective-noun-1234-sessions.json

Eval Session Format

Each eval file is a JSON session that captures a complete conversation. The key fields for evaluation are the user message, the expected tool calls (recorded from a real session), and optional eval criteria:

{
  "id": "41b179a2-ed19-4ae2-a45d-95775aaa90f7",
  "title": "Counting Files in Local Folder",
  "messages": [
    {
      "message": {
        "message": {
          "role": "user",
          "content": "How many files in the local folder?"
        }
      }
    },
    {
      "message": {
        "agent_name": "root",
        "message": {
          "role": "assistant",
          "tool_calls": [
            {
              "id": "call_abc123",
              "type": "function",
              "function": {
                "name": "list_directory",
                "arguments": "{\"path\":\"./\"}"
              }
            }
          ]
        }
      }
    },
    {
      "message": {
        "agent_name": "root",
        "message": {
          "role": "assistant",
          "content": "There are 2 files in the local folder..."
        }
      }
    }
  ],
  "evals": {
    "relevance": [
      "The response mentions exactly 2 files",
      "The response lists README.md and agent.yaml"
    ],
    "size": "S",
    "working_dir": "my-project",
    "setup": "echo 'hello' > test.txt"
  }
}

Eval Criteria

The evals object inside each session controls what gets scored:

Assertions

Each entry in assertions is a code-based check evaluated deterministically against the agent's output, without an LLM judge:

"assertions": [
  { "name": "mentions file count", "type": "contains", "value": "2 files" },
  { "name": "no error message", "type": "not_contains", "value": "error" },
  { "name": "used list_directory", "type": "tool_called", "value": "list_directory" },
  { "name": "under budget", "type": "cost_threshold", "value": "0.05" }
]

Each assertion has a name (shown in results), a type, and a value checked against the agent's response, cost, or tool calls:

Scoring Metrics

Docker Agent evaluates agents across four dimensions:

Repeat Metrics (pass@k / pass^k)

Running with --repeat <k> (k > 1) repeats each eval k times and, once the run completes, prints two additional consistency metrics alongside the regular summary:

• pass@k — the fraction of unique evals that passed on at least one of the k repetitions. Measures whether the agent can produce a correct answer at all.
• pass^k — the fraction of unique evals that passed on every one of the k repetitions. Measures determinism / reliability.

  Repeat metrics (k=5, 3 unique evals):
   pass@5: 100.0% (passed at least once)
   pass^5: 66.7% (passed every time)

These metrics are also included in the JSON results (repeat_metrics) and, when comparing against a --baseline, are reported as informational deltas (see Regression gate).

Serve-safety verification and rollback

When changing an agent served over MCP HTTP, chat, or A2A, add an evaluation that attempts an approval-requiring tool call and verifies the resolved safety policy and authentication behavior. Run the evaluation with the same explicit --safety setting used in deployment. If a rollout must be reversed, stop the affected listener, restore the prior agent configuration and explicit safety flag, then restart only after confirming non-loopback listeners still require authentication. Do not restore an unauthenticated network listener as a rollback shortcut.

Creating Eval Sessions

The easiest way to create eval sessions is from real conversations:

1. Run your agent interactively: docker agent run agent.yaml
2. Have a conversation that tests the behavior you care about
3. Use the /eval slash command in the TUI to save the session as an eval file
4. Edit the generated JSON to add evals criteria (relevance, size, etc.)

CLI Flags

$ docker agent eval <agent-file>|<registry-ref> [<eval-dir>|./evals]

Regression gate

--baseline compares the run against a previous one and exits non-zero when quality regressed, so an eval suite can gate CI:

$ docker agent eval ./agent.yaml --baseline results/2026-08-01-run.json

The baseline is the run JSON written by a previous invocation — <output>/<run-name>.json — so there is no separate artifact to produce.

The comparison covers every aggregate quality rate that both runs have data for: size pass rate, tool F1 mean, relevance rate, and — since docker/docker-agent#4103 — assertion rate. When both runs used --repeat, pass@k and pass^k are also reported as deltas, but purely informationally: they never gate, since they derive from the same per-eval pass/fail that the other metrics already gate on.

Four rules decide the verdict, and they are worth knowing before wiring this into CI:

• The tolerance governs aggregate rates only. An LLM judge does not return the same score twice, so without a tolerance the gate flaps. --regression-tolerance 0.05 lets an aggregate rate fall five points before it counts.
• An evaluation that passed and now fails always gates, regardless of the tolerance. That transition is the signal the gate exists to catch, so it is never absorbed.
• Cost is reported but never gates. A provider price change is not a quality regression.
• An added failing evaluation gates via the aggregate rate, even though no existing evaluation regressed. A suite that got worse should say so — but it means committing a known-failing eval needs a tolerance bump or a fix.

A baseline that carries no evaluations, or a run that produced none (an --only pattern that matched nothing), is rejected rather than reported as passing: a gate that cannot fail is worse than no gate.

Provider Credentials

Eval containers are isolated from your host environment. Dedicated model provider API keys (for example ANTHROPIC_API_KEY or OPENAI_API_KEY) are forwarded into eval containers automatically, so most provider setups work without extra flags.

docker agent eval agent.yaml ./evals -e GITHUB_TOKEN

docker agent eval agent.yaml ./evals \
  --env-from-file /path/to/secrets.env \
  -e GITHUB_TOKEN

Note that the LLM judge runs on the host, not inside the eval container. If the token is not forwarded, judge validation can succeed while every evaluated agent run fails to authenticate.

Custom Base Images

When --base-image is set, the eval harness builds a derived image on top of your base image at evaluation time. Two things happen automatically:

1. The docker-agent binary is injected — by default it is copied from docker/docker-agent:<version>, pinned to this CLI's own release version, so eval results are reproducible against a known build rather than a moving target. A dev build (compiled from main, without a release version) falls back to docker/docker-agent:edge. This injection applies to every eval run, not just those using --base-image. Use --agent-image <ref> to inject a specific image instead — for example to pin CI to an older release, or to test against docker/docker-agent:edge deliberately. Pass --agent-image none to skip injection entirely and trust whatever /docker-agent binary is already present in your base image.
2. The entrypoint is overridden — Docker Agent replaces your base image's entrypoint with its own /run.sh wrapper.

Your base image therefore only needs to provide the runtime environment: language runtimes, installed dependencies, test fixtures, the appropriate working directory, and so on. Any ENTRYPOINT or CMD defined in your base image is ignored.

Output

After a run completes, Docker Agent produces:

• Console summary — Pass/fail status per eval with metric breakdowns
• JSON results — Full structured results for programmatic analysis
• SQLite database — Complete sessions for detailed investigation and debugging
• Sessions JSON — Exported session data for analysis
• Log file — Debug-level log of the entire evaluation run

$ docker agent eval demo.yaml ./evals

  ✓ Counting Files in Local Folder
    ✓ tool calls  ✓ relevance 2/2
  ✓ Checking the Content of README.md File
    ✓ tool calls  ✓ relevance 1/1

✅     Tool Calls: 100.0% avg F1 (2 evals)
✅      Relevance: 3/3 passed (100.0%)

Total Cost: $0.012345
Total Time: 12s

Sessions DB: ./evals/results/happy-panda-1234.db
Sessions JSON: ./evals/results/happy-panda-1234-sessions.json
Log: ./evals/results/happy-panda-1234.log

Example

Here's a minimal evaluation setup:

# agent.yaml
agents:
  root:
    model: openai/gpt-4o
    description: Test agent
    instruction: You know how to read/write and list files.
    toolsets:
      - type: filesystem

# Create evals from interactive sessions
$ docker agent run agent.yaml
# ... have conversations, then use /eval to save them

# Run the evaluations
$ docker agent eval agent.yaml ./evals