Evals

runEvalSuite (2.3) in @deuz-sdk/core/testing runs versioned eval tasks and measures every attempt. It builds on runEval and runGradedEval: a task's acceptance criteria are graded the way runGradedEval grades a case. On top of that come a start snapshot, repeats, a timeout and the metrics an agent harness needs: success, false completion, cost, duration, human rescues, repeated effects and recovery.

2.3 ships a baseline, not a benchmark. The repository's test suite runs 22 offline tasks and 33 fault-injection scenarios on every npm test. Eleven real-model tasks are defined but were not run for 2.3, so there are no real-model results. This page ranks nothing and makes no claim about other SDKs.

2.4 adds 56 acceptance scenarios for its ten areas (versioned memory, context, transcript search, skill proposals, the goal supervisor, the durable scheduler, Docker, chat and stream durability with React, acceptance evidence, retention) under packages/core/test/scenarios-2-4/. 16 of them are fault scenarios: each runs three times on fresh stores and is never retried. They run on every npm test; the legs that need Postgres, Redis or Docker run when those test services are configured. They are offline too, so they make no real-model claim either.

A task

A task manifest has a stable id and a version you bump whenever its start state, acceptance or grading changes, so two versions are never compared as one task. run performs one attempt and returns its outcome. A native AgentResult is an outcome as it stands, so a task can return runAgent(...) directly.

import { runAgent } from '@deuz-sdk/core/agent';
import { createMockModel, runEvalSuite, type EvalTaskManifest } from '@deuz-sdk/core/testing';

const countOrders: EvalTaskManifest = {
  id: 'orders/count',
  version: 1,
  start: { messages: [{ role: 'user', content: 'How many orders does ada@example.com have?' }] },
  tools: { allowed: ['list_orders'] },
  acceptance: [
    { name: 'completes', check: (outcome) => outcome.status === 'completed' },
    { name: 'answers 3', check: (outcome) => (outcome.text ?? '').includes('3 orders') },
  ],
  run: (context) =>
    runAgent({
      model: createMockModel({
        responses: [
          { toolCalls: [{ toolName: 'list_orders', args: { email: 'ada@example.com' } }] },
          { text: 'ada@example.com has 3 orders.' },
        ],
      }),
      messages: context.messages,
      tools: {
        list_orders: {
          description: 'List the orders of a customer.',
          parameters: {
            type: 'object',
            properties: { email: { type: 'string' } },
            required: ['email'],
          },
          execute: () => ['o-1', 'o-2', 'o-3'],
        },
      },
      signal: context.signal,
      // The observer feeds the allowed-tools check.
      deps: { observer: context.observer },
    }),
};

const report = await runEvalSuite([countOrders], { repeats: 3 });
console.log(report.aggregate.offline?.successRate); // 1

What an attempt gets

run receives a context for one attempt:

• messages, files and store: copies of the start snapshot. Changes stay in this attempt, and the scope check compares files with the start.
• observer: an in-memory observer. Pass it as deps.observer; its tool.started events are the tool calls the allowed-tools check reads.
• signal: aborted when the attempt times out.
• effect(key): record one execution of an external effect by its real-world identity, such as charge:order-7. A key recorded twice is a repeated effect.
• approve(requests, approved?): a human answers approval requests. It returns the verdicts for approvalResponses, with signed tokens echoed back.
• rescue(reason): a human stepped in where the task needed no one.
• interrupt(boundary): the injected fault fired.
• artifact(name, value): keep a JSON value in the report.

A required approval is not a rescue. Approving a tool the manifest lists in approvals counts as a required approval; approving any other tool, or calling rescue, counts as a human rescue:

import { createInMemoryAgentRunStore, resumeAgent, runAgent } from '@deuz-sdk/core/agent';
import { createMockModel, type EvalTaskManifest } from '@deuz-sdk/core/testing';

const deploy: EvalTaskManifest = {
  id: 'deploy/approved',
  version: 1,
  approvals: ['deploy'],
  acceptance: [{ name: 'completes', check: (outcome) => outcome.status === 'completed' }],
  async run(context) {
    const options = {
      model: createMockModel({
        responses: [{ toolCalls: [{ toolName: 'deploy', args: {} }] }, { text: 'Live.' }],
      }),
      prompt: 'Ship v2.3.0.',
      session: { store: createInMemoryAgentRunStore(), runId: 'deploy', scope: 'eval' },
      tools: {
        deploy: {
          parameters: { type: 'object' as const, properties: {} },
          needsApproval: true,
          execute: () => {
            context.effect('deploy:v2.3.0');
            return 'deployed';
          },
        },
      },
      deps: { observer: context.observer },
    };
    const first = await runAgent(options);
    if (first.status !== 'suspended') return first;
    return resumeAgent({ ...options, approvalResponses: context.approve(first.pendingApprovals) });
  },
};

Metrics

Each attempt in the report carries:

recovery is 'not-needed' when nothing was interrupted. After an interruption it is 'recovered' (completed and accepted without a rescue), 'rescued' (completed and accepted after a rescue), 'contained' (accepted, but the documented outcome is a stop or a failure: the work did not finish) or 'failed' (not accepted, or an effect ran twice). An attempt that repeated an effect is never recovered, rescued or contained. The other checks below, a recovery path other than fault.recovery included, fail success but leave recovery as acceptance decides it.

Beyond its acceptance, an attempt fails when it:

• recorded one effect key twice;
• called a tool outside tools.allowed;
• changed a file outside tools.scope;
• went over budget;
• did not record a declared artifact;
• declared a fault that never fired, or recovered another way than fault.recovery says.

A run that throws is an 'error' attempt and a run that outlives its timeout a 'timeout' attempt; neither stops the suite. A malformed suite (a duplicate id, a version that is not a positive integer, a task without acceptance or without a runner) throws a TypeError before any task runs.

The report

runEvalSuite returns the report and writes nothing unless you ask. Pass report as a file path (written as JSON with node:fs, its directory created) or as a callback:

import { runEvalSuite, type EvalTaskManifest } from '@deuz-sdk/core/testing';

declare const tasks: EvalTaskManifest[];

await runEvalSuite(tasks, {
  name: 'nightly',
  timeoutMs: 60_000,
  report: (report) => {
    for (const task of report.tasks)
      console.log(task.id, task.version, `${task.metrics.successes}/${task.metrics.attempts}`);
  },
});

The report has format: 'deuz-eval-suite' and version: 1, the tasks in suite order with their attempts and metrics, and an aggregate per tier. Offline and real-model numbers are separate experiments, so they are never added together. A path needs a runtime with node:fs (Node, Bun or Deno); elsewhere, use a callback.

Real-model tasks

A task with tier: 'real-model' is meant for a real provider. List its keys in env and skip it when one is missing; a skipped task appears in the report with its reason and no attempts:

import { runEvalSuite, type EvalTaskManifest } from '@deuz-sdk/core/testing';

declare const tasks: EvalTaskManifest[];

const report = await runEvalSuite(tasks, {
  skip: (task) => {
    const missing = (task.env ?? []).filter((name) => !process.env[name]);
    return missing.length > 0 ? `no ${missing.join(', ')}` : undefined;
  },
});
console.log(report.aggregate['real-model']?.skipped);

The repository defines eleven real-model tasks, each the twin of an offline task: the bug fix, the migration, the sourced research, the invoice extraction, the approved deploy, the run with memory and a chat, the fail-over (to a real fallback, behind a primary that stays offline), the approval through the UI bridge, the goal's run, the lookup on an MCP server, and the batch. Each runs on a model from the SDK's registry and price table, under a USD budget, with three attempts, and reuses its offline twin's start state, tools and acceptance. The first ten run only through npm run test:live, never in npm test or CI. The batch is defined apart, with its skip reason: a provider can take up to 24 hours to end a batch, so no runner submits it. None of them is live-verified: no run against a provider was made for 2.3.

The 2.3 baseline

npm test runs the offline tasks under packages/core/test/eval/. Their models are scripted, so they measure the SDK rather than a model: that the loop, receipts, approvals, resume and accounting carry a known plan to an accepted result, exactly once. They cover a tool loop (parallel and chained calls), structured output and its repair, a bug fix judged by hidden tests, a config migration, sourced research against a mock search index, an approval granted and one denied, a nested agent and a sub-agent approval, an interrupted and resumed run, an MCP hot refresh against the in-process MCP server, a client-side tool, a verifier retry, and a runaway loop that must stop instead of claiming success. Six more cover what 2.3 added: a native run that continues a stored chat and acts on a preference it recalls from memory, a fail-over from a primary that answers 529, an approval given through the UI bridge, a goal woken by its schedule and completed on a tool's evidence, a batch submitted and collected on an in-memory stand-in of a batch API, and a tool that createMcpServer serves to an in-process client. All 22 succeed, with no false completion, no repeated effect and no human rescue; the four approvals are required ones, and the two interrupted tasks, the resumed run and the fail-over, recover by themselves.

The fault-injection suite (packages/core/test/eval/faults/) injects one fault per scenario through the same hooks the regression tests use: a store that fails one write, a lease provider whose renewal fails, a budget store that fails a reserve or a settle, a model stream that breaks off, a provider that answers 529, a chat store that is down, a client that hangs up, and a nested run that asks for memory outside its scope. Every scenario asserts that no effect ran twice, that no billed call is missing from the accounting, and the documented recovery:

Contained is not a defect in itself: it is the documented safe outcome. A cancel, an elapsed deadline, a refusal, a turn cut off by max_tokens, a nested run that asks for memory outside the scope of the run it serves, and a chat repair that would overwrite a newer chat end the work by design. The other contained cases have an opt-in way on in 2.3, each shown by the recovered scenario under it. A mid-stream transport failure and an admission store failure end the native run as failed, and a plain resume returns that result; a resume with retryFailed goes on from the last checkpoint. After an unknown outcome the run stops for reconciliation, and in the contained scenario a person checks the payment system; when the application checks it and records what it found with reconciledToolResults, the run recovers without a person and without running the charge again. A crash between a hot refresh and its record stays contained under the default onCatalogChange: 'stop'; a resume with 'accept' records the servers' new catalog and recovers.

The recovered fail-over scenarios keep what a fail-over leaves behind by design: the failed attempt stays held at its estimate, in the ledger and, with persistent admission, as an open hold in the budget store, so a fallback never spends past the shared budget. The budget scenario's crash comes after its calls settled. A process that dies after a call's hold was admitted and before its settlement was saved leaves that hold open where the settlement did not land (a reserved entry in the last persisted ledger snapshot, an open hold in the budget store, or both), to settle or release as When a settlement fails and Open holds describe.

Set DEUZ_EVAL_REPORT=<path> or DEUZ_FAULT_REPORT=<path> when running the core tests to write either suite's JSON report.

See also

• Autonomy: Testing: createMockModel, runEval and the golden-replay helpers.
• Native agents: When the store fails: the receipt boundaries the fault suite injects at.
• Budgets: reservations, settlement and persistent admission.
• Observability: the events behind context.observer.