An eval harness for a user-facing agent

Career Copilot is a Telegram bot that builds a private career profile, saves job postings, analyzes fit, and reports back to the user.

It is an AI agent, so normal unit tests miss important behavior: replies vary, tools touch external systems, private data can leak, and failures span multiple turns.

TL;DR

• Contract lane: Runs the real runtime offline with scripted model responses, ledgers, deterministic assertions, and canary scans.
• Quality lane: Replays selected scenarios with a real model and an independent judge scoring a redacted transcript.
• Fail-closed rule: A quality score can explain a result, but it can never override a contract failure.

If you are building an eval harness for your own agent, the two-lane boundary is the main idea to take.

Why normal tests are not enough

A normal test maps fixed inputs to expected outputs. An agent needs tests for behavior across turns, tool calls, state changes, privacy boundaries, and failure recovery.

The rule that keeps the lanes honest: no quality score can rescue a deterministic failure.

The design in one picture

Figure: Harness architecture: the real runtime with everything outside it faked, and the quality lane on the side.

Two things matter here. The runtime is the real production code: routing, onboarding state machine, tools, storage, failure handling. Nothing is mocked at the application level. What we fake is everything outside it: the model, the network, the database files, the clock. That is the trick that makes the deterministic lane possible while still testing the real thing.

Layer 1: the contract lane, deterministic and offline

The contract lane answers: did anything break? Each YAML scenario describes a user journey, its fixtures, and the behavior that must or must not occur.

The runner:

1. Creates a fresh sandbox with a fixed clock, IDs, and database.
2. Replays the real runtime against scripted model responses.
3. Records model calls, tool calls, replies, state, and database events, then checks deterministic assertions.
4. Scans every sink for forbidden canaries and reports the scenario as passed, failed, or incomplete.

Figure: The contract lane: deterministic, offline, fail closed.

npm run eval:test runs the full corpus offline with no credentials, network, or model spend. The fake model is the only mocked layer; the application runtime remains real.

The fake model must still support the runtime’s tool calls and structured responses. Inert fixture defaults matter because an accidental default reply can change the scenario path.

Layer 2: the quality lane, real model, human judgment

The contract lane catches regressions; the quality lane evaluates whether the user-facing behavior is clear, grounded, truthful, and recoverable.

Figure: The quality lane: gate, real replay, cross-family judge, and every status branch.

Three steps per scenario:

1. The contract gate. The same deterministic replay from Layer 1 runs first. If it fails, quality never runs. This is the “no quality score rescues a deterministic failure” rule in action.
2. The quality replay. The same scenario runs again, but with the real production model. Its assertions are still enforced, and every call must report usage and cost.
3. The judge. A second model, from a different model family than the one being tested, scores the conversation against ten rubrics on a 1 to 5 scale: task completion, tool selection, grounding, privacy safety, truthfulness, memory use, onboarding discipline, job analysis quality, conversational quality, recovery quality.

This quality lane adds a new boundary for the project: the external judge sees only a redacted transcript, bounded evidence excerpts, and rubric definitions—not the raw conversation. It must cite evidence for every score; a score other than 3 needs a rationale, and a critical failure such as a fabricated success forces score 1 and fails the scenario.

Why a different model family? Because a model grading its own family is a cheerleader. The scorer must be independent of the scored.

Canaries: proving secrets do not leak

A canary is a synthetic secret planted in test data, such as canary@example.test or PAGE_INJECTION_CANARY. If a canary shows up anywhere it should not, we know a real secret would have leaked there too.

Every canary declares where it is allowed to appear. A resume canary is allowed nowhere. An injection canary is allowed only in the model prompt, because that is the point of the test. The scanner checks every sink: replies, logs, database, reports, traces. An exact match in a forbidden sink fails the run, no questions asked.

Figure: Canary model: sink-aware allowlist at runtime, deny-all at the judge boundary.

The judge boundary is deliberately stricter than the runtime: this is the project’s first quality-evaluation path with redaction enforced before an external judge sees the transcript. No canary is ever allowed across that boundary, no matter what the fixture says. This came from a real review finding, where direct identifiers like emails and phone numbers were classified as judge-allowed and would have been sent to the judge provider verbatim.

Budgets: the guardrails

Quality runs spend real money, so they have hard budgets. Exceeding any of them terminates the scenario as incomplete.

If any live call lacks usage or cost data, the run is unmetered: it can complete only with an explicit --allow-unmetered flag, and unmetered runs can never be pinned as a baseline.

What the harness has caught so far

The deterministic lane is not decorative. It has found and pinned production bugs with regression tests:

• Unsafe errors looked safe. Fetch failures with HTTP status codes and unsupported-host messages were reported as generic errors instead of safe categories, so users got no honest explanation.
• Analysis crashed on save. The nested job-analysis call ran without a thread, so the memory processor threw before the model was called. Every save-path scenario would have been blocked.
• Fixture defaults stole scripted responses. The default model plan answered chat turns instead of staying inert, corrupting scenario expectations.
• PDF extraction read too much. Page and character bounds were enforced after extraction instead of during it, and bodyless download responses were buffered whole.

Each of those shipped with a focused regression test. That is the pattern: the harness finds the bug, the fix lands at the owning boundary, and the test keeps it fixed.

How to run it

npm run eval:test                    # contract lane: offline, deterministic, CI
npm run eval:quality -- --scenario s01-onboarding-collect   # quality lane: needs model keys

The contract lane needs nothing: no keys, no network. The quality lane needs the SUT model key and a judge key from a different provider family. Results land as redacted aggregates; raw transcripts never leave the machine.

Known limits

Every harness has edges, so here is the honest list:

• Restart durability is bounded. Pending saves and reply dedupe live in process memory. A durable ledger for post-restart replay is a documented seam that does not exist yet, so those subcases are explicitly incomplete, never silently passed.
• The failure-injection scenarios cannot run in the quality lane. Scenarios that inject a malformed model response need a scripted model; a real model will not cooperate. Their quality replay fails deterministically by design.
• Resume ingestion is out of the quality scope. The issue that defined this work excluded resume, PDF, and image ingestion from quality judging, so those scenarios are contract-only.
• Baselines and CI wiring are next. Three-run median baselines with immutable pinning, and the CI job for the contract lane, are the remaining slices.