What Fixture Replay Can't Catch

Agents are hard to test end to end for one boring reason: the model does not return the same thing twice. Ask it the same question in two runs and you get two different sentences, two different orderings, sometimes a tool call and sometimes not. Write an assertion against that and you have written a coin flip.

The usual fix is to stop calling the model. You capture its responses once, save them to disk as fixtures, and replay them on every run — same request in, same bytes back, forever. The tests go deterministic, CI stops spending money on tokens, and the agent under test never knows it is talking to a recording.

We test our whole demo fleet that way. This post is about the bill, because replay is not free and the charge does not show up where you would look for it.

Here is the shape of it. Every deterministic harness buys its determinism by deleting a dimension of the real thing. Ours deletes time — deliberately, with the reason written in the source — and one specific class of bug vanishes along with it.

So the interesting question about a harness is not whether it is green. It is which dimension you deleted, because that is the list of bugs it cannot report.

Where do you put the mock?

At the model provider, not the app.

Our end-to-end suite starts a mock OpenAI server, then spawns the real agent server with its base URL pointed at that mock — langgraph dev for the LangGraph apps, uvicorn for the AG-UI ones:

const mock = await startMock({ mode: 'replay', fixturePath: opts.fixturesDir });
 
spawn('uv', ['run', 'langgraph', 'dev', '--port', String(langgraphPort)], {
  env: {
    ...process.env,
    OPENAI_BASE_URL: mock.baseUrl,   // the only thing that isn't real
    OPENAI_API_KEY: 'test-not-used',
  },
});

Everything above that line is the real thing. A real Angular app, the real streaming transport, a real Python server, real graph nodes with their edges and conditional routing. The model is the only stand-in.

The alternative is worth naming. A test that mocks the agent at the app boundary proves your component renders what you handed it. It cannot tell you that your graph's conditional edge routes correctly, that your transport merges deltas in the right order, or that a tool call round-trips. Push the seam out to the provider and all of that is under test, because none of it was replaced.

Replacing the model — and only the model — is also what makes it cheap enough to run everywhere: no API spend, no rate limits, no coin flips. We have 53 fixture files holding 169 entries across 36 apps — 34 workspace capabilities and two example apps — all on the same harness.

This is the outer tier. For in-process fakes at the unit level, the testing guide covers provideFakeAgent(), mockLangGraphAgent(), and MockAgentTransport, which are a different tool for a different job.

What does a fixture match on?

The shape of the request — and the order you list the entries decides which one wins.

A fixture file is a list of entries, and each one is a pair: a match block describing which request it answers, and the response to hand back when a request fits.

The obvious discriminator is the user message, but there are richer ones: a parent LLM's first call and its continuation after a tool round carry the same user message. Something has to tell them apart.

That something is hasToolResult, and matching is first-match-wins:

{
  "fixtures": [
    { "match": { "userMessage": "book a flight", "hasToolResult": true }, "response": "..." },
    { "match": { "userMessage": "book a flight" }, "response": "..." }
  ]
}

Swap those two entries and the run never terminates.

The continuation arrives carrying the same user message, matches the looser entry first, and gets handed the response that asks for the tool call again. The model calls the tool. The result comes back. It matches the looser entry again. Nothing errors. The assistant simply never finalizes.

For me that is the sharpest thing about fixture files: they are data, so they look inert, but the ordering is executable.

What did we trade away?

The streaming, by default.

The mock is constructed with a chunk size large enough that every response arrives in one or two server-sent events. Here is the note that sits above it:

// Use a large default chunkSize so ordinary fixture responses arrive in 1-2
// SSE deltas: most e2e assertions measure the final rendered DOM, and big
// chunks keep them deterministic. This is a determinism default, not a
// workaround — streaming-progressive behavior is covered by the unit
// variance tables and by fixtures that opt into small per-fixture
// chunkSize/latency values (see the fence fixture in
// cockpit/chat/messages/angular/e2e/fixtures/c-messages.json).
const mock = new LLMock({ port: 0, chunkSize: 4096 });

Structural assertions — a code fence renders as a block, a list is a list — are final-state invariants, and big chunks keep them from depending on where a token boundary happened to fall.

That 4096 is a default, not a law, and the second half of that comment is the useful half. Targeted streaming regressions opt into smaller per-fixture chunks: fixtures that set chunk sizes of three, four, six, eight, twenty-three, thirty-six, with latencies from 25 to 750 milliseconds, and e2e tests that sample the mid-stream DOM while it renders. What a real model's chunking does to a triple-backtick fence is covered one tier down too, in unit tables that drive the markdown parser one character at a time.

So "we deleted time" is too tidy. What we did was delete it by default and buy it back per fixture, in the places somebody decided the progressive render was the thing under test.

Which turns the question into a better one. Not what did the harness give up, but which fixtures opted back in — because a tier where nothing does is flying blind on everything mid-stream.

What does that hide?

Bugs that exist only while the stream is open and fix themselves before it closes.

Here is the shape, built from a mechanism the subgraphs post walks through. A child graph's tokens stream in under a namespace. A consumer that merged them into the transcript would show an extra chat bubble mid-run — the child's internal notes, rendered as if the assistant said them. Then the run settles, the parent publishes its authoritative state, the transcript is rebuilt from it, and the extra bubble vanishes.

Read that sequence from a test's point of view. The end state is correct. Two messages, in the right order. Assert on the finished DOM and it passes — not by luck.

That is why our transport routes child tokens structurally instead of by merge decision — and it is why verifying that property means driving a live model and sampling the DOM on a tight interval for the length of a full run, watching that the message count never crosses two. Not something a final-state suite does, and not something it could tell you.

Let us generalize, because this is not specific to us. Any assertion that runs after an await sees a settled system. A self-correcting bug is precisely one that settles. So the class of defects a final-state suite cannot see is not random — it is exactly the ones that repair themselves.

What runs alongside it?

A live pass for what replay structurally cannot see, and a weekly drift run for the model itself.

The live pass is unglamorous: for anything whose failure mode is mid-stream, drive it against a real model in a real browser before it ships. There is no clever tooling in it. The deterministic suite is for final-state invariants, and mid-stream behavior is not one of them.

Drift is the other half, and the tempting design is the wrong one. Re-record fixtures and diff the bytes and you detect that a response changed size while learning nothing about whether it changed meaning — the same trade again, one layer up. A model that starts returning something equally long and completely different sails straight through a size check.

So the drift check is not a new metric. It is the assertions we already trust.

The drift run takes a tagged subset of the same e2e suite — contract assertions only: a reply renders, the research dispatch surfaces a subagent card, the interrupt panel appears — and points it at the live provider through the mock's record-proxy, which any app on the harness can switch on with an environment variable. No fixtures judged, no thresholds invented. If today's model stops calling the research tool, or the graph's prompts stop eliciting the interrupt, a spec we already believe in goes red and a weekly job opens an issue. Meaning drift is caught by construction.

One rule makes the subset work: a tagged assertion may depend on structure or on the prompt's own terms — an element exists, a reply to "say hi" matches /hi/i — never on the content of a canned response. A spec that expects the fixture's exact words fails against a live model whether or not anything drifted, so it stays in replay where it belongs. A structural differ runs alongside as a diagnostic, and it files recordings the recorder itself flagged as incomplete under their own category instead of counting them as drift — an instrument reporting on its own blind spot rather than disguising it.

Conclusion

My recommendation is simple: push your seam as far out as you can afford. The further out it goes, the more of your stack is under test rather than simulated, and provider base URL is far out for how little it costs.

Then write down which dimension you deleted to make it deterministic, in the file where you deleted it. Not in a wiki. Six months later that comment is the difference between "the suite is green" and "the suite is green, and here is what green does not cover."

For us the remaining list is short and specific: mid-stream behavior on fixtures that never opted into real chunking, and the model itself moving under the fixtures. Both are answered by pointing what you already trust at the real thing — the tagged specs at the live provider on a schedule, a live browser at anything mid-stream before it ships. Widening either net is a fixture opt-in or a spec tag, not a new design.

The testing guide has the in-process tier, and the subgraphs post has the streaming attribution model this post leans on.

If your agent suite is green today, I would like to know what you think it cannot see.