What Fixture Replay Can't Catch

Every deterministic test harness buys its determinism by deleting a dimension. Ours deletes time, deliberately, and the reason is written in the source.

So the interesting question about a harness isn't whether it's green. It's which dimension you deleted, because that's the list of bugs it can't report.

Where do you put the mock?

At the model provider, not the app.

Our end-to-end suite starts a mock OpenAI server, then spawns the real agent server with its base URL pointed at that mock — langgraph dev for the LangGraph apps, uvicorn for the AG-UI ones:

const aimock = await startAimock({ mode: 'replay', fixturePath: opts.fixturesDir });
 
spawn('uv', ['run', 'langgraph', 'dev', '--port', String(langgraphPort)], {
  env: {
    ...process.env,
    OPENAI_BASE_URL: aimock.baseUrl,   // the only thing that isn't real
    OPENAI_API_KEY: 'test-not-used',
  },
});

Everything above that line is the real thing. A real Angular app, the real streaming transport, a real Python server, real graph nodes with their edges and conditional routing. The model is the only stand-in.

Let's take the alternative. A test that mocks the agent at the app boundary proves your component renders what you handed it. It cannot tell you that your graph's conditional edge routes correctly, that your transport merges deltas in the right order, or that a tool call round-trips. Push the seam out to the provider and all of that is under test, because none of it was replaced.

Replacing the model — and only the model — is also what makes it cheap enough to run everywhere: no API spend, no rate limits, no coin flips. We have 50 fixture files holding 129 entries across 34 apps — 32 cockpit capabilities and two example apps — all on the same harness.

This is the outer tier. For in-process fakes at the unit level, the testing guide covers provideFakeAgent(), mockLangGraphAgent(), and MockAgentTransport, which are a different tool for a different job.

What does a fixture match on?

The shape of the request — and the order you list the entries decides which one wins.

Each entry carries a match block. The obvious discriminator is the user message, but there are richer ones: a parent LLM's first call and its continuation after a tool round carry the same user message. Something has to tell them apart.

That something is hasToolResult, and matching is first-match-wins:

{
  "fixtures": [
    { "match": { "userMessage": "book a flight", "hasToolResult": true }, "response": "..." },
    { "match": { "userMessage": "book a flight" }, "response": "..." }
  ]
}

Swap those two entries and the run never terminates.

The continuation arrives carrying the same user message, matches the looser entry first, and gets handed the response that asks for the tool call again. The model calls the tool. The result comes back. It matches the looser entry again. Nothing errors. The assistant simply never finalizes.

For me that's the sharpest thing about fixture files: they're data, so they look inert, but the ordering is executable.

What did we trade away?

The streaming, on purpose.

The mock is constructed with a chunk size large enough that every response arrives in one or two server-sent events. Here's the whole note that sits above it:

// Use a large chunkSize so each response arrives in 1-2 SSE deltas. This
// intentionally turns off the partial-markdown streaming path for harness
// tests: structural assertions (code fence, list) measure the FINAL rendered
// DOM, not the progressive render. With aggressive default chunking, the
// partial-markdown parser sometimes can't recover a triple-backtick fence
// that gets split mid-token, and the final state ends up as inline <code>
// instead of <pre><code>. Streaming-progressive behavior is covered by the
// Phase 1 unit-variance tables; the e2e harness is for final-state
// invariants and cross-stack integration.
const mock = new LLMock({ port: 0, chunkSize: 4096 });

A real rendering bug, but a streaming one, and it was making structural assertions flaky for reasons that had nothing to do with what they asserted. So the timing went away and the property moved down a tier.

I think that's the right trade, and the reason isn't that the flakiness went away — it's that the property didn't.

The tempting alternative is to replay the recorded chunk boundaries instead of re-chunking, so you get the timing back for free. That doesn't buy what it looks like it buys. Faithful boundaries make the fence failure deterministic rather than absent — the parser bug is still there, and now every structural assertion in the suite fails for a reason none of them are about.

Now the part that's easy to get wrong without going looking, and it's the useful half.

That 4096 is a default, not a law. A second harness serves our two example apps, and its version of that comment says so outright — ordinary fixtures get the big chunk size, and targeted streaming regressions opt into smaller per-fixture chunks. Those fixtures set chunk sizes of three, four, six, twenty-three, thirty-six, with latencies from 25 to 750 milliseconds. There are e2e tests over there that sample the mid-stream DOM while it renders.

So "we deleted time" is too tidy. What we did was delete it by default and buy it back per fixture, in the places somebody decided it was worth the cost.

Which turns the question into a better one. Not what did the harness give up, but which tier opted back in — because the tier that didn't is the one flying blind.

What does that hide?

Bugs that exist only while the stream is open and fix themselves before it closes.

We shipped one recently and fixed it, and where it lived is the whole argument: a cockpit capability, on the harness with no per-fixture opt-in.

A demo runs a child graph as a plain node — the shape the subgraphs post ends on. A plain subgraph node's events aren't tagged as a delegated subagent, so the bridge merges the child's tokens into the transcript as they arrive.

The child in that graph produces an internal research brief, meant for the parent to write its answer from. Without the option that whitelists which nodes count as transcript, that brief renders as its own chat bubble, and the message list transiently reaches three.

Then the run settles. The parent publishes its authoritative state, the transcript is rebuilt from it, and the extra bubble vanishes.

Read that sequence again from a test's point of view. The end state is correct. Two messages, in the right order. Assert on the finished DOM and it passes — not by luck.

Confirming the fix meant driving it against a live model and sampling the DOM on a tight interval for the length of a full run, watching that the message count never crossed two. Not something the suite does, and not something it could tell us.

Let's generalize, because this isn't specific to us. Any assertion that runs after an await sees a settled system. A self-correcting bug is precisely one that settles. So the class of defects a final-state suite cannot see isn't random — it's exactly the ones that repair themselves.

What runs alongside it?

A live pass for what replay structurally can't see, and a weekly drift run for the model itself.

The live pass is unglamorous: for anything whose failure mode is mid-stream, drive it against a real model in a real browser before it ships. That's how the bubble above got caught. There's no clever tooling in it — the point is just that the deterministic suite was never going to be the thing that found it.

Drift is the other half, and it got built properly on the way to this post. Our first design re-recorded fixtures and compared byte size — which detects that a response changed size while saying nothing about whether it changed meaning. The same trade again, one layer up. A model that starts returning something equally long and completely different sails straight through a size check.

So we threw the metric away and kept the thing we already trusted.

The assertions are the drift check.

The drift run takes a tagged subset of the same e2e suite — contract assertions only: a reply renders, the research dispatch surfaces a subagent card, the interrupt panel appears — and points it at the live provider through the mock's record-proxy. No fixtures judged, no thresholds invented. If today's model stops calling the research tool, or the graph's prompts stop eliciting the interrupt, a spec we already believe in goes red and a weekly job opens an issue. Meaning drift is caught by construction.

One rule makes the subset work: a tagged assertion may depend on structure or on the prompt's own terms — an element exists, a reply to "say hi" matches /hi/i — never on the content of a canned response. A spec that expects the fixture's exact words fails against a live model whether or not anything drifted, so it stays in replay where it belongs.

The first run flagged drift that wasn't there. The diagnostic differ reported that both tool-calling responses had "drifted" to plain text — while the specs proving those tools fired were green. The recorder had saved empty content because it couldn't parse tool-call deltas out of the stream, warning "fixture may be incomplete." The check's first finding was a blind spot in its own instrument; it now reports that case in its own category instead of as drift. It's run clean against the live model since.

Conclusion

Push your seam as far out as you can afford. The further out it goes, the more of your stack is under test rather than simulated, and provider base URL is far out for how little it costs.

Then write down which dimension you deleted to make it deterministic, in the file where you deleted it. Not in a wiki. Six months later that comment is the difference between "the suite is green" and "the suite is green, and here is what green does not cover."

Ours was written down, which is the only reason it could be checked at all. Checking it surfaced two things: the deletion was a default two of our apps had already bought back, and our first drift design was measuring the wrong thing.

For us the list is short and specific: anything whose failure mode is mid-stream on a capability that never opted back into real chunking, and the model itself moving under the fixtures. Both are answered by pointing what you already trust at the real thing — the tagged specs at the live provider on a schedule, a live browser at anything mid-stream before it ships. Widening the drift net is tagging more specs, plus one chore: porting record mode to the harness the other thirty-two apps share. Widening the stream check is opting more fixtures into real chunking. Neither is designing anything new.

The testing guide has the in-process tier, and the subgraphs post has the bug that started this.

If your agent suite is green today, I'd like to know what you think it can't see.