Catch AI agent regressions before you ship

This lab records a real Claude agent run, freezes the recordings into a regression suite, and replays them against a cheaper model and against a prompt edit. An evaluator then says which change is safe to ship.

The agent

The agent is an on-call triage assistant. Given an alert, it calls three tools:

• get_service_status and get_recent_deploys are read-only. Replays answer them from the baseline recording.
• page_oncall has a side effect. It is stubbed, so a replay can never page a person.

The rule the agent must follow is to page the owning team when a service is degraded or down, and never when it is healthy. The fixtures hold six incidents: four should page and two should not.

The evaluator

The evaluator reads one recorded session and returns three verdicts: whether it paged exactly the expected team or correctly stayed quiet, whether it paged twice, and whether a summary is present. Kitaru's built-in cost evaluator runs alongside it.

The two experiments

• haiku-candidate: the same prompt, with the model swapped to a smaller one. The question is whether cost can drop without changing decisions.
• loose-prompt: the same model, with the rule "never page a healthy service" removed from the system prompt. The question is whether the evaluator catches that regression before it ships.

Both experiments share one fail-closed tool policy. Any tool the policy does not name stops the replay instead of running live.

The code, the walkthrough video, and the results are in the kitaru-claude-replay-demo repo.

More labs

• 
• 
• 

← All posts