You changed one rule. What else changed?

Change one thing and keep the cases the same

In this hypothetical example, a shipment goes to review if its score meets the threshold and its value is at least USD 10,000. The current rule uses 0.80; the proposed rule uses 0.70. Keep the value requirement, cost limit, access checks, and source-age checks unchanged. Only the score threshold changes.

Run both rules on the same eight records. Each supplies a customer account, permissions to read and prepare a review, a fresh-or-stale flag, a risk score, shipment value, and incurred cost. We supplied these facts for teaching. No model predicted the scores, and a flag marked fresh does not measure source age. A real test needs traceable records, current permissions, explicit time rules, and cases that suit the question being tested.

Save the exact rule, code, inputs, and test versions. If a model is involved, save its prompt, model version, prepared context, tools, and settings too. Changing several of them at once makes it harder to tell what caused a difference. A familiar filename is not enough to identify what ran.

Before: teaching-routing@1, score >= 0.80
Candidate: teaching-routing@2-candidate, score >= 0.70
Shared: shipment value >= USD 10,000
Inputs: the same eight declared cases
Decision: propose coordinator review, never a carrier command

Find the first decision that changes

Two cases move into review. In both, the first difference in the step-by-step record is route_selected; the earlier read, evidence, and score checks match. We can see where the behavior changed. We cannot call it better: no independently checked answer tells us which shipments actually needed intervention.

The old rule selects one of eight cases; the new rule selects three of eight. Those are counts of selected teaching cases, not accuracy or successful interventions. A coordinator may open a case and do nothing. They may act and still fail to avoid a delay. To judge benefit, agree on the result that matters, obtain independently supported references, and observe what follows. Leave later outcomes unknown until you have them.

Keep the unchanged cases too. Selecting the extra cases correctly cannot compensate for exposing another customer's data. The companion checks the customer account before using evidence, stops stale cases before routing, and stops the over-budget case before proposing review. Those are local checks; they do not test a real identity service, clock, or remote review queue.

When a run has several steps, look for the first wrong one. A change to context preparation could drop a needed fact and cause a later routing error. Record the omission and its later consequences separately. If the saved steps cannot show where it began, leave the cause unknown.

Follow the extra cases into the review queue

The current rule selects one of the eight cases; the candidate selects three. Suppose each review needs six minutes and the team has twelve minutes available for this entire hypothetical case pack. Current demand is six minutes. Candidate demand is eighteen. Under those assumptions, the candidate exceeds the declared capacity by six minutes.

This is planning arithmetic, not a forecast of daily demand. We haven't measured six-minute reviews or a real queue. Inspect actual arrivals, busy periods, review times, and quality, including the other work reviewers must set aside. More review may help, but it may also require a smaller initial workload or more qualified people.

Ask the workflow and service owners to decide: reject the change, test it on less work, or arrange capacity and test again. Sending overflow to a less qualified person changes the proposal. If you consider that option, assess its risks and results separately rather than hiding it in the original comparison.

Keep the change on hold when the queue cannot cope

For these assumptions, keep the change on hold: eighteen minutes of review will not fit into twelve available minutes. We also lack checked real-world answers, representative work, the receiving team's acceptance, and actual operating permissions. The companion has no deployment connection or release approval. Its passing tests cannot supply those missing decisions.

Before a real release, the responsible owner needs compatible software and settings, results for affected cases, agreement from the operating team, rollout limits, and a recovery plan. Decide what stops new work, who gets the alert, and who handles pending cases. Test that response in the actual environment.

Restoring 0.80 won't remove the work already sent for review. Keep its operation IDs, saved records, and responsible people across the version change. Check those cases through the agreed recovery process. An empty-looking queue is no improvement if its unresolved work has merely disappeared from view.

Keep earlier results with the setup that produced them. If only the rule used to grade an unchanged saved answer changes, you may be able to grade that answer again, provided you retained everything the new check needs. Changing the shipment-routing threshold is different: it can change what the service does, so rerun the affected steps as this comparison does. Never loosen checks or remove difficult cases to make the change pass.

Run the small comparison

node --input-type=module -e 'import { replayCases, currentPolicy, candidatePolicy, comparePolicies } from "./release-replay.mjs"; console.log(comparePolicies(replayCases, currentPolicy, candidatePolicy));'

// currentReviews: 1; candidateReviews: 3
// declaredCurrentMinutes: 6; declaredCandidateMinutes: 18
// candidateFitsDeclaredCapacity: false

The download runs this comparison offline without training a model, calling a carrier, or changing a live service. For your own proposed change, keep the inputs fixed, name the old and new versions, and say which decisions should differ. Save the safety cases that should stay unchanged, the first observed differences, and the extra work. Take that record and the remaining questions to the person who can approve or stop the next step.