In this note
A shipment coordinator needs the facts behind a risk alert before deciding what to do. The proposed service prepares and saves a review case for that person; it cannot redirect the shipment. Suppose its report shows five passing tests. Before trusting the score, ask whether those tests cover the information the coordinator needs and the failures the team must prevent.
Who approved the expected route? Does a missing risk score stop the service? Can a short explanation drop the fact a coordinator needs? What happens if a case is delivered twice? None of those questions disappears because the five selected cases passed.
We'll inspect an incomplete list of tests for a hypothetical shipment service. Its local tests use supplied risk scores; they do not measure a prediction model or a live shipping service. The shipment worksheet is a paper coverage audit, not that service's runnable test suite.
Revision note, 16 October 2026: added a separate prepared invoice-extraction lab for inspecting case coverage, expected answers, and runnable local offline checks. The live model comparison has not run.
Write down the decision before collecting examples
Preparing a case means assembling the shipment facts and saving the result for the coordinator. The table below calls that staging. Other permitted results are to leave the shipment alone, refuse an unauthorized request, or ask a person to resolve a problem. A test for the review path should check that the coordinator gets the necessary facts. A test for an unauthorized request should check that the caller gets no shipment information. The service must issue no carrier command on any path.
For each test, keep the request, the version of the shipment record, who is asking, the rule that applies, and the result you expect. Include what must never happen. Someone who knows the shipping policy needs to check the expected answer; copying yesterday's model output only gives you another unchecked answer.
Ask the person responsible for logistics policy to confirm the rule, then have a separate qualified reviewer check the test against it. If they disagree, keep the disagreement and name who will settle it. Nobody from a real shipping operation has approved the teaching answer below, so it cannot qualify their service for release.
Decision: stage a coordinator review case
Input revision: permitted shipment snapshot
Expected result: review_required under the declared routing policy
Forbidden effect: carrier command or shipment change
Reference basis: source snapshot and routing-policy revision
Target reference owner and independent approval: not supplied
Qualification status: teaching case onlyFind the missing row behind the passing score
Suppose a report shows only these five checks from a larger test suite.
| Decision or condition | What the selected case checks |
|---|---|
| High risk and high value | Stage one coordinator review case; issue no carrier command. |
| Stale evidence | Escalate before prediction or review staging. |
| Explanation requests a forbidden route | Keep routing under deterministic policy. |
| Wrong tenant | Deny before context disclosure. |
| Budget exceeded | Stop before explanation or review staging. |
Where is the ordinary low-risk shipment? Where is the request with no risk score? What if the same request arrives twice? The local suite tests those cases, but the five-case report doesn't show them. Find the missing tests in your own service or record that they are still missing.
Ask the team which cases cause trouble today. Perhaps two sources disagree, access is removed halfway through a request, or the review queue is already full. Add the applicable failures to the list. If you lack permission to obtain an example, mark the gap and name who can help. A similar-looking test does not automatically cover it.
Keep ordinary work in the set too. Deliberately collecting dangerous exceptions helps test defenses, but it won't tell you how often those exceptions occur. Report ordinary and difficult cases separately. Ten selected failures and ten routine requests are twenty examples, not an estimate of the business's failure rate.
Test the facts that disappear between steps
One teaching test supplies an explanation that says only that a carrier update exists. The text has the right format, but leaves out the risk score and the need for review. The separate case record still keeps the score, routing decision, source revision, model version, and rule version. The test confirms that a poor explanation does not erase those stored facts.
A coordinator who reads only the explanation could still miss the problem. Look at the screen or message the person actually receives. Can they find the score and the reason for review there? A fact stored elsewhere, or a link nobody opens, may not help them make the decision.
Try removing one necessary fact from the material passed to the next step while leaving the source unchanged. Remove an exception, currency, or expiry date. Decide beforehand whether the next step should fetch the source, stop, or ask a person. Don't reward it for producing the same confident answer after the evidence it needs has disappeared.
Use the run's step-by-step record to find where the fact first disappeared. If extraction drops an expiry date, the later routing error is a consequence of that omission. Keep both in the report. If the record cannot show where it happened, leave the cause unknown rather than inventing an explanation.
Make a known bad result fail
Test the test. Supply a result you know is wrong: a carrier command the service must never issue, another customer's data, or a review case missing required evidence. Check that the grader rejects it. This is a negative control, a deliberately bad example that shows whether the check catches that particular mistake. It doesn't establish that the check catches every kind of bad result. Add it separately; don't weaken the release tests to make a demonstration pass.
Use code for exact checks: which customer's data appeared, which source version was used, which route was chosen, and how many records changed. A qualified person can judge whether an explanation helps the coordinator. If you use a model for that judgment, first compare it with independently checked examples and inspect disagreements. Save its model and prompt versions. Two models agreeing can still mean two wrong answers.
Set some cases aside before changing the prompt or system. Use the others while developing, then stop editing and test the chosen version on the untouched cases. Otherwise you may only learn how to pass the five examples you've been staring at all week. If model answers vary between runs, retain that variation for each case instead of reporting the best run.
Count distinct business situations, not just test runs. A hundred rephrasings of one request are not a hundred independent cases. Requests from the same customer or incident may share the same problem too. Describe how cases were selected and account for those relationships when estimating uncertainty. Five hand-picked examples can show five behaviors; they cannot establish production reliability.
Add the missing case to your next review
Use the worksheet to inspect the five-case sheet before looking at its suggested answers. Pick one missing case. Write what should happen, where the correct answer comes from, who will check it, and who will obtain the example. Leave approval or coverage unknown where you cannot support it.
In the report, say which behaviors passed and which questions remain open. These local tests can check routing and access rules. They have not shown whether real coordinators notice errors or whether the service works on representative shipping data. Bring the missing case and its owner to the next review, alongside the passing score.