# Evaluation coverage audit Inspect which situations a passing test report leaves out. This hypothetical exercise supplies a paper audit, not a runnable test suite. It establishes no performance on representative target cases, live model measurement, business result, or release approval. Read [Which cases belong in your evaluation set?](https://caisi.dev/notes/build-a-credible-ai-evaluation-set/). ## Audit this intentionally incomplete sheet first The service prepares and saves shipment facts for a coordinator to review; the tables call that staging a case. The service cannot command a carrier. This sheet selects five checks from a wider local test suite; it is not the complete inventory. Ask whether each expected result follows the shipping policy and whether the coordinator receives the facts needed to act. | Selected condition | Expected observable behavior | | -------------------------------------- | ---------------------------------------- | | High risk and high value | One coordinator case; no carrier command | | Stale evidence | Escalate before prediction or staging | | Explanation requests a forbidden route | Deterministic routing remains in charge | | Wrong tenant | Deny before disclosure | | Budget exceeded | Stop before explanation or staging | Write three missing conditions and who can check their expected results before continuing. Do not mark a type of case covered because its description resembles an existing test. | Missing condition | Expected result / evidence | Owning role / current status | | ----------------- | -------------------------- | ---------------------------- | | Your first gap | Record | Record | | Your second gap | Record | Record | | Your third gap | Record | Record | ## Suggested audit, not an exhaustive coverage standard | Gap | Required check / teaching evidence limit | | ------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Low-risk ordinary path | No review case or explanation call; wider local tests check this, absent from the five-case sheet | | Missing decision-critical score | Escalate before explanation or staging; absence is not low risk | | Duplicate delivery | Reuse the business operation's staged case; local single-world behavior does not prove remote concurrency | | Facts dropped from the summary | Inspect the downstream record; structured score, route, and revisions survive one weak explanation, but reviewer attention and explanation completeness are untested | | Target reference authority | Proposed logistics policy owner and independent evaluation approver; actual people and target approval absent | | Representative qualification | Target population, held-out partition, error uncertainty, and user behavior absent | ## Proposed case-authority record For a real service, ask the shipping-policy owner which rule applies, then have a separate qualified person check the expected result against the source. Keep any disagreement for the person authorized to settle it. The labels in this table are proposed responsibilities, not approvals by real shipping staff. Qualification here means evidence that the chosen version meets requirements on permitted, representative cases kept out of development; that evidence is absent. | Field | Hypothetical exercise value | | ----------------------------------- | -------------------------------------------------------------------------------------------- | | Scope | Stage a review case or take the declared safe disposition; no shipment intervention | | Reference basis | Permitted snapshot and declared routing-policy revision | | Source owner | Target logistics source owner, not yet supplied | | Label author / independent approver | To be assigned separately where required | | Conflict path | Hold, ask accountable policy/source owner, retain disagreement and disposition | | Negative control | Deliberately bad route/effect must fail the exact relevant check | | Model judge, if introduced | Versioned model/prompt/rubric; independently reviewed calibration examples and disagreements | | Development / qualification split | Not established; no qualification claim | | Current disposition | Teaching coverage audit; target readiness unproved | ## Paired evidence-loss check Keep the source, policy, and other inputs fixed. Remove one necessary fact, such as an exception or expiry date, from the summary passed to the next step. Decide beforehand whether that step should fetch the permitted source, withhold its answer, or ask a person. Keep both versions and record where the fact first disappeared; leave the cause unknown if the saved steps cannot show it. Retain later errors and costs separately. A citation alone does not prove the next step received or used the fact. Do not reward the same confident answer after its required evidence disappears. ## Blank record for your permitted workflow Start with the decision and one missing case. Write its expected result, supporting source, and checker before filling in the rest. A negative control is a deliberately bad result: record whether the relevant check rejected that specific mistake, without claiming that it catches all possible failures. | Question | Record before a qualification claim | | ------------------------------------------------------------------------------------------------------ | ----------------------------------- | | What decision is being tested, for which cases, and with which permitted effects? | Record | | What is the case ID, and which input, source, and policy versions apply? | Record | | Who owns the source, how sensitive is it, and what use is permitted? | Record | | What should happen, and which source supports that expected result? | Record | | Who wrote the expected answer, who independently approved it, and when must it be reviewed? | Record | | Which ordinary and high-consequence groups are covered, and which are missing? | Record | | Which deliberately bad result did you supply, and did the relevant check reject it? | Record | | In the evidence-loss pair, where did the first wrong step occur, and what can you not locate? | Record | | Does code, an expert, or a calibrated model judge the result? Which exact version or rubric applies? | Record | | Which cases were used during development, and which are frozen for the independent qualification test? | Record | | How many repeated runs are needed, and how much do results vary for each case? | Record | | What claim is supported, what remains uncertain, and who owns each gap? | Record | Do not paste private records, credentials, customer identifiers, or restricted source locations into a public worksheet.