# Evidence diagnosis record

Use this for one failed question. Start with the source that should answer it and the exact evidence the system returned. Then check whether the source exists, the caller may read it, it is current, search found it, and the answer follows it. A matching citation needs its own check. This record grants no source access, policy authority, model-provider permission, or approval to act.

The completed example reports a local search exercise. This download records the question, expected source, result, and checks; it does not include a runnable search program or generate an answer.

## Completed hypothetical lexical-miss case

- Question: “Who can greenlight a corrected bill before booking?”
- Decision need: identify who must approve an invoice correction before posting; no posting action is requested or authorized.
- Required source: `policy-ap17-v3`, AP-17 revision 3, owned by the controller.
- As-of date: 2026-09-12.
- Corpus revision: `invoice-policy-corpus-2026-09-12-v1`.
- Question-set revision: `invoice-policy-queries-2026-09-12-v1`.
- Caller: hypothetical reviewer-17 in northwind, with the required policy-reading scope.
- Admission check: source exists, belongs to the requested tenant, is active and effective, is not superseded, and the declared permission is current at the as-of date.
- Expected evidence: current AP-17 says a designated accounts-payable reviewer must approve every invoice correction before posting; automatic posting is prohibited.
- Actual lexical-baseline result: `insufficient_evidence`, with an empty evidence list.
- First failed layer: retrieval vocabulary/ranking. The query uses greenlight, corrected bill, and booking; the required policy uses approve, invoice correction, and posting.
- Expected reference is maintained outside the candidate. The candidate must not rewrite it to agree with its result.
- Immediate fallback: withhold the unsupported policy answer; request bounded clarification or use an already approved exact policy lookup if available. No source access can be widened.
- Proposed repair: compare controlled query handling or another admissible retrieval candidate on the same cases. No winning hybrid or generated-answer result is established.
- Review owner: source owner confirms applicability; search owner evaluates the retrieval change; workflow owner accepts the fallback.
- Qualification boundary: this public case is inspectable and can become development data if used for tuning. A real qualification claim needs new protected cases and target-system evidence.

## Nearby cases to keep

| Layer                | Case and required behavior                                                                                                              |
| -------------------- | --------------------------------------------------------------------------------------------------------------------------------------- |
| Source absence       | Required policy does not exist; preserve the gap and ask its owner                                                                      |
| Permission           | Source belongs to another tenant or caller lacks scope; do not return its protected content                                             |
| Freshness            | Superseded policy or expired exception; do not use it as current evidence                                                               |
| Ranking              | Current admitted source is missed by unfamiliar wording; preserve the failed query                                                      |
| Conflict             | Advisory planning proposal and current policy both return; preserve authority labels and request disposition where genuinely unresolved |
| Citation             | Wrong source ID, revision, owner, or excerpt; reject the evidence item                                                                  |
| Interpretation       | The answer claims more than admitted sources support; remove the claim or route to review                                               |
| Instruction boundary | A document tells the agent to ignore policy; keep it as untrusted content, never authority                                              |

A good ranking score cannot compensate for unauthorized evidence. A matching quotation does not prove source truth or correct interpretation. If current AP-17 never reaches the answer step, investigate search. If it arrives intact but the answer treats the unapproved planning proposal as the rule, investigate interpretation. The reported exercise stops at finding evidence, so it establishes no generated-answer result.

## Blank diagnosis

### Start with these questions

1. Which question failed, and which source should have answered it?
2. Did the caller have permission to read the applicable version?
3. Did search return that evidence intact?
4. If an answer was generated, did it follow the evidence that actually reached it?

### Question and required evidence

Have a qualified reviewer identify the expected source before showing them the search results. Otherwise a plausible-looking result may become its own unchecked answer key.

- What question failed, and what decision was the answer meant to support?
- What answer or outcome could be accepted, and who checks it independently?
- Which authoritative source IDs, owners, revisions, and as-of date apply?
- What proves those sources exist and apply to this question?
- Which customer's records and fields may this caller read, for which purpose and destination?
- What shows the source is current, has not been replaced, appears in the search collection when it should, and is withheld after access is removed?

### Actual result

- Which search method, query, and document collection produced this result, and which versions were used?
- Which exact sources, revisions, and excerpts returned, and in what order?
- Which required sources were missing, and which sources were forbidden?
- Which sources conflict, and which are approved rules rather than advice or proposals?
- If an answer was generated, what did it claim?
- Does each citation's source ID, revision, owner, and quotation match the source?
- If an answer was generated, which source supports each claim? Which claims exceed the evidence?
- Where did the first failure occur, and what evidence locates it there?
- What remains unknown?

### Repair and decision

- What bounded repair or fallback is proposed?
- Who owns the source, search, workflow, and independent review?
- Which cases will be used to develop the repair, and which fresh cases will remain untouched until the independent test?
- How will you test source permission/currentness, search ordering, conflicts, quotations, withholding an unsupported answer, and interpretation separately?
- May the sources be indexed, sent to any external service, and retained for the proposed purpose?
- Full cost, review burden, and latency limits:
- Disposition: repair / constrain / pause / stop:
- Next action, review date, and rollback trigger:

Preserve the original failure and independently owned expected evidence. Test the repair at its failed layer and rerun neighboring safety cases before promoting it.