In this note
A reviewer asks, “Who can greenlight a corrected bill before booking?” The current policy says a designated accounts-payable reviewer must approve every invoice correction before posting. Search returns no evidence. The policy exists, and this reviewer may read it. The words in the question failed to match the words in the source.
This failure was reproduced in a local search exercise over hypothetical records. It returns documents, not generated answers. This note and its worksheet report the failed question, expected source, and result; they don't include a runnable search program.
I’d inspect those layers before paying for a more capable model. Sometimes the evidence is available but hard to find. Sometimes access is denied, a policy expired, or two documents disagree. Each condition needs its own repair and fallback.
Find the missing step
On the exercise date, 12 September 2026, the policy is AP-17 revision 3. It says approve, invoice correction, and posting. The question says greenlight, corrected bill, and booking. The baseline search method, BM25, scores word matches. It misses the shared meaning and returns insufficient_evidence without the required policy.
Save the failed question as a regression test. Try checked synonyms, query changes, or a search method that compares meaning. Test each against the same permitted documents and independently chosen expected sources. A fix for this question doesn't prove it handles every paraphrase. Once you've used a question to tune search, use fresh held-out questions to assess it.
Before showing the hits, ask a reviewer which source should answer the question. Otherwise, a related-looking search result can become its own answer key. Here the reviewer requires the current AP-17 policy. A question about an invoice balance would need the current invoice record instead.
| Job | Check or mechanism |
|---|---|
| Look up a known policy ID | Direct source lookup or exact search, followed by current-version and access checks |
| Find a policy from unfamiliar wording | Compare word matching with search that compares meaning, using only permitted, applicable sources |
| Read a specific invoice’s state | Request the invoice record through an interface that checks the caller's identity, access, and required revision |
| Explain how evidence applies | Check the explanation separately: does it say only what the returned evidence supports? |
| Approve or post a correction | The authorized review and action service; retrieved prose cannot grant permission |
If the workflow already knows the policy ID, a direct lookup may be enough. Keep the source ID and revision with the result. You don't need an agent searching the whole organization for a record you can request by name.
Permission and freshness come before ranking
Before ranking documents, the exercise checks the organization, caller permissions, source status, effective dates, permission expiry, and replaced versions. Only allowed, applicable documents enter search. Old AP-17 permits a form of automatic posting that the current policy prohibits. A strong word match can't make the old rule applicable.
Other test questions ask for another organization's private policy, a policy the caller can't read, or a June-only exception in September. The exercise withholds those documents. That tests its authored permission and date checks; it doesn't verify a real identity provider, ingestion service, cached permission, or revoked access.
If the organization never wrote the policy, wider search won't find it. Ask the policy owner what the team may do while it is missing. Authorized diagnostics should distinguish a missing document from denied access, without revealing private contents to the requester.
Permission to read a document doesn't necessarily allow sending it to an external embedding or answer service. Check the destination, data classification, retention, and permission to index it before testing that route. Exclude a route that cannot meet the policy; a better ranking score doesn't repair the permission gap.
The first result can be the wrong authority
A planning note proposes automatic posting for small corrections. It says the controller hasn't approved it and AP-17 still applies. Search returns both documents, with the planning note first. The top search result is a proposal, not the governing rule.
Keep both documents and their revisions. The answer should distinguish the proposal from the current rule. If two governing sources genuinely disagree, ask the policy owner to resolve the conflict and use the agreed fallback meanwhile. Search scores cannot decide which person or document has authority.
The exercise checks each citation's source ID, revision, owner, and excerpt against its document set, catching altered citations. A matching quotation still may be irrelevant, advisory, or wrong. A system that generates answers needs a separate test of whether its claims follow from the evidence.
If current AP-17 never reaches the answer step, investigate search. If it arrives intact but the answer still treats the unapproved automatic-posting proposal as the rule, investigate how the model used the evidence. Returning the right document is necessary for that answer, but doesn't establish that the answer follows it.
Inspect the excerpt the model will receive. If it says corrections may post automatically but omits the nearby sentence saying the proposal is unapproved, it will mislead. Keep qualifications, dates, and exceptions with the quote. Having the full document somewhere in the index isn't enough.
One supplier attachment tells an AI system to ignore the policy. Search may return that attachment when it is relevant, but the exercise marks every evidence item instruction_authority: false. Retrieved text is source material, not an instruction that can change permissions or policy. Preserve that distinction when building the model's context.
Record the layer that failed
Record which check failed instead of keeping only one good-answer percentage. Did search find the required document? Was it current and permitted? Did it show the conflict and a valid citation? Once those checks pass, test whether the answer interprets that evidence correctly. Better retrieval shouldn't hide a permission failure.
Set the allowed sources and number of attempts. A miss shouldn't expand access or start an unlimited search loop. Keep any permitted second attempt with the first so you can see whether it found the evidence and what it cost. Name who handles the question if search still fails.
| Failed layer | Repair or fallback |
|---|---|
| Source absent | Source owner supplies the missing evidence or approves a constrained fallback |
| Permission or freshness | Repair the trusted access/current-version boundary; stop before returning inadmissible content |
| Ranking or vocabulary | Test a search change on fixed cases, including the preserved miss |
| Conflicting evidence | Keep the disagreement visible and request an owner disposition |
| Citation mismatch | Reject the evidence item or rebuild against the correct source revision |
| Unsupported interpretation | Remove the claim, request evidence, or send the packet to review |
The worksheet includes the failed greenlight query. For your own failure, record the question, expected source, permission and version checks, actual results, and next owner. That gives the team more to investigate than a ticket saying the agent hallucinated.
Before replacing the model, save one failed question and the exact sources and results it received. If an answer was generated, keep the unsupported claim too. Repair the first failed check and rerun the case. Test finding the evidence separately from using it well.