In this note
Someone asks an agent, “What was revenue for September?” The billing report says $120,000. The approved finance report says $95,000. Both dashboards use the label revenue. A model can write a convincing explanation for either number while answering a different question from the one the requester meant.
In this hypothetical company, the two reports measure different things. Neither total is wrong merely because they disagree. We’ll use the definition approved by the finance owner, rather than ask the agent to decide how the company should recognize revenue.
I’d resolve the meaning before testing prompts or adding a database. Ask which report the answer must agree with, who owns its definition, and what to do when the request could refer to more than one metric.
Agree which number the request needs
In our hypothetical request, the controller confirms that the user needs recognized revenue from the approved September management report. The billing report measures billed amounts. Rename the two fields so they remain distinct in queries, tool outputs, and answers. “Revenue” alone is too vague to act as a metric identifier.
Write down which accounts and period to include, the currency, and what one row means. That last property is often called grain: one row might be an invoice line or a monthly account total. If you mix those up in a join, you can count the same money twice. The table records the definition for this request.
| Contract field | Agreed entry |
|---|---|
| Metric and owner | management_recognized_revenue; controller |
| Purpose and population | Internal management reporting; approved customer accounts in this company only |
| Period and currency | September 2026 reporting period; CAD; no conversion in this request |
| Grain and calculation | Sum approved account-period totals from the finance report; one total per account and period |
| Definition and source | Definition rev-4; approved finance snapshot fin-sep-7 |
| Exceptions | Only entries already included in the approved snapshot; the controller must decide whether and how a later credit changes the requested report |
Here, the approved report supplies the amount. The agent does not work out accounting treatment from raw invoices. Ask the owner to confirm a different definition when the requested period, accounts, or reporting basis changes. Use the same metric name and version wherever that definition is reused.
Separate definitions, relationships, records, and context
You may hear four labels when discussing this design: semantic layer, ontology, knowledge graph, and context. For the revenue question, they describe different jobs. The table shows what each would contribute; it doesn’t imply you need four new systems.
| Responsibility | What it contributes here |
|---|---|
| Semantic layer | The metric’s calculation, grain, filters, reporting calendar, and valid joins |
| Ontology | The business types and relationships: an account has subscriptions and report entries; a report entry belongs to an account and period |
| Knowledge graph | Connected records or facts, with attributable links to their sources and revisions |
| Context assembly | The permitted definition, evidence, task instructions, and unresolved questions packaged for this request |
A versioned metric document, ordinary tables with tested keys, and a query may do all the work needed here. Add a graph when following relationships solves a demonstrated problem. Drawing an account-to-invoice link cannot tell you whether the amount is right or the requester may see it.
Software must still check those relationships. Describing an entry as belonging to one account does not prevent an incorrect join. Describing someone as an approver does not grant them access. The services reading or changing records enforce those rules.
Prepare the evidence for one question
We can now give the model a specific job: explain the approved September total for two accounts the caller may review. Software checks access and selects the report entries before preparing the packet below. It includes the question and metric definition, without sending the whole finance warehouse.
request: explain September total for accounts A and B
metric: management_recognized_revenue@rev-4
source: approved finance snapshot fin-sep-7
period: September 2026; currency: CAD
account A approved total: 60,000
account B approved total: 35,000
approved total: 95,000
unresolved: later credit C-9 has no approved treatment
permitted response: explain the approved snapshot; flag C-9
prohibited: infer credit treatment or alter finance recordsIn this teaching example, $60,000 plus $35,000 equals $95,000. Code can check the addition against the approved snapshot. The model may explain what the supplied entries say. Neither component can decide how credit C-9 changes the report when that decision is absent from the approved evidence.
Keep a link to the report and record both its reporting date and when you fetched it. Fetching an old report today doesn’t make its figures current. Reviewers should be able to open the original source. Access checks must also cover fields brought in by joins, not only the first table queried.
Keep the task instructions separate from report text. A note saying “ignore the credit” cannot override the owner’s rule. Show the conflict to the owner and ask them to decide. Keep the same access and permitted-use limits when saving summaries, caching the packet, retrying, or logging it.
Ask for a decision when the evidence ends
The agent can answer, “The approved September management report totals CAD 95,000 for accounts A and B. Credit C-9 is unresolved and is not reflected in this snapshot.” If the user asks for an adjusted amount, ask the controller whether C-9 changes this report and which approved figure should replace the snapshot total. Until that decision and its supporting record are available, return only the approved total with the limitation visible.
If the original request remains “September revenue,” ask whether it means billed amounts or the approved management metric. When the system is designed to use one fixed metric, display that choice before the user accepts the answer. Do not let a model silently pick the larger total because it seems more plausible.
If you save the agent’s summary, label who generated it, which report it used, and whether anyone reviewed it. Saving it beside approved finance records must not make it look like another approved fact.
Test meaning as well as arithmetic
Build cases around the question the owner has approved. A request for billed amounts must not return the management-report total. A September question must not include an October entry. The same account appearing twice must not be counted twice. An unresolved credit must remain unresolved. A caller outside account B’s scope must not receive its amount through the answer, a citation, or a log.
Have someone check the expected answers before running these tests. If you’re testing calculation, keep the answer key out of the model’s input. Our packet deliberately supplies the approved total, so it can test whether the agent explains that total accurately and flags the credit. It cannot test calculation from raw records.
When the definition changes, keep rev-4 and compare proposed rev-5 on the same permitted cases. Check the affected queries and answers. Use those cases for a model change too: an answer can keep the same format while quietly switching which revenue it reports.
Bring one disputed metric to its owner
Choose a measure that appears in two reports. Sit with its owner and write which question each report answers. Fill in the definition and one small request packet using records you may inspect. If nobody can approve the meaning or check the answer, leave that question open and name who needs to resolve it.