# Invoice extraction: a prepared public experiment **No live model run has been completed.** This bundle contains newly authored invoice texts, separately reviewed expected answers, candidate code, an offline grader, and local negative-control tests. It contains no customer documents or measured model results. The live results article is deferred until a run can be completed and reviewed. The question is small: can an extractor return the explicitly stated current amount due and supported currency, and ask for review when the source cannot establish them? ## Try the offline checks first Unpack the bundle into a new directory. With Node.js 22 or later, run: ```sh node --test self-test.mjs node run.mjs ``` Neither command sends a request or needs an API key. The first tests authored good and bad outputs. It changes USD 120.50 to USD 120.00 without breaking the record's shape, then checks that the source comparison rejects it. It also checks missing currency, competing due lines, gross versus unpaid amounts, malformed outputs, and a candidate that asks for review on every case. These are code tests, not model responses. The second command verifies the frozen files and reports `prepared_no_live_model_run`. It does not print an accuracy score or pretend that preparation is an experiment result. ## What the comparison will test There are 15 development documents and 30 test documents, with five test cases in each of six groups: clear amounts, competing amounts with a final due figure, varied prose or formatting, explicitly current revised/credited balances, missing or conflicting evidence, and irrelevant instructions embedded in documents. All texts and amounts were authored from scratch. No customer or employer documents were transformed, anonymized, or reconstructed. The deliberate coverage is useful for inspecting failure behavior; it does not represent the frequency of invoice problems in a business. Three routes receive the same source input: | Route | What it does | Important limitation | | ------------------------ | -------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Literal-line code | Reads a single `Total:` line with an explicit supported currency. | A tiny illustrative rule, not the best possible deterministic extractor. A gross total may not be the remaining amount due. | | Structured model output | Requests the declared fields or a review decision from one fixed model snapshot. | A correctly shaped record can still contain the wrong fields. | | Model plus source checks | Rechecks that same saved model output against narrow due-line rules. | It accepts only a unique `Amount due:`, `Balance due:`, or `Total due:` line, and deliberately sends more complex sources to review. It is not a general semantic or prompt-injection verifier. | The last two routes use the **same** response; they do not make two model calls. This isolates what the added checks change. They can reduce coverage by returning a good extraction for review, which must be counted alongside caught mistakes. A model that refuses every case does not earn a successful-extraction score. ## Keep the answers out of the request `development.json` and `holdout.json` contain source inputs. `references.json` is a separate offline answer key. The API receives only one case's four input fields and the fixed prompt/schema. It receives no expected amount, group label, case ID, other case, or grading result, and has no tools. The source checker reads source text, not the answer key. The grader loads references after the model calls finish. Correctness is scored on status, requested invoice, currency and integer cents, with a verbatim source-quote presence check. This does not independently establish that a quote explains the right business meaning, that a person accepted the work, or that a source revision is current in a live system. Those would need separate checks. A separate agent read all 45 source/reference pairs before publication and found no label disagreement. That is agent review, not human finance approval. Source and reference hashes are recorded in the corpus manifest. The candidate author did not inspect the test texts or their answers before freezing the candidate setup. Separation in the shared authoring workspace was procedural, not operating-system secrecy. The complete corpus is public now. Once you inspect its test cases or tune against their results, do not describe it as an unseen test of a later candidate. Preserve failed results and use new, separately held-out cases for a new generalization claim. ## Live use is a separate, explicit action Live mode sends these authored texts to OpenAI and can incur API charges. Do not add private documents. Configure `OPENAI_API_KEY` through your normal secure environment setup, never in a command, source file, or result file. Then, if you choose to make the calls: ```sh node run.mjs --live --split development --runs 1 --budget-usd 1 --out development-run.json node run.mjs --live --split holdout --runs 3 --budget-usd 1 --out holdout-run.json ``` Each output path must be new; the runner refuses to overwrite earlier evidence. Visiting the Caisi site or opening these documents never starts a model call. There is no website API endpoint, server-side model integration, or bundled credential. The runner saves snapshots before requests and on handled failures, but those writes are not a crash-atomic journal. A power loss or forced process kill during a write can leave an unreadable result file. Do not count that file as a completed comparison or silently replace it. This small runner is not a durable production execution service. The prepared setting is `gpt-5.4-nano-2026-03-17`, reasoning effort `none`, strict structured output, no tools, standard service tier, and `store: false`. Live execution is deferred until working API access is configured. No model response or usage has been collected. `store: false` is not a promise of zero provider retention; review the provider's current data terms before live use. The runner permits at most three repetitions, 105 requests, 600 output tokens per request, a 45-second request timeout, a 15-minute total deadline, and a maximum USD 1 reserved list-price budget per invocation. It reserves a conservative input/output allowance before each call, keeps unknown/failed-call reservations, and does not retry automatically. It stops after an HTTP rejection, transport problem, unaccounted usage, changed model snapshot, or exhausted limit. A stopped file is partial evidence, not a completed comparison. The starting price calculation uses USD 0.20 per million uncached input tokens, USD 0.02 cached input, and USD 1.25 output, checked on 16 October 2026. It computes a **list-price estimate from returned token usage**, not your invoice, discounts, tax, or full workflow cost. Verify current rates before future use; changing frozen rates requires a new recorded setup. Official references: [model snapshot and features](https://developers.openai.com/api/docs/models/gpt-5.4-nano), [structured outputs and refusal handling](https://developers.openai.com/api/docs/guides/structured-outputs), [pricing](https://developers.openai.com/api/docs/pricing), and [data controls](https://developers.openai.com/api/docs/guides/your-data). ## Read a later result carefully The output retains sanitized request attempts, raw model outputs, model identity, usage, elapsed time, candidate records, source-check reasons, and scores by route and group. Provider refusals and incomplete responses remain provider failures; a safe wrapper sending them to review does not count as a correct model answer. The same input repeated three times produces dependent observations, not 90 independent documents. The deterministic route is repeated for alignment, not to gain confidence from identical outputs. The protocol requires a later readout to show useful correct extractions, wrong extractions, appropriate and unnecessary review, invalid outputs, provider failures, variation between runs, request latency, and usage-based cost. Do not report only a combined pass percentage. A review count is not measured human effort or proof that someone will catch an error. There are no posting, payment, OCR, live source updates, human-review trials, or production qualification in this experiment. If a result disagrees with a reference, keep the original output and score while checking the source and policy. Record any correction separately. Do not tune the frozen candidate, relabel a difficult case, or silently rerun failed calls to make the result look better.