The 12 Factors of AI Value Engineering

What are you counting as success?

Choose something you can count and someone can reject. For an invoice correction, the reviewer approves the exact change and a ledger check confirms it was recorded correctly. That is an accepted outcome. A recommendation the assistant has only drafted is an earlier event.

A review-only assistant has a smaller job. You might count a recommendation when the reviewer checks its evidence and accepts it as usable. Track posting, reduced delay, or financial benefit separately; accepting a draft cannot establish those later results.

Follow the work from the cases the system could handle, through actual use and acceptance, to any business improvement. Keep pending and unknown results in the report. Counting runs or logins cannot tell you how much useful work finished.

Four gates before the next investment

Before starting or continuing a pilot, answer each question in the table. Mark each gate pass, fail, or unknown. Keep the answers separate: a large projected saving cannot make up for an unauthorized action or a result nobody can check.

A failed or unknown gate gives you a specific next decision. Investigate the gap, narrow the proposal, pause it, or stop. You can fund that investigation while keeping the system out of live operation. Even four passes leave the usual security, release, and operating checks to complete.

1. Observe the real work

Put a recent case on the screen and follow it. Include something ordinary, something difficult, and something returned for correction. Ask the operator to show the mailbox, spreadsheet, and approval queue that the process diagram leaves out.

Record who decides, which sources they trust, and what happens when information is missing. Separate handling time from waiting. An invoice spending three days in an approval queue may gain little from a faster summary. A repeated chase for a missing field could be a submission-form problem.

Use a small sample you may inspect to find questions. Record what happened separately from what someone says usually happens. A workaround may preserve a necessary judgment or compensate for a broken handoff; check which before removing it. Keep the case history so someone else can challenge your interpretation.

2. Own the outcome

“Produce invoice recommendations” leaves too much open. Which invoices? What must the recommendation contain? Who checks it, and who decides whether it was useful? Write the result with the person responsible for the queue, including the record that will show it happened.

The sponsor funding the project may not control invoice approval, and the engineer may not own the queue. Name who owns each decision, including whether the service continues. Ask those people to confirm their responsibilities instead of inferring them from job titles.

For a review-only build, agree what evidence makes a recommendation usable and how the reviewer records acceptance or rejection. Leave record changes out until separately authorized. Two reviewers should be able to inspect the same case and agree what “done” means.

3. Bound the eligible work

Count only cases the system may and can handle. Exceptions covered by policy with current documents might qualify. Disputed invoices, unsupported currencies, and missing evidence might not. Write those exclusions before calculating use or acceptance rates.

Keep three counts: cases that qualify, cases that reach the system, and cases that pass review. Check for missing records and duplicates. Otherwise, good results on a small easy group can look like improvement across the department.

If you remove harder cases, the acceptance rate may rise without the system improving. Keep the previous eligibility rule and period. Save the query or event definition that selected the cases, with its source version, exclusions, and owner, so the change is visible.

4. Compare with what would happen otherwise

The counterfactual is what would have happened without your change. Start with comparable work in the current process: how long it takes, what goes wrong, and what it costs. Compare the proposal with a smaller repair or an existing product too.

A better month may reflect easier invoices or another person joining the team. Record those changes. When comparing models, hold the cases and surrounding system constant. When changing the whole process, choose a comparison that can test that wider change.

Agree when you’ll measure results. If you cannot establish that the AI change caused the difference, report what changed and what else might explain it. Use the uncertainty to choose the next comparison.

5. Name the verifier

Choose a check that can reject the result. A schema can establish that an amount is a number; it cannot establish that the amount matches the current invoice. That needs a separate comparison with the right source and revision.

For a record change, check the authoritative system after the action. For a review packet, give the reviewer the source evidence and a meaningful reject path.

If you use another model as a judge, test its agreement and known failures against independently established references. Using a second model does not by itself establish independence or correctness.

Ask who checks and maintains the test answers, and how they settle disagreements. Keep that answer key away from the system being tested. Count the cost of checking work too: a reviewer who must reconstruct every answer may save little time.

6. Engineer the workflow and adoption

Show where the operator will see and use the result. A recommendation in a separate chat can add copying and another check. Putting it in the existing queue may work better. Watch someone try it before assuming either design fits.

Follow the first place cases get stuck. People may lack access or see the recommendation too late. Reviewers may spend too long correcting it, or the queue may exceed their capacity. Find the cause before prescribing more training.

Use a short working session to resolve one current case, with the people who can fix its problem. Record who will act and by when. Plan for bursts of review work, absences, escalation, and a manual fallback. Office hours can help people participate, but they still need support when office hours are over.

7. Use the smallest sufficient intelligence

Break the workflow into decisions. Looking up an order can use a source query. Enforcing an approval threshold belongs in code. Estimating risk from structured history may suit statistics or ordinary machine learning. Allocating limited capacity under explicit constraints may suit an optimizer.

A language model may interpret a messy message or draft commentary from permitted evidence. An agent may help when its next search or tool call depends on what it finds. Keep a person responsible for decisions whose stakes or unclear standards require their judgment.

Compare the plausible options on the same cases. Include mistakes, response time, full cost, and what happens when each option fails. Keep calculations, permissions, and known rules in trusted code. A model explaining a policy cannot change it. Write why the chosen mix is enough and what would justify adding complexity.

8. Bound authority and loss

Write what this system may read or change, for whom, and why. Include sending information to providers and storing it in logs. Permission to read an invoice cannot automatically cover those other uses.

Have software enforce those limits when an action happens. Check current identity, access, policy, and any required approval for the exact proposal. If a request times out, ask the source system what happened before retrying a change it may already have applied.

Consider missed results and wrong answers as well as unauthorized actions. Several individually permitted actions can still combine into harm. Agree when to stop and how to recover. Mark remaining loss unknown if it has not been measured; the absence of a harm report cannot establish no additional harm.

9. Price the whole service

For the invoice assistant, the bill starts with building it and connecting it to the invoice records. Spread that initial cost over a stated period when comparing monthly costs. Then count what each month needs: models, tools, storage, source checks, and the people who review and support the work.

A rejected invoice recommendation has already used some of those resources. Correcting it or recovering from a failed attempt adds work too. Include those costs, along with maintenance changes and incidents, even when another team pays. Divide the full cost by accepted outcomes; excluding failed and escalated work from the bill would make each useful result look cheaper.

Work backward from the benefit to a maximum affordable cost. Test the assumptions most likely to exceed it. If required data repair or review makes the service too expensive, narrow the job or stop. Don’t remove a required control to make the spreadsheet balance.

10. Prove it on representative work

Use permitted cases from the work you intend to handle, with someone responsible for checking the expected answers. Include routine work, difficult cases, and costly failures. Try old or conflicting sources, unavailable services, changed permissions, duplicate events, and more review work than the average day.

Keep some real test cases out of development. Public teaching examples cannot play that role. Inspect errors in each important group, try outputs you know should fail, and repeat routes whose behavior varies. When a run fails, find the first wrong step before blaming later steps that merely used its answer. A perfect score on a small set cannot establish accuracy for the whole queue.

Before a trial, agree its duration, reporting cutoff, participants, and who decides what happens next. Check technical behavior, operator acceptance, use, business value, cost, and readiness separately. If you try a limited live release, monitor its limits and exercise the rollback path first.

11. Check what changed for the business

Ask where the time went. If the invoice team uses the freed hours to clear a backlog, check which additional invoices they finished and whether that work was needed. If paid overtime falls, check the spending records. Multiplying hours by a salary rate estimates what freed time is worth, but payroll may stay unchanged.

Report freed capacity, reduced spending, evidenced avoided costs, and extra accepted work separately. Don’t claim the same hour as both salary savings and extra output. Subtract new review and recovery effort, and count costs and remaining losses over the same period as the benefit.

Label forecasts, trial results, and observed operating value separately. Record the cases people didn’t use, changed, reworked, or left unresolved, along with incidents and successes. If the source, process, or mix of cases changes, keep the earlier measurement and reconsider the comparison. Don’t turn a short pilot into a year of savings without evidence for that extrapolation.

12. Expand, constrain, or retire from evidence

Name who decides whether the service continues and who can change, support, recover, or stop it. Have a backup practise those jobs. A runbook and training call cannot establish that the service will survive its builder leaving.

Review results with that owner: are people using the service, accepting its work, and getting enough value for the full cost? Check reliability, review workload, harm, and recovery too. Agree new scope and permissions before expanding. Narrow or pause when limits fail, and retire a service nobody can own or whose value no longer covers its cost.

A new model or another business unit needs a fresh decision. Preserve the tested release and rollback version. Check the receiving team’s sources, policies, and acceptance evidence rather than assuming the first team’s results transfer. Review each workflow separately so a good overall average cannot hide an unsafe or uneconomic one.

A hypothetical invoice decision

Suppose the brief promises automatic invoice correction and posting. A walkthrough reveals that a reviewer must approve each correction first. Keep the original brief and ask the responsible owner whether the team should instead prepare recommendations and stage corrections for review. Keep the manual queue running while they decide.

Investigating that smaller proposal does not give the team permission to post. Its review-only build would stop when a reviewer accepts a recommendation. The forecast below considers the eventual verified invoice resolution, which happens later. Keep those two counts separate.

The forecast gives about $4.17 per accepted outcome and $320 of room after the modeled cost. We haven’t established how the freed time will be used or assessed additional loss, so this is not cash savings or a complete net-value result. Reduce the net time gain to four minutes, keeping everything else fixed, and the balance becomes −$120. That is enough uncertainty to investigate before rollout.

All four gates remain unknown. Writing down a proposed result and checker has not shown that reviewers can use the recommendations or reject bad ones on this work. Passing local code tests would not answer that question. I’d defer the pilot or rollout and fund a small investigation of the source inputs, review time on comparable cases, and permission to save corrections for review. A later trial comparing recommendations alongside manual work would still need its gates resolved and approval, as would one that changes live records.

Ask who will provide each missing answer, by when, and at what cost. Compare the assistant with fixing the submission fields. If that repair removes enough of the problem, do the repair rather than force an AI build into the plan.

Bring one decision to the review

Take one decision to the workflow owner and the people who can answer the missing questions. Say whether you’re reviewing a forecast, a limited demonstration, or actual operation. Use the scorecard to give each gate its own status and name the source, responsible person, and missing evidence.

For each factor, distinguish undefined, declared, and demonstrated evidence: we haven’t defined it, we’ve written it down, or we’ve shown it works under stated conditions. You can mark those 0, 1, and 2, but don’t total them into a pass mark. Use them to find what needs checking next.

Finish with one next decision: who can make it, which assumption could change it, and when you’ll return with evidence. Keep earlier assessments when the evidence changes. The worksheet records that conversation; release and compliance approvals still follow their own processes.