Caisi
Notes
Notes on choosing, building, and checking AI-enabled work.
Each note takes one practical question, works through an example, and shows what the evidence supports. Teaching examples are labeled so you can judge how they apply to your work.
All notes
The 12 Factors of AI Value Engineering
How to decide what to build, whether it works, and whether it’s worth the cost.
Read the noteWhat changes for the customer?
A customer can have a ready account and still be unable to do the job they bought the software for. Follow that job before choosing an AI feature.
Read the noteWhere does unfinished work live?
A request can outlive the chat, the worker, and an approval. Save what was requested, what happened, and who must follow up so another person can continue safely.
Read the noteRead-only can still expose the wrong data
Permission to fetch a record does not automatically include permission to send every field to a model.
Read the noteHow do you keep customers apart?
A service can check the right customer at sign-in and still return another customer's cached answer or let queued work outlive permission.
Read the noteAn override is a clue, not a new rule
A person changed the answer. Find out why before teaching the system to repeat it.
Read the noteWhich parts should become shared capability?
Repeated work may justify shared code. First separate the reusable job from each team's data, policy, and responsibilities.
Read the noteWhat evidence earns the next round of funding?
The next useful investment may be a few hours to test an assumption, rather than another month building the demo.
Read the noteWhat should a tool check before changing a record?
An approval can be valid when it is given and unusable when the change arrives. The receiving service has to check again.
Read the noteThe SQL ran. Did it answer the right question?
A query can execute without errors and return the wrong total. Check what each row represents before accepting the number.
Read the noteDoes this need an agent or a fixed workflow?
A workflow can contain a model without handing it the choice of what happens next. Start by writing down which decisions really need that freedom.
Read the noteYou changed one rule. What else changed?
A lower threshold sends more cases to review. Check what happens to the queue before calling the change an improvement.
Read the noteThe success rate went up. What else changed?
Before crediting an AI change for a better result, check whether it received easier work, different evidence, or more help from people.
Read the noteWhich cases belong in your evaluation set?
Five passing tests can hide a missing sixth. Look at the cases behind the score before deciding what the system can do.
Read the noteThe cheaper model can cost more
A lower model bill can leave the team with more checking and repair. Count that work before choosing the model.
Read the noteBuild, buy, adapt, or leave it alone?
Your current software may do enough already. Check the difficult case before deciding what to buy or build.
Read the noteWhen the job needs a solver
A schedule can read well and still put one person in two places. When the rules are explicit, use code to find and check the possible assignments.
Read the noteThe pilot works. Can anyone else run it?
Give someone other than the builder a small change and a failure to handle. Watch where they need help.
Read the noteHuman review has a workload too
An approval step needs usable evidence and enough time to check it. Start by auditing one proposed result, then work out what happens when the queue grows.
Read the noteDoes the model need to think, or find the evidence?
Before buying a stronger model, check what information it received. The answer may depend on a document search never found.
Read the noteBefore you retry, check what happened
A timed-out request may already have changed the record. Check what happened before sending it again.
Read the noteWhere did the saved hours go?
Saving time doesn't always reduce spending. Check whether the hours went into more useful work, lower overtime, or free capacity, and count each hour once.
Read the noteHow to test whether ordinary machine learning is enough
If the job is predicting late shipments from existing records, compare a rule and an ordinary predictive model before asking a language model to reason through every case.
Read the noteThe work takes minutes. Why does it take a week?
A faster draft won't help much if the case still waits three days for a missing record. Find where the time goes before choosing what to automate.
Read the noteHow much can you afford to spend getting this right?
Count review, support, and maintenance before setting the model budget. A cheap call can leave the whole service too expensive to run.
Read the noteHow much data repair does this decision need?
Find the records needed for the job before starting a company-wide cleanup. A missing budget can block a finance draft; duplicate mailing addresses might not matter.
Read the noteWhy do people keep working around the agent?
People may avoid an agent because it adds copying, searching, or uncertainty. Watch someone try to finish the job before prescribing more training.
Read the noteWho needs to be in the room?
An engineer can build the draft, but who can check the figures, approve its use, and support it next month? Reserve those people's time before promising a pilot.
Read the noteWhich revenue definition should the agent use?
Two reports can both say revenue and mean different things. Settle the definition, scope, and unresolved facts before letting an agent answer.
Read the noteFind your first workflow by following recent cases
Start with recent work and the people who handle it. A small investigation can reveal a useful question without pretending to measure the whole company.
Read the noteThe API returned 200. Did all the data arrive?
A successful request can leave out the order that would change your decision. Check which records should have arrived before using the extract.
Read the noteThe brief is wrong. What do you do next?
Keep the original promise, show where it conflicts with the work, and ask what the team should build instead.
Read the noteWhat counts as a successful result?
Count what happened: preparing a case, getting it accepted, and improving a delivery need different evidence.
Read the noteWhich parts of this workflow need a model?
A model can read a vague request while code checks the rules. Split those jobs before choosing a technology.
Read the noteThe wrong amount passed validation
Change USD 120.50 to USD 120.00. The record still looks valid. A separate source check has to catch the error.
Read the noteDraw the system before choosing an agent
Follow one request through its records, decisions, approval, and confirmation. Then decide where a model belongs.
Read the noteIs this workflow worth automating?
A painful process can still be a poor investment. Follow the work, check the numbers, and find out what would have to be true for a change to pay off.
Read the note