Topic
Verification and operations
How will you know it worked?
A response can match its schema and still contain the wrong amount. A tool can time out after changing a record. Each needs a check against what happened in the source system.
The notes cover source checks, review failures, and recovery after a lost receipt. Preserve the operation identity, reconcile the effect, and keep unresolved work owned. Tests show the behavior they exercise; a live release needs evidence from its own environment.
Notes in this topic
Where does unfinished work live?
A request can outlive the chat, the worker, and an approval. Save what was requested, what happened, and who must follow up so another person can continue safely.
Read the noteRead-only can still expose the wrong data
Permission to fetch a record does not automatically include permission to send every field to a model.
Read the noteHow do you keep customers apart?
A service can check the right customer at sign-in and still return another customer's cached answer or let queued work outlive permission.
Read the noteAn override is a clue, not a new rule
A person changed the answer. Find out why before teaching the system to repeat it.
Read the noteWhat evidence earns the next round of funding?
The next useful investment may be a few hours to test an assumption, rather than another month building the demo.
Read the noteWhat should a tool check before changing a record?
An approval can be valid when it is given and unusable when the change arrives. The receiving service has to check again.
Read the noteThe SQL ran. Did it answer the right question?
A query can execute without errors and return the wrong total. Check what each row represents before accepting the number.
Read the noteYou changed one rule. What else changed?
A lower threshold sends more cases to review. Check what happens to the queue before calling the change an improvement.
Read the noteThe success rate went up. What else changed?
Before crediting an AI change for a better result, check whether it received easier work, different evidence, or more help from people.
Read the noteWhich cases belong in your evaluation set?
Five passing tests can hide a missing sixth. Look at the cases behind the score before deciding what the system can do.
Read the noteThe pilot works. Can anyone else run it?
Give someone other than the builder a small change and a failure to handle. Watch where they need help.
Read the noteDoes the model need to think, or find the evidence?
Before buying a stronger model, check what information it received. The answer may depend on a document search never found.
Read the noteBefore you retry, check what happened
A timed-out request may already have changed the record. Check what happened before sending it again.
Read the noteHow to test whether ordinary machine learning is enough
If the job is predicting late shipments from existing records, compare a rule and an ordinary predictive model before asking a language model to reason through every case.
Read the noteHow much can you afford to spend getting this right?
Count review, support, and maintenance before setting the model budget. A cheap call can leave the whole service too expensive to run.
Read the noteThe API returned 200. Did all the data arrive?
A successful request can leave out the order that would change your decision. Check which records should have arrived before using the extract.
Read the noteWhat counts as a successful result?
Count what happened: preparing a case, getting it accepted, and improving a delivery need different evidence.
Read the noteThe wrong amount passed validation
Change USD 120.50 to USD 120.00. The record still looks valid. A separate source check has to catch the error.
Read the noteDraw the system before choosing an agent
Follow one request through its records, decisions, approval, and confirmation. Then decide where a model belongs.
Read the note