In this note
An automation sends an authorized invoice correction. The target applies it, but the response never reaches the caller. From the caller's side, that can look much like a request that never arrived. Sending another correction immediately could create a second effect.
Ask the target system whether that specific change happened. A model's account of the attempt or an old approval message can't confirm a saved correction. The software and recovery operator need the target's record before deciding what to do next.
This hypothetical walkthrough reports a local simulation; the downloadable worksheet is a paper record, not the simulator. Reloading its stored records represents a restart, not a real process crash. Its recovery function checks and escalates pending work without automatically sending it again. It establishes no payment-integration, production-storage, or exactly-once-delivery guarantee.
Keep the business operation separate from the attempt
Give the intended change one stable operation key. Here it identifies tenant-a, invoice inv-100, resolution resolve-inv-100-demo, and the action to commit it. Keep that identity when retrying or restarting a worker. If each attempt gets a new key, the target may treat it as a new correction and apply it twice.
Keep two kinds of record. The operation record describes the intended invoice correction, its scope, the decision supporting it, and the approval and policy revisions checked before submission. Each attempt records a particular try at sending that same correction. A restart may create another attempt, but it must not silently turn the correction into a new business operation. These records show what you intended and tried to send, not what the target changed.
If the local record is missing, don't assume the request was never sent. Another component may have sent it before the record was saved or lost. The simulation reports no-record and makes no change. In a real service, the recovery owner must investigate the target's permitted records before allowing another attempt.
Check how the target prevents duplicates and lets you look up an operation. Which tenant and action does its key cover? How long does it keep the key? What if the same key arrives with different data? Logging a key locally doesn't force an external service to reject duplicates. Test the actual target before relying on that behavior.
Follow the effect past the missing response
- Record intentRetain the stable operation identity and checked authority; local status is pending.
- Effect occursThe simulator stores one committed effect for that identity.
- Receipt is lostA simulated interruption occurs before the caller records completion.
- Reload and reconcileReload local state, read back the matching simulated effect, and create no new effect.
After the interruption, the caller still says pending, but the simulator has already saved the correction. Recovery reloads both records, matches the saved change to the tenant and invoice, and reconciles the caller's status. There is one stored change before recovery and one afterward. Recovery creates no new change.
The replay shows the decision you want after a lost response: confirm the existing change without repeating it. The temporary JSON records are enough to teach that decision, not to establish reliable production storage.
In a real service, read back the result through an authorized target lookup. Match the operation key, tenant, invoice, action, and final state. An invoice that looks plausible may reflect another change. If the lookup can't distinguish this operation, leave it unresolved for reconciliation rather than guessing.
Use a decision table, not a default retry
| Evidence available | Disposition and effect boundary |
|---|---|
| Matching effect exists and readback is available | Reconcile the existing effect. Create no new effect; retain the receipt and source evidence. |
| Simulator has no effect; approval or policy revision changed | Escalate and hold. The earlier authority cannot authorize a new effect. |
| Simulator has no effect; authority still matches | Escalate for an operator decision. The teaching recovery function still does not automatically resubmit. |
| Readback is unavailable or cannot establish the result | Keep reconciliation pending and escalate. Missing evidence is not evidence of no effect. |
| Caller has wrong tenant or lacks the required scope | Deny the request. Do not create an effect or use recovery to cross the boundary. |
An empty lookup doesn't always mean nothing happened. The target may still be processing the request, its replica may lag, or the lookup may use the wrong identifier. Check how the target reports a final result. Until you have enough evidence, hold the operation with a named recovery owner.
Even if no change occurred, the old approval may have expired or the policy may have changed. In one simulation branch, recovery finds a changed policy and escalates. In the unchanged-policy branch, it also asks the operator what to do. Neither branch automatically retries because an earlier plan said to continue.
Recheck authority before any new effect
Permission to look up a saved correction doesn't grant permission to make another one. Before any new write, the trusted service must recheck the caller, tenant, allowed action, current policy, and any required fresh approval. An old trace or prompt cannot replace those checks.
Record what the operator found, what remains unknown, and the permitted next step. Keep the original operation and attempts if new approval is needed. Follow the target's rules for duplicates and changed requests. Don't invent a new key just to make a submission succeed; that could hide the earlier change and repeat it.
Assign pending work to a service owner with a next review time. Keep cancellation, expiry, and intake limits in force while they investigate. Restarting a worker or model run must not turn a held case into permission to try again. The receiving service needs access to the unresolved record.
When handing the case to another team, send the operation key, target reference, attempts, approval and policy revisions, and last verified state. A message saying retry failed makes the next operator reconstruct the uncertainty from scratch.
Test the failure at the target boundary
Test the real target with a duplicate request, a stop before the write, and a stop after the write but before the response. Include changed approval, changed policy, unavailable lookup, and the wrong tenant or permission. Check the number of actual changes and the final record through the target's authorized interface, not only the client's success response.
Two workers can both see a pending operation and try to handle it. Test that contention at the target as well as an orderly single-worker replay. Also test out-of-order attempts, missing or corrupted records, backup restoration, and retention. Agree how much data loss and recovery time the service can tolerate. The local simulation cannot establish those storage or coordination guarantees; don't copy its file-writing approach into a production service.
Use the worksheet for one ambiguous write. Record its operation key, what the target can confirm, and who decides next. If the result is still unknown, keep it pending. Don't send another write merely to get a response you can mark complete.