In this note
The system created a review case. The dashboard counted a success. Nobody has looked at the case yet, and the shipment is still on the road. That green tick needs a narrower label.
The software prepared something for review. A coordinator still needs to check it, decide what to do, and perhaps intervene. Whether the delivery improves is another question again. I’d put separate labels on those events before agreeing on a success rate.
Choose a unit someone else can reject
For this workflow, count one accepted result when a coordinator checks a correctly routed case, finds it usable, and records their decision. They may decide no intervention is needed. That can still be good prepared work. If they reject the case because its route or evidence is wrong, keep it out of the accepted count.
Agree why a reviewer would reject a case. Perhaps it has old evidence, a missing record, the wrong route, or an explanation they cannot use. Give the coordinator a way to return it or mark it unresolved. Opening the case or clicking a generic approval button cannot establish that they checked it.
The check could be a responsible reviewer, a comparison with the official record, or code applying an owner-approved rule. Whoever produces the result must not be able to approve their own unsupported answer. A second model is no help if it repeats the same mistake from the same weak evidence.
Follow the events without merging them
- Eligible shipmentThe shipment belongs to the group and reporting period the owner has agreed to follow.
- Workflow processes itSoftware checks the evidence and score, then applies the routing policy or stops the run.
- Review case stagedThe local implementation ends here when review is required. No coordinator decision is recorded.
- Coordinator records a decisionProposed later event: the coordinator records what to do about the shipment and separately accepts or rejects the prepared case.
- Intervention confirmedProposed later event: a permitted action is taken, and a check of the shipment system confirms it happened.
- Delivery observedProposed later event: the delivery system records what happened. At the agreed reporting cutoff, missing results stay pending.
Some shipments should pass without review. Count those separately from cases the software could not process. A dashboard showing only prepared cases cannot tell you whether routine shipments were correctly skipped or never reached the system.
The local program behind this example checks evidence, uses a supplied test score, and stops at “review ready.” It records none of the later coordinator or delivery events. This page supplies a measurement worksheet, not the program. Before building this workflow for a real queue, check routing on representative cases with independently confirmed answers. The local tests establish neither trained predictive accuracy nor improved delivery.
Keep the denominator beside the percentage
Suppose we’re reporting on 100 unique eligible shipments. Eighty reach the workflow: 30 pass without review, 40 produce review cases, and 10 stop or escalate. We choose a reporting cutoff: the date and time at which we take these counts, even if some work is still pending. By then, coordinators have recorded decisions on 32 of the 40 review cases. These counts are hypothetical.
| Measure and denominator | Count and interpretation |
|---|---|
| Exposure / all eligible shipments | 80 / 100 = 80%; 20 eligible shipments did not reach this workflow |
| Completed routing / exposed shipments | 70 / 80 = 87.5%; 30 no-review routes plus 40 staged cases; correctness still needs verification |
| Accepted review work / staged cases | 28 / 40 = 70%; four recorded reviews rejected the prepared work and eight cases remain pending |
| Accepted review work / cases with recorded decisions | 28 / 32 = 87.5%; the same accepted count, excluding pending cases from this denominator |
| Confirmed intervention / accepted review work | 12 / 28; an intervention count, not a delivery-improvement rate |
Both acceptance rates describe the same 28 good cases. The 87.5% figure counts only cases reviewed so far; the 70% figure includes the eight still waiting. Show both labels and the pending count. A waiting case, a rejected case, and a shipment the workflow never saw need different follow-up.
Count unique shipments and cases so retries and repeated notifications don’t inflate the totals. Keep the rule that selected these 100 shipments. If eligibility changes halfway through the period, report the affected groups separately rather than silently changing who the percentage covers.
Give slow outcomes time to arrive
Suppose delivery records are available for 60 of the 100 shipments at the cutoff, with 40 still pending. That is 60 observed results, not a 60% delivery success rate. To judge success, agree what a good delivery means: for example, arrival against the recorded promised date, with the owner deciding how cancellations and exceptions count.
Set the reporting windows with the owner before examining results. A review may finish before the shipment arrives, so the windows can differ. When a late delivery record appears, attach it to the original shipment, case, and system version. Correct the dated report instead of crediting a newer model for older work.
Find out why a result is missing. It may be too early, the source may be unavailable, or someone may not have recorded a decision. Keep it unknown while checking. You can still stop new work for a safety breach or cost limit before the reporting window closes; make sure existing cases are reconciled and supported.
Check whether accepted work helped
A coordinator can accept a case and decide no intervention is useful. A confirmed intervention can still arrive too late. Even an on-time delivery may have happened without the new workflow. The accepted unit makes the work countable, but it does not supply the comparison needed to attribute benefit.
Compare similar shipments with the agreed baseline or a suitable comparison group. Check whether the carrier, shipment mix, staffing, demand, or policy changed too. Include missed cases, unnecessary reviews, delay, corrections, cost, and harm. Inspect difficult groups separately so a better average cannot hide their worse results.
When calculating cost per accepted review result, include what you spent on rejected and pending cases too. Add the agreed share of delivery, review, support, and recovery costs. You may also measure the cost per independently checked routing decision, but give it a different label. The two numbers answer different questions.
If no accepted results exist, report the cost and zero accepted count. Do not report a zero cost per accepted result or quietly switch to completed calls as the denominator. If the verifier or observation source is missing, the value claim remains unproved.
Agree on the event map before the dashboard
Choose a result you currently call successful. Follow one case from arrival through preparation, review, and any later business result. For each step, identify the record that shows it happened and the person who owns that record. The worksheet helps record the IDs, query versions, and reporting windows needed to count those events without duplication.
Walk through a permitted case with the operator and the person responsible for the metric. Ask them to show why it counts as accepted, and where a pending or rejected case goes. If the coordinator’s acceptance has no record, plan how to capture it before reporting accepted cases. Apply the same check to each later event you intend to count.