In this note
The dashboard shows accepted results rising from 58% to 82%. Someone has already put the improvement into a presentation. Before changing the rollout, I'd ask which cases reached the service in each period.
If the second period contains more easy cases, a higher overall rate can hide unchanged performance on every kind of work. More reviewer help, a new eligibility rule, and missing outcomes can change the comparison too. Those explanations need checking before the team credits a model or workflow change.
Keep the counts behind the rate
Imagine a service collecting facts for a review team. Count a result as accepted only after a separate reviewer checks it against the same source and completeness rules in both periods. A model finishing its answer is not that event. Check what the dashboard counts before comparing percentages.
In this hypothetical example, we label cases easy or difficult before knowing their results. The first period has 20 easy cases and 80 difficult cases; the second reverses that mix. Each period has 100 eligible cases, all processed and checked by an independent reviewer by the reporting cutoff. Retries and corrections do not create extra cases.
| Case group | Baseline / candidate period |
|---|---|
| Easy cases | 18 accepted out of 20 / 72 accepted out of 80: 90% in both periods. |
| Difficult cases | 40 accepted out of 80 / 10 accepted out of 20: 50% in both periods. |
| All eligible cases | 58 accepted out of 100 / 82 accepted out of 100: 58% / 82%. |
| Known not accepted | 42 / 18. No unresolved or missing cases in this main example. |
The overall rate rises by 24 percentage points. But easy cases still succeed 90% of the time and difficult cases still succeed 50% of the time. The second period looks better because it contains more easy work. The arithmetic is correct; crediting a better model or workflow would not be. These are constructed counts, not an observed improvement.
Give both periods the same mix
Try asking what each rate would be with half easy and half difficult work. Both periods give 0.5 × 90% + 0.5 × 50% = 70%. Applying the same mix removes the apparent advantage. Choose and record a mix relevant to your decision before using this calculation to argue for a change.
That 70% is a calculated comparison, not another observed group or a forecast. Here, the two categories explain the headline difference. In real work, easy and difficult may hide other differences or contain too few examples to estimate a stable rate. Giving groups the same mix does not, by itself, prove what caused a change.
Check when the labels were assigned. Calling a case difficult because the service failed makes the result determine the category. Use facts available before the work began and save the labeling rule. Otherwise your breakdown can be as misleading as the overall rate.
Keep the real counts beside the calculated comparison. A shift toward easier demand still matters for running the service. If the team deliberately stopped processing difficult cases, record where they went, who now handles them, and what that costs. A smaller workload must not make the remaining business problem disappear from the report.
Make missing outcomes visible
Now try a separate example. Keep the second period's 82 accepted and 18 known not-accepted cases, but add 10 eligible cases still unresolved at the cutoff. There are 110 cases. Reporting 82% counts only the 100 with known outcomes. You can label it that way, but it is not the final success rate for all 110.
So far, 82/110 cases are confirmed accepted, about 74.5%. If all ten unresolved cases fail, that becomes the final rate. If all succeed, it becomes 92/110, about 83.6%. Those limits assume the same fixed group and a final yes-or-no result. They cannot tell you when the ten cases will finish or why they are unresolved.
The missing cases may be the slow, difficult ones. Don't assume they will behave like the completed easy cases. Keep their IDs, reasons for being unresolved, reporting cutoff, and eventual results. At the next review, follow those same cases rather than replace the old report with a cleaner-looking percentage.
Decide which change you are testing
Decide what you want to credit. To test a retrieval setting, keep the model, permitted cases, sources, checks, resource limits, and review policy unchanged where possible. Record the setting you changed. Upgrade the model and reviewer screen at the same time and you can no longer isolate retrieval's contribution.
You may instead want to test the whole changed process against a sensible alternative, including today's process. Use the same eligible work and count accepted results, customer or operator outcomes, waiting, checking, corrections, support, recovery, and full cost. A good result would support the combined change, not establish which component caused it.
With the right permissions and safe alternatives, you might randomly assign cases to the old and new process. Other settings may need a staged rollout with a credible comparison group, or comparison cases chosen to be similar on recorded characteristics such as the preassigned difficulty group. Ask people who understand the work, risk, and measurement to choose the approach. Matching those characteristics cannot remove a difference you never measured.
Count distinct cases rather than retries, and account for related tickets from the same incident. Before seeing the results, agree on the observation period, missing and delayed outcomes, how to estimate uncertainty, safety limits, and when to stop. Replaying saved cases can compare technical behavior, but it doesn't show whether people will use the changed process or whether it creates business value.
Ask for the counts behind the next claim
At your next review, pick one claimed improvement and ask for the counts behind it. Did the work get easier? Did source data, policy, or reviewer help change? Which cases are still unresolved? Use the worksheet to keep the comparison and unanswered questions together. Don't expand on the strength of a higher percentage if you cannot distinguish a better system from easier work.