# Prediction comparison record

Use this to plan one repeated prediction and what a person or service will do with it. A model predicts risk; a cutoff rule decides which cases enter review; a person checks the case and may intervene. Later delivery outcomes need separate evidence. The completed demonstration below uses authored values; it establishes no model quality, calibration, customer result, or production approval.

## Start with these questions

1. Which shipments need a prediction, and when must it be available?
2. Which arrival and promised-time records will tell you whether each shipment was late?
3. Who checks those records and settles a disputed label?
4. How many review cases can the coordinators actually handle, and what happens to the rest?

Use the demonstration to see how the review cutoff changes the queue. Use the target record below to establish the sources, permissions, comparison, and limits for your own shipments.

## Completed hypothetical threshold demonstration

- Decision: route eligible shipments to a coordinator review queue. A score cannot authorize an intervention.
- Teaching route: review when `score >= threshold`.
- Cases: scores and labels are hand-declared, with no model training, real shipments, or interventions.

| Case | Score | Authored label |
| ---- | ----- | -------------- |
| S-1  | 0.95  | late           |
| S-2  | 0.85  | on time        |
| S-3  | 0.75  | late           |
| S-4  | 0.65  | on time        |
| S-5  | 0.55  | late           |
| S-6  | 0.45  | on time        |
| S-7  | 0.35  | late           |
| S-8  | 0.15  | on time        |

There are four late and four on-time teaching labels. A true positive is a late-labeled case selected for review; a false positive is an on-time-labeled case selected. A false negative is a late-labeled case outside review; a true negative is an on-time-labeled case outside review.

| Threshold     | Selected reviews | True positives | False positives | False negatives | True negatives |
| ------------- | ---------------- | -------------- | --------------- | --------------- | -------------- |
| At least 0.80 | 2                | 1              | 1               | 3               | 3              |
| At least 0.50 | 5                | 3              | 2               | 1               | 2              |

- Capacity assumption: three reviews for this batch, at three minutes each; total nine minutes.
- At 0.80: two reviews × three minutes = six minutes, within the teaching allowance.
- At 0.50: five reviews × three minutes = fifteen minutes, exceeding the teaching allowance.
- Disposition: neither threshold is approved. These counts illustrate a trade-off; the owner has not supplied real error costs, representative cases, or measured review capacity.
- Calibration: not evaluated. A score of 0.80 is not established as an 80% delay probability.
- Benefit: not evaluated. A selected review is not a useful intervention or avoided late delivery.

## Proposed real shipment comparison: prerequisites remain open

A hand-written departure-delay rule and a predictive model learned from historical records should receive the same shipment information available at the decision time. The ordinary predictive model is sufficient only if the independent test shows it meets the agreed error, timing, review-capacity, and full-cost limits. A rule may meet those requirements too. A language-model step needs its own demonstrated job, such as interpreting carrier text; a fluent explanation is not evidence of better predictions.

| Requirement             | Proposed work and unresolved evidence                                                                                                                                   |
| ----------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Decision and population | One owned carrier/route segment, assessed before promised arrival; source scope and eligibility need confirmation                                                       |
| Label                   | Authoritative observed arrival versus the applicable promised time; owner must decide revised promises, cancellations, missing events, corrections, and observation lag |
| Label authority         | Logistics owner plus independent approver; disagreements and adjudication retained; no approved target reference yet                                                    |
| Inputs                  | Only source signals available at decision time; feature revisions, timestamps, permissions, and lineage need inspection                                                 |
| Baselines               | Current triage and a policy-owned rule; a statistical/ML candidate where labels support it; language interpretation only if it solves a demonstrated input problem      |
| Holdout                 | Time-appropriate and entity-separated where needed; protect labels and freeze candidate/threshold versions before evaluation                                            |
| Direct evidence         | False positives/negatives by consequential slice, abstention, missing inputs, calibration where probabilities are claimed, latency, and uncertainty                     |
| Downstream evidence     | Review capacity, independently accepted coordinator work, interventions, and delivery outcomes kept as separate events                                                  |
| Cost                    | Preparation, inference, explanations, review, correction, support, recovery, and full cost per accepted result; no target measurements yet                              |
| Authority               | Deterministic policy owns route; explanation cannot change it; coordinator retains intervention authority                                                               |
| Operations              | Named data/model/policy/service owners, drift monitors, source freshness, manual fallback, rollback and retirement triggers; capacity not yet accepted                  |
| Current disposition     | Proposed trial only. No trained-model winner, production admission, calibrated probability, or avoided-loss claim                                                       |

Record interventions alongside outcomes. On-time delivery after a rescue is not evidence that the risk alert was a false positive about what would have happened without intervention. An observed outcome does not supply that counterfactual. Agree on a suitable comparison before claiming causal benefit.

## Blank target record

### Decision and reference

- What repeated decision will the prediction support? What may the service do, and what remains outside its permission?
- Which shipments qualify, which are excluded, and at what point is each prediction made?
- Which source records may supply the inputs?
- What exactly counts as late? Which promised time and arrival event apply?
- Who owns that definition? Who separately checks and approves the expected labels, and who settles a disagreement?
- Which source revisions and observation window support each label? How will delayed, missing, and corrected events remain visible?
- Where will you record interventions separately from delivery outcomes?
- What result must someone accept, and who can independently reject it?

### Controlled comparison

Choose and tune candidates on development cases. Keep a separate set untouched until you test the chosen version; that is the holdout. Use later shipments and separate related records where needed so the test does not reveal answers seen during development.

- How does today's triage work, and which simple rule will you compare with it?
- Which prediction methods will you compare? Why is each included or excluded?
- Which access is permitted, which fields may each method use, where may they be sent, and how long may they be kept?
- Which input fields were genuinely available when the decision would be made, and which could reveal an outcome learned later?
- Which cases are for development, and which later or unrelated cases remain protected for the independent test?
- Which model, input preparation, probability adjustment if any, and queue-policy versions are frozen for that test?
- Which cutoffs were chosen using development cases? Who approved the costs assigned to missed late shipments and unnecessary reviews?
- For important carriers, routes, shipment values, and data ages, how many unnecessary reviews and missed late shipments occur?
- Which cases receive no prediction, and how uncertain are the results? If you claim probabilities, do they match observed rates?
- How many reviews can the team finish in time? What happens when a review is late or the queue exceeds capacity?
- What is the full cost per accepted result, and how quickly must the prediction arrive?
- What limits must the prediction meet, and what separate evidence shows the coordinator can use its review cases?

### Decision and operation

- What did the tests establish, and what remains unknown?
- Which candidate, if any, is supported? Should the work instead be deferred, constrained, or stopped?
- Who may approve that decision, for which scope and limitations, and when will it be reviewed?
- Who owns the data, model, routing policy, and service? What time and backup have they accepted for operating it?
- What change in inputs or results would trigger investigation, rollback, the manual fallback, or retirement?

Do not select a candidate using a vendor benchmark, a fluent explanation, or this eight-case demonstration. Promotion needs evidence from the permitted target workload and the complete downstream review path.