In this note
A dispatch team wants to know which shipments may arrive late. The inputs are structured carrier events, departure delay, promised arrival time, and current shipment state. Before asking a language model to reason about every shipment, I’d compare the existing triage process, a simple rule, and an ordinary predictive model. The rule uses a cutoff someone writes; the predictive model learns patterns from historical shipment records with checked late/on-time outcomes.
Start with the decision the prediction will support. Here it sends qualifying shipments to a coordinator's review queue. A person decides whether to intervene. A useful prediction might help that person, but a queue entry doesn't establish that a delivery improved.
Specify what the score predicts
Choose which shipments to assess and when. For example, use one carrier's in-transit shipments before their promised arrival. Define late from the authoritative arrival record and that promise. Agree how to handle cancellations, changed promises, missing arrivals, and time zones, including when a result is final enough to check.
Use only information available when the prediction would be made. A field calculated after delivery, or after a coordinator acted, can leak the answer into the inputs. Keep the source revision and availability time for each field. Have the logistics owner and an independent reviewer check late/on-time labels and resolve disputed cases.
Reserve cases the team won't use to build or tune the candidates. This is the holdout. Later shipments can test how a model performs after its training period. Prevent related shipments or repeated observations from leaking between sets where that would reveal answers. Choose fields and cutoffs on development data, then freeze them before testing the holdout. Once you tune on a holdout, it is no longer an independent test.
Give each candidate the same job
Start with today's triage and a simple rule, such as a departure-delay cutoff approved by the policy owner. Test a predictive model if you have permitted historical records with reliable labels. A language model may help when carrier messages need interpretation. Test it for that job if appropriate, but don't send data to a route that violates access, destination, or timing requirements. You don't have to test every model category.
Give candidates the same shipments, information available at the same time, checked labels, and review task. Record cases they can't answer or lack data for. A convincing explanation won't compensate for more consequential misses. A cheap model can also become expensive if it sends too many unnecessary reviews to coordinators.
Changing the threshold changes the work queue
| Case | Score and authored outcome label |
|---|---|
| S-1 | 0.95 · late |
| S-2 | 0.85 · on time |
| S-3 | 0.75 · late |
| S-4 | 0.65 · on time |
| S-5 | 0.55 · late |
| S-6 | 0.45 · on time |
| S-7 | 0.35 · late |
| S-8 | 0.15 · on time |
These eight scores and labels are hand-declared teaching inputs; no model was trained or accuracy measured. Send a case to review when its score reaches the cutoff, or threshold. Four labels are late. At 0.80, the queue gets one late case and one on-time case, missing three late ones. At 0.50, it gets three late and two on-time cases, missing one late case.
| Threshold | Counts from the declared scores and labels |
|---|---|
| At least 0.80 | Review 2: 1 true positive, 1 false positive; outside review: 3 false negatives, 3 true negatives |
| At least 0.50 | Review 5: 3 true positives, 2 false positives; outside review: 1 false negative, 2 true negatives |
A true positive is a late-labeled case sent to review. A false positive sends an on-time case; a false negative misses a late case. These counts compare queue routing with the teaching labels. They don't show rescued shipments or actual losses avoided.
Suppose the team has nine minutes for three reviews at three minutes each. The higher cutoff needs six minutes. The lower needs fifteen. Catching more late-labeled cases would exceed the available review time. Neither cutoff is approved here: the owner needs to weigh the consequences of missed cases, check capacity, and agree what to do with overflow.
The example calculates queues from invented scores; it doesn't establish which model or cutoff performs best. For a real comparison, choose cutoffs using development cases and agreed error costs before testing the holdout. Don't quietly raise the cutoff during a busy shift. That changes which risky cases get attention and needs an agreed policy.
A score of 0.80 needs its own interpretation check
A score can rank risk without representing a probability. If you tell a coordinator that 0.80 means an 80% chance of delay, test that claim. Do similar shipments scored around 0.80 arrive late roughly 80% of the time? That is calibration. It needs enough representative cases to assess uncertainty; eight invented scores can't establish it, and changing the queue cutoff doesn't fix it.
Check errors by carrier, route, shipment value, and data age. Find out whether a miss came from the predictor, an old input, a bad label, or the routing rule. A later reviewer should be able to reconstruct the score and why it entered the queue. Keep the model version, how its inputs were prepared, any adjustment used to turn scores into probabilities, and the cutoff policy with the responsible owners. Those records also let the team return to an earlier version.
Keep cases with unresolved labels in the trial record and explain how they affect the reported rate. A carrier that rarely supplies reliable arrival events may look accurate only because you can check its easiest shipments. Show that gap and decide whether the service can safely cover that carrier.
Record what coordinators did as well as when shipments arrived. If someone rescued an at-risk shipment, its on-time arrival doesn't prove the alert was wrong. You don't know how it would have arrived without help. Agree on a suitable comparison before claiming the system prevented delays or losses; checking arrival predictions alone won't establish that benefit.
Select against the full operating cost
Count preparing inputs, running the model, generating optional explanations, review, corrections, recovery, and support. Measure how long it takes the coordinator to get a usable result, not only the model's response time. An explanation must not change the fixed routing rule or grant permission to intervene. If it fails, keep review available with a checked evidence summary.
An ordinary predictive model is enough for this job if the independent test shows it meets the owner's error limits, gets useful results to coordinators in time, fits the review capacity, and stays within the full-cost limit. A simple rule that meets the same requirements may be enough too. Add a language-model step only for a demonstrated need, such as interpreting carrier messages, and test whether it improves that part of the work without breaking the others.
The worksheet supplies the teaching counts and a proposed trial plan, not that selection evidence. You still need permitted data, checked labels, a protected holdout, actual costs, agreed cutoffs, and acceptance evidence. Assign someone to monitor changing performance and run the fallback. Old inputs, missing labels, an overloaded queue, or worsening errors may mean narrowing coverage, returning to manual triage, or stopping the candidate.
For your next prediction task, write down what the score predicts and what a person will do with it. Add a simple baseline, the cost of each kind of mistake, and the reviews the team can handle. Obtain permitted cases and agree on the independent test before selecting a model.