In a factory, a model that predicts “machine fine” every hour can look excellent. In a simulation for our episode on AI in industry, a model that always said “fine” was 99.2% accurate hour by hour and warned of none of 60 breakdowns. Accuracy measured over the easy cases says almost nothing about the rare one that matters.
Different tools catch different failures
On the same simulated data, a trained classifier caught bearing faults in 23 of 24 cases but overheating in only 2 of 36, with about 9.5 false alarms per machine per year. An anomaly detector trained only on normal running caught all 36 overheating cases and 10 of 24 bearing faults, with no false alarms. Neither is “the best model”; the right choice depends on which failure you cannot afford to miss and how many false alarms the team will tolerate. We found and fixed a bug in the first round, and we kept both rounds in the episode.
Hospitals, factories and banks ask different questions
- Medicine works to plans fixed in advance (ICH E9) and to performance criteria set before testing. Meeting a pass mark is not a race and does not show that one system is better than another.
- Banks work under model-risk guidance. In the US, SR 26-2 (April 2026) replaced SR 11-7, covering validation, effective challenge and model inventories; it is guidance, and it excludes generative and agentic AI. Lending decisions also need clear reasons when credit is refused.
- Factories want warnings that operators act on. In our own warehouse work, loading-area procedure checks run without face recognition, and in a carbon-compliance project the rulebook is kept separate from the data, so a rule change never silently changes the numbers.
We are clear about our limits: we have no hospital-AI or bank model-risk project experience, and the episode says so. What carries across every sector is the habit: decide what failure costs before you choose a model, and measure on the cases that matter.