Offline metrics such as AUC describe how well a model separated cases in historical data. They do not answer whether it still works after go-live.
We recommend tracking three families of metrics together: discrimination, calibration, and stability across sites and population subgroups.
A periodic review cadence matters as well. When referral patterns or coding habits shift, calibration is usually the first indicator to drift.
The accuracy of a risk prediction depends first on data quality, not model complexity. Missing encounters, inconsistent lab units and duplicate registrations across systems are all amplified in the model output. In practice we start with a completeness assessment of the key fields: which measures have the highest missing rate, and whether missingness concentrates in…
Structured fields support stable quantitative calculation, while narrative notes and follow-up records often contain signals that appear earlier. In chronic disease the two are complementary: lab trends provide an objective baseline, and narrative text hints at adherence and changes in circumstances. Aligning them requires care with timing. Anchoring on the encounter event rather than the…