Veridical Data Science and PCS Uncertainty Quantification

Data Science is central to Al and has driven most of the recent advances in biomedicine and beyond. Human judgment calls are ubiquitous at every step of the data science life cycle (DSLC): problem formulation, data cleaning, EDA, modeling, and reporting. Auch judgment calls are often responsible for the “dangers” of Al by creating a universe of hidden uncertainties well beyond sample-to-sample uncertainty. To mitigate these dangers, veridical (truthful) data science is introduced based on three key principles: Predictability, Computability and Stability (PCS). The PCS framework (including documentation) unifies, streamlines, and expands on the ideas and best practices of statistics and machine learning.

This talk showcases PCS through a collaborative project on finding genetic drivers of a heart disease and through uncertainty quantification (PCS-UQ) with applications to prediction including deep learning. It compares PCS-UQ and makes connections with Conformal Prediction (CP). Over a collection of 17 regression tabular datasets, 6 multi-class tabular datasets, and 4 deep learning datasets, PCS-UQ reduces the size of the prediction intervals or sets by around 20% on average when compared to the best CP method among the ones used by PCS-UQ, and has better subgroup coverages than CP overall.