Enriched designs for assessing predictive performance – analysis of bias and variance.
View Presentation View Presentation
Brandon D Gallas, FDA/CDRH/OSEL/DIAM  *Paul Pinsky, DIAM, OSEL, CDRH,FDA 

Keywords: bias,variance, AUC, sensitivity, IPW

In evaluating predictive performance of a new modality in a screening setting, a logistical constraint is that disases prevalence is typically very low, implying that, under a standard study design, large numbers of subjects have to be evaluated with the new modality. However, if a predicate modality exists in clinical practice, one can base study inclusion on the clinical results from the predicate in order to “enrich” the study population with diseased subjects. If this enrichment is not accounted for when estimating sensitivity, specificity and AUC, these “naïve” estimates may be substantially biased. When such estimates are “corrected” for the sampling weights using inverse probability weighting (IPW), however, the variances of the estimates of the above quantities are inflated. We analyze here the bias and variance associated with such enriched designs.