Binary Classifiers as Dilations

Joint with Gabriel Ziegler.

Abstract: Performance of binary classifiers is often measured against a reference classifier when the true class is not readily observable. Reference classifiers are commonly also imperfect. A prominent example are diagnostic tests whose performance is measured against an established reference test. An imperfect reference generally leads to partial identification of relevant performance measures, which induces ambiguity in the interpretation of a predicted class. Seidenfeld and Wasserman (1993) define dilation as an extreme notion of non-informativeness when information is ambiguous. This paper characterizes precise conditions under which dilation arises in the context of binary classifiers when the reference classifier is imperfect and develops procedures for statistical inference based on methods for subvector inference in moment inequality models. We illustrate the usefulness of our approach through the application of two distinct case studies: radiologists' assessment of CT Chest scans for COVID-19 in Ai et al. (2020), and the utilization of artificial intelligence for the detection of COVID-19 in Mei et al. (2020).