Binary Classifier Performance Metrics
Hard Classification
When hard classification are made, (after converting the probabilities using threshold or returned by the algorithm), there can be four cases for a certain observation:
The response actually negative, the algorithm predicts it to be negative. This is known as true negative (TN).
The response actually negative, the algorithm predicts it to be positive. This is known as false positive (FP).
The response actually positive, the algorithm predicts it to be positive. This is known as true positive (TP).
The response actually positive, the algorithm predicts it to be negative. This is known as false negative (FN).
All the observations fall into one of the four categories stated above and form a confusion matrix.
| Predicted Negative (0) | Predicted Positive (1) | |
|---|---|---|
| Actual Negative (0) | True Negative (TN) | False Positive (FP) |
| Actual Positive (1) | False Negative (FN) | True Positive (TP) |
Following are some popular performance metrics, when observations are hard classified:
Misclassification: Misclassification rate, or error rate is the most common metric used to quantify a binary classifier. This is the probability that the classifier makes a wrong prediction, which can be expressed as: \[ Misclassification\ rate=Pr(\hat{Y}\neq Y)=\frac{FN+FP}{TN+FN+TP+FP} \]
Accuracy: This simply accounts for the number of correct classifications made. \[ Accuracy = Pr(\hat{Y}=Y)=1-Misclassification\ rate \]
Sensitivity: Sensitivity measures the proportion of the positive responses that are correctly identified as positive by the classifier (Altman and Bland 1994a). In other words, it is the true positive rate and can be calculated directly from the entries of confusion matrix. \[ Sensitivity=Pr(\hat{Y}=1|Y=1)=\frac{TP}{TP+FN} \] Other terms used to represent the same metric are true positive rate (TPR), recall. The term sensitivity is popular in medical test (Altman and Bland 1994b). In the credit world the use of the term TPR has been noticed (Siddiqi 2012). While in machine learning, natural language processing, use of recall is common Huang and Efthimiadis (2009)
Specificity: Specificity measures the proportion of the negative responses that are correctly identified as negative by the classifier (Altman and Bland 1994a). In other words, it is the true negative rate and can be calculated directly from the entries of confusion matrix. \[ Specificity=Pr(\hat{Y}=0|Y=0)=\frac{TN}{TN+FP} \]
Specificity is also known as true negative rate (TNR).
Positive predictive value (PPV): Positive predictive value (PPV) is the probability that an observation classified as positive is truly positive. It can be calculated from the entries of confusion matrix: \[ PPV=Pr(Y=1|\hat{Y}=1)=\frac{TP}{TP+FP} \]
Negative predictive value (NPV): Negative predictive value (NPV) is the probability that an observation classified as negative is truly negative. It can be calculated from the entries of confusion matrix. \[ NPV=Pr(Y=0|\hat{Y}=0)=\frac{TN}{TN+FN} \]
Diagnostic likelihood ratio (DLR): Likelihood ratio is another form of accuracy measure of a binary classifier. This is a ratio. From true statistical sense, this ratio is a likelihood ratio, but in the context of accuracy measure, this is called diagnostic likelihood ratio (DLR) (Pepe 2003). There are two kinds of DLR metrics are defined:
\[Positive\ DLR=\frac{TPR}{FPR}\] \[Negative\ DLR=\frac{TNR}{FNR}\]
- F-Score: F-Score (also known as F-measure, F1 -Score) is another metric which is used to assess the performance of a binary classifier. It is often used in information theory, to assess search, document classification, and query classification performance (Beitzel et al. 2007). It is defines as the harmonic mean of precision (Positive predictive value, PPV) and recall (True positive rate, TPR).
\[ F\text{-}Score=\frac{2}{\frac{1}{PPV} +\frac{1}{TPR}}=2\times \frac{PPV\times TPR}{PPV+TPR} \]
Observation Are Scored
Rather than making simple classification, often models give probability scores, \(Pr(Y=1)\). Using certain cutoff or threshold values, we can dichotomize the scores and calculate these metrics. This is also true when some certain diagnostic variable is used to categorize the observations. For example, having a hemoglobin A1c level of lower than 6.5% being treated as no diabetes, and having a level equal to greater than 6.5% being treated as having the disease. Here the diagnostic measure is not bound in between 0 and 1 like the probability measure, yet all the metrics stated above can be derived. But these metrics give a sense of performance measure only at certain threshold. There are metrics, that measure overall performance of the binary classifier considering the performance at all possible thresholds. Two such metrics are
- Area under receiver operating characteristic (ROC) curve
- KS statistic
Receiver operating characteristic (ROC) curve Bewick, Cheek, and Ball (2004) is a simple yet powerful tool used to evaluate a binary classifier quantitatively. The most common quantitative measure is the area under the curve (Hanley and McNeil 1982). ROC curve is drawn by plotting the sensitivity (TPR) along \(Y\) axis and corresponding 1-specificity (FPR) along \(X\) axis for all possible cutoff values. Mathematically, it is the set of all ordered pairs \((FPR(c), TPR(c))\), where \(c\in R\).
Some Properties of ROC curve
ROC curve is a monotonically increasing function, defined in the \((+,+)\) quadrant.
ROC curve is that it is invariant of strictly increasing transformation of the diagnostic variable, when estimated empirically.
The ROC curve always contains \((0,0)\) and \((1,1)\). These are the extreme points when the threshold is set to \(+\infty\) and \(-\infty\).
If the diagnostic variable is unrelated with the binary outcome, the expected ROC curve is simply the \(y=x\) line. In a situation where the diagnostic variable can perfectly separate the two classes, the ROC curve consists of a vertical line (\(x=0\)) and a horizontal line (\(y=1\)). For a practical data, usually the ROC stays in between these two extreme scenarios. Figure below illustrates some examples of different types of ROC curves. The red and the green curves illustrate two extreme scenarios. The random line in red is the expected ROC curve when the diagnostic variable does not have any predictive power. When the observations are perfectly separable, the ROC curve consists of one horizontal and a vertical line as shown in green. The other curves are the result of typical practical data. When the curve shifts more to the north-west, it means better the predictive power.
For more details, see Pepe (2003).
Common approaches to estimate ROC curve
- Empirical: The empirical method simply constructs the ROC curve empirically, applying the definitions of TPR and FPR to the observed data. Figure 1 is an example of such approach. For every possible cutoff value c, TPR and FPR are estimated by:
\[ \hat{TPR}(c)=\sum_{i=1}^{n_Y}I(D_{Y_i}\geq c)/n_Y \]
\[ \hat{FPR}(c)=\sum_{j=1}^{n_{\bar{Y}}}I(D_{{\bar{Y}}_j}\geq c)/n_{\bar{Y}} \] where, \(Y\) and \(\bar{Y}\) represent the positive and negative responses, \(n_Y\) and \(n_{\bar{Y}}\) are the total number of positive and negative responses, \(D_Y\) and \(D_{\bar{Y}}\) are the distributions of the diagnostic variable in the positive and the negative responses. The indicator function has the usual meaning. It evaluates 1 if the expression is true, and 0 otherwise. The area under empirically estimated ROC curve is given by:
\[ \hat{AUC}=\frac{1}{n_Yn_{\bar{Y}}} \sum_{i=1}^{n_Y}\sum_{j=1}^{n_{\bar{Y}}} (I(D_{Y_i}>D_{Y_j})+ \frac{1}{2}I(D_{Y_i}>D_{Y_j})) \] The variance of AUC can be estimated as (Hanley and McNeil 1982): \[ V(AUC)=\frac{1}{n_Yn_{\bar{Y}}}( AUC(1-AUC) + (n_Y-1)(Q_1-AUC^2) + (n_{\bar{Y}}-1)(Q_2-AUC^2) ) \] where, \(Q_1=\frac{AUC}{2-AUC}\), and \(Q_2=\frac{2\times AUC^2}{1+AUC}\).
An alternate formula is developed by DeLong, DeLong, and Clarke-Pearson (1988) which is given in terms of survivor functions: \[ V(AUC)=\frac{V(S_{D_{\bar{Y}}}(D_Y))}{n_Y} +\frac{V(S_{D_Y}(D_{\bar{Y}}))}{n_{\bar{Y}}} \]
A confidence band can be computed using the usual approach of normal assumption. For example, a \((1-\alpha)\times 100\%\) confidence band can be constructed using:
\[ AUC\pm\phi^{-1}(1-\alpha/2)\sqrt{V(AUC)} \]
The above formula does not put any restriction on the computed values of upper and lower bound. However, AUC is a measure bounded between 0 and 1. One systematic way to do this is the logit transformation (Pepe 2003). Instead of constructing the interval directly for the AUC, an interval in the logit scale is first constructed using:
\[ L_{AUC}\pm \phi^{-1}(1-\alpha/2)\frac{\sqrt{AUC}}{AUC(1-AUC)} \]
where \(L_{AUC}=log(\frac{AUC}{1-AUC})\) is the logit of AUC. The logit scale intervals can then be inverse logit transformed to find the actual bounds of AUC.
Confidence interval of ROC curve: For large values of \(n_Y\) and \(n_{\bar{Y}}\), the distribution of \(TPR(c)\) at \(FPR(c)\) can be approximated as a normal distribution with following mean and variance:
\[ \mu_{TPR(c)}=\sum_{i=1}^{n_Y}I(D_{Y_i}\geq c)/n_Y \]
\[ V \Big( TPR(c) \Big)= \frac{ TPR(c) \Big( 1- TPR(c)\Big) }{n_Y} + \bigg( \frac{g(c^*)}{f(c^*) } \bigg)^2\times K \] where, \[ K=\frac{ FPR(c) \Big(1-FPR(c)\Big)}{n_{\bar{Y}} } \]
\[ c^*=S^{-1}_{D_{\bar{ Y}}}\Big( FPR(c) \Big) \] and, \(S\) is the survival function given by, \[ S(t)=P\Big(T>t\Big)=\int_t^{\infty}f_T(t)dt=1-F(t) \] For details, see Pepe (2003).
- Binormal: This is a parametric approach where the diagnostic variable in the two groups are assumed to be normal.
\[ D_Y\sim N(\mu_{D_Y}, \sigma_{D_Y}^2) \]
\[ D_{\bar{Y}}\sim N(\mu_{D_{\bar{Y}}}, \sigma_{D_{\bar{Y}}}^2) \]
When such distributional assumptions are made, ROC curve can be defined as:
\[ y(x)=1-G(F^{-1}(1-x)), \ \ 0\leq x\leq 1 \] where by \(F\) and \(G\) are the cumulative density functions of the diagnostic score in the negative and positive groups respectively, with \(f\) and \(g\) being corresponding probability density functions. For normal condition, the ROC curve and AUC under curve are given by:
\[ ROC\ Curve: y= \phi(A+BZ_x) \]
\[ AUC=\phi(\frac{A}{\sqrt{1+B^2}}) \]
where, \(Z_x=\phi^{-1}(x(t))=\frac{\mu_{D_{\bar{Y}}}-t}{\sigma_{D_{\bar{Y}}}}\), \(t\) being a cutoff; and \(A=\frac{|\mu_{D_{{Y}}}-\mu_{D_{\bar{Y}}}|}{\sigma_{D_{{Y}}}}\), \(B=\frac{\sigma_{D_{\bar{Y}}}}{\sigma_{D_{{Y}}}}\).
Confidence interval of ROC curve: To get the confidence interval, variance of \(A+BZ_x\) is derived using:
\[ V(A+B Z_x)=V(A)+Z_x^2V(B)+2Z_xCov(A, B) \] A \((1-\alpha)\times100\%\) level confidence limit for \(A+Z_xB\) can be obtained as
\[ (A+Z_xB)\pm \phi^{-1}(1-\alpha/2)\sqrt{V(A+Z_xB)} \] Point-wise confidence limit can be achieved by taking \(\phi\) of the above expression.
- Non-parametric: Non-parametric estimates of \(f\) and \(g\) are used in this approach. Zou, Hall, and Shapiro (1997) presented one such approach using Kernel densities:
\[ \hat{f}(x)=\frac{1}{n_{\bar{Y}}h_{\bar{ Y}}}\sum_{i=1}^{n_{\bar{ Y}}} K\big( \frac{x-D_{\bar{ Y}i} }{h_{\bar{ Y}}} \big) \]
\[ \hat{g}(x)= \frac{1}{n_{{Y}}h_y}\sum_{i=1}^{n_{{ Y}}} K\big( \frac{x-D_{{ Y}i} }{h_Y} \big) \]
where \(K\) is the Kernel function and \(h\) smoothing parameter (bandwidth). Zou, Hall, and Shapiro (1997) suggested a biweight Kernel:
\[ K\big(\frac{x-\alpha}{\beta}\big)=\begin{cases} \frac{15}{16} \Big[ 1-\big(\frac{x-\alpha}{\beta}\big)^2 \Big] , & x\in (\alpha - \beta, \alpha + \beta)\\ 0, & \text{otherwise} \end{cases} \]
with the bandwidth given by, \[ h_{\bar{Y}}=0.9\times min\big( \sigma_{\bar{ Y}}, \frac{IQR(D_{\bar{ Y}})}{1.34} \big)/ (n_{\bar{ Y}} )^{\frac{1}{5}} \] \[ h_{{Y}}=0.9\times min\big( \sigma_{{ Y}}, \frac{IQR(D_{{ Y}})}{1.34} \big)/ (n_{{ Y}} )^{\frac{1}{5}} \]
Smoother versions of TPR and FPR are obtained as the right-hand side area (of cutoff) of the smoothed \(f\) and \(g\). That is,
\[ \hat{TPR}(t)=1-\int_{-\infty}^{t}\hat{g}(t)dt=1-\hat{G}(t) \]
\[ \hat{FPR}(t)=1-\int_{-\infty}^{t}\hat{f}(t)dt=1-\hat{F}(t) \] When discrete pairs of \((FPR, TPR)\) are obtained, trapezoidal rule can be applied to calculate the AUC.
