Definitions

• A type I error occurs when the null hypothesis (H0) is true, but is rejected. Also known as false positive, where positive means rejection of a null hypothesis.

• A type II error occurs when the null hypothesis is false, but erroneously fails to be rejected. Also called as false negative, where negative refers to retaining a null hypothesis.

• TPR (True Positive Rate) = # True positives / # positives = Recall = \( TP \over (TP+FN) \)

• FPR (False Positive Rate) = # False Positives / # negatives = \( FP \over (FP+TN) \)

• Precision = # True positives / # predicted positive = \( TP \over (TP+FP) \)

• \( F-measure = \frac{(\beta^2 + 1)PR}{\beta^2P+R} \)

Majority of positive samples — ROC is a better metric

small positive class — precision and recall are better

reference

1. What metrics should be used for evaluating a model on an imbalanced data set? (precision + recall or ROC=TPR+FPR)