Notes on chapter three of Deep Learning with Javascript
3.2.2 Precision Recall Accuracy and ROC Curves of Binary Classification
Confusion Matrix
Key ↘︎
The four result types in classification |
prediction | prediction | |
---|---|---|---|
positive | negative | ||
truth | positive | true positive | false negative |
truth | negative | false positive | true negative |
Example ↘︎
The four result types in classification |
prediction | prediction | |
---|---|---|---|
positive | negative | ||
truth | positive | 30 | 28 |
truth | negative | 40 | 02 |
Accuracy
We can understand accuracy from the table above as the in the following steps:
(30 + 02)/(30 + 02 + 40 + 28)
(32)/(100)
0.32
32.00%
/**
Accuracy:
=> ((#TP + #TN) / (#examples))
=> ((#TP + #TN) / (#TP + #TN + #FP + #FN))
**/
;(30 + 02) / (30 + 02 + 40 + 28)
Uneven distributions can make accuracy a misleading metric in vbinary classification.
Precision and Recall address this shortcoming in using accuracy to evaluate classification models.
Precision
The ratio of positive predictions made by the model that are actually positive.
(30 )/(30 + 40)
30 / 70
0.42857142857142855
or42.86%
While this is an imporovement we can still game even this metric by intentionally setting a conservative policy on emmiting positive predictions.
Recall
The ratio of actual positive examples that are classified as positive.
(30 )/(30 + 28)
30 / 58
0.5172413793103449
or51.72%
To game this metric one can declare all examples positive and score 100 recall.
Precision and recall is about tuning the model.
We tune the model to:
- find at least X% of positives, Precision at X% Recall
- adjust tricky places where there is uncertainty about the correct answer warrant this kind of analysis.
The threshold value applied to the sigmoid output doesnt necessarily have to be 0.5 by setting it lower it can allow the model to be more liberal in applying positive outputs (higher recall lower precision). Setting thresholds higher can lead to higher precision and lower recall by being more cautious in applying positive labels.
3.2.3 The ROC Curve - Tradeoffs in Binary Classification
Receiver operating characteristic or ROC stems from early radar systems.
- The horizontal access of an roc curve is the false positive rate
FPR = #FP / (#FP + #TN)
- the ratio of actually negative examples to the number false positives
- The vertical axis is the true positive rate.
TPR = #TP / ( #TP + #FN) = recall
- this is the same definintion as recall