Deep Learning with Javascript 3.2.2-3.2.2

Posted on April 10, 2020

Notes on chapter three of Deep Learning with Javascript

3.2.2 Precision Recall Accuracy and ROC Curves of Binary Classification

Confusion Matrix

Key ↘︎
The four result types
in classification		 prediction prediction
positive negative
truth positive true positive false negative
truth negative false positive true negative
Example ↘︎
The four result types
in classification		 prediction prediction
positive negative
truth positive 30 28
truth negative 40 02

Accuracy


We can understand accuracy from the table above as the in the following steps:


  • (30 + 02)/(30 + 02 + 40 + 28)
  • (32)/(100)
  • 0.32
  • 32.00%

/** 
Accuracy: 
=> ((#TP + #TN) / (#examples)) 
=> ((#TP + #TN) / (#TP + #TN + #FP + #FN))
**/
;(30 + 02) / (30 + 02 + 40 + 28)

Uneven distributions can make accuracy a misleading metric in vbinary classification.

Precision and Recall address this shortcoming in using accuracy to evaluate classification models.

Precision


The ratio of positive predictions made by the model that are actually positive.


  • (30 )/(30 + 40)
  • 30 / 70
  • 0.42857142857142855 or 42.86%

While this is an imporovement we can still game even this metric by intentionally setting a conservative policy on emmiting positive predictions.

Recall


The ratio of actual positive examples that are classified as positive.


  • (30 )/(30 + 28)
  • 30 / 58
  • 0.5172413793103449 or 51.72%

To game this metric one can declare all examples positive and score 100 recall.

Precision and recall is about tuning the model.

We tune the model to:

  • find at least X% of positives, Precision at X% Recall
  • adjust tricky places where there is uncertainty about the correct answer warrant this kind of analysis.

The threshold value applied to the sigmoid output doesnt necessarily have to be 0.5 by setting it lower it can allow the model to be more liberal in applying positive outputs (higher recall lower precision). Setting thresholds higher can lead to higher precision and lower recall by being more cautious in applying positive labels.

3.2.3 The ROC Curve - Tradeoffs in Binary Classification

Receiver operating characteristic or ROC stems from early radar systems.

  • The horizontal access of an roc curve is the false positive rate
    • FPR = #FP / (#FP + #TN)
    • the ratio of actually negative examples to the number false positives
  • The vertical axis is the true positive rate.
    • TPR = #TP / ( #TP + #FN) = recall
    • this is the same definintion as recall