Evaluating performance measures of the classification model is often significantly trickier. There are many performance measures available. In the previous two tutorials, we discuss Confusion Matrix, Precision, Recall, and F1 score. So grab another coffee and get ready to learn one more performance measurement metrics.
The ROC(receiver operating characteristic) curve is used with binary classifiers. It is very similar to the precision/recall curve, but instead of plotting precision versus recall, the ROC curve plots TPR(the true positive rate) versus FPR (false positive rate).
The TPR is the ratio of positive instances that are correctly classified as positive while FPR is the ratio of negative instances that are incorrectly classified as positive. It is equal to 1-TNR(true negative rate), which is the ratio of negative instances that are correctly classified as negative.
To compute the ROC curve, you first need to have a set of predictions probability, so they can be compared to the actual targets. You could make predictions on the validation set.
y_val_cat_prob=model.predict_proba(x_val)
The roc_curve() function computed the TPR and FPR for various threshold values.
from sklearn.metrics import roc_curve,roc_auc_score
fpr , tpr , thresholds = roc_curve ( y_val_cat , y_val_cat_prob)
The first parameter to roc_curve() is the actual values for each sample, and the second parameter is the set of model-predicted probability values for each sample. The method produces the FPR and TPR.
Then you can plot the FPR against the TPR using Matplotlib.
def plot_roc_curve(fpr,tpr):
plt.plot(fpr,tpr)
plt.axis([0,1,0,1])
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.show()
plot_roc_curve (fpr,tpr)
One way to compare classifiers is to measure the area under the ROC curve, whereas a purely random classifier will have a ROC AUC equal to 0.5. Scikit-Learn provides a function to get AUC.
auc_score=roc_auc_score(y_val_cat,y_val_cat_prob) #0.8822AUC is the percentage of this area that is under this ROC curve, ranging between 0~1. The ROC and AUC score are much better way to evaluate the performance of a classifier.
Related Post
- Calculate F1 Macro in Keras
- Calculate and Plot AUC ROC Curve for Multi-Class Classification
- Calculate Precision, Recall and F1 score for Keras model
- Micro and Macro Averages for imbalance multiclass classification
- Split Imbalanced dataset using sklearn Stratified train_test_split().