After the predictive model has been finished, the most important question is: How good is it? Does it predict well?
Evaluating the model is one of the most important tasks in the data science project, it indicates how good predictions are. Very often for classification problems we look at metrics called precision and recall, to define them in detail let’s quickly introduce confusion matrix first.
Confusion Matrix for binary classification is made of four simple ratios:
- True Negative(TN): case was true negative and predicted negative
- True Positive(TP): case was true positive and predicted positive
- False Negative(FN): case was true positive but predicted negative
- False Positive(FP): case was true negative but predicted positive
After fully understanding the confusion matrix, calculating precision and recall is easy.
Precision – is the ratio of correctly predicted positive observations to the total predicted positive observations, or what percent of positive predictions were correct?
Precision = TP/TP+FP
Recall – also called sensitivity, is the ratio of correctly predicted positive observations to all observations in actual class – yes, or what percent of the positive cases did you catch?
Recall = TP/TP+FN
There are also two more useful matrices coming from confusion matrix, Accuracy – correctly predicted observation to the total observations and F1 score the weighted average of Precision and Recall. Although intuitively it is not as easy to understand as accuracy, the F1 score is usually more useful than accuracy, especially if you have an uneven class distribution.
Example Python Code to get Precision and Recall:
from sklearn.linear_model import LogisticRegression
from sklearn import datasets
from sklearn.cross_validation import train_test_split
from sklearn.metrics import precision_recall_fscore_support as score
data = datasets.load_iris()
X = data['data']
y = data['target']
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0)
model = LogisticRegression()
model.fit(X_train,y_train)
preds = model.predict(X_test)
precision, recall, fscore, support = score(y_test, preds)
print('precision:',precision)
print('recall:',recall)