ML Evaluation Metrics Beyond Accuracy: Precision, Recall, AUC and MCC

Accuracy feels intuitive, which is exactly why it causes so many problems in production. Early in my career I shipped a fraud detection model that reported 99.2% accuracy in staging, only to miss nearly every fraudulent transaction when we audited the live predictions. The dataset had 0.8% positives, so a model that predicted "not fraud" for every row looked excellent on paper and was useless in practice. That experience changed how I evaluate classifiers. For any non-trivial binary or multiclass problem, I now start from the confusion matrix and build a metric stack that reflects real costs, not just correct predictions. In this article I will walk through the metrics that actually inform model selection for imbalanced and cost-sensitive tasks: precision, recall, F1, ROC-AUC, PR-AUC, and Matthews Correlation Coefficient (MCC).

Why Accuracy Collapses When Classes Are Skewed

Accuracy is defined as (TP + TN) / (TP + TN + FP + FN). It treats every error equally and every correct prediction equally. That assumption rarely holds. In medical screening, a false negative (missing a disease) is orders of magnitude more expensive than a false positive (an unnecessary follow-up test). In spam filtering, the opposite is often true. When the class distribution is skewed, accuracy hides that cost asymmetry by being dominated by the majority class.

In my experience, the failure mode is predictable. If 95% of your samples are negative, a naive classifier that always predicts negative achieves 95% accuracy while learning nothing. I have seen this pattern in churn prediction, manufacturing defect detection, and content moderation. The training loop converges, the validation accuracy looks stable, and the first business review reveals zero recall on the minority class. Accuracy also obscures calibration and threshold effects. A model can output poorly calibrated probabilities yet still achieve high accuracy at a default 0.5 threshold simply because most negatives sit far from the boundary.

Reading the Confusion Matrix First

Before I look at any aggregated metric, I unpack the 2x2 table. True positives and true negatives tell me where the model is confident, while false positives and false negatives tell me where the cost lives. For imbalanced problems I also track the prevalence (positive rate) explicitly, because the same confusion matrix means something very different at 1% prevalence versus 30% prevalence. If you are working with tabular models and experimenting with sampling or weighting strategies, this is where understanding your data distribution matters as much as the algorithm. I often reference my earlier walkthrough on Handling Imbalanced Classes: SMOTE, Class Weights and Threshold Tuning when deciding whether to fix the data, the loss, or the decision threshold.

A practical habit that has saved me weeks of debugging: log the raw confusion matrix for every validation fold, not just the mean metric. When you use strategies discussed in Cross-Validation Strategies: Beyond K-Fold for Real-World Data, you will often find that a single fold with a different prevalence drives most of the variance in your averaged accuracy. That variance is a signal, not noise.

Precision and Recall: The Tension You Control With the Threshold

Precision and recall separate two questions that accuracy conflates. Precision asks, "Of all the items the model flagged as positive, how many were actually positive?" Recall asks, "Of all the truly positive items, how many did the model find?" Formally, precision = TP / (TP + FP) and recall = TP / (TP + FN). The first penalizes false positives; the second penalizes false negatives.

This distinction maps directly to operational cost. In my work on transaction monitoring, product demanded high precision because each false positive created a manual review ticket. We tuned for precision at the expense of missing some fraud. In contrast, for a cancer screening model I consulted on, the clinical team required recall above 0.95 even if it meant precision dropped to 0.30, because missing a case was unacceptable. There is no universal "good" precision or recall value. There is only the tradeoff your product and risk tolerance can accept.

How the Decision Threshold Moves the Tradeoff

Most classifiers output a score or probability. The threshold converts that score to a label. At threshold 0.9, you get high precision and low recall. At threshold 0.1, the opposite. I have found that teams spend too much time tuning hyperparameters and not enough time tuning the threshold on a held-out calibration set. The precision-recall curve, which plots precision vs. recall across thresholds, makes this tradeoff visible. Unlike a single point estimate, the curve shows whether any threshold can satisfy both constraints at once. If the curve hugs the top-right corner, you have headroom. If it collapses early, you need better features or a different model family, not just threshold tricks. For guidance on feature work that actually moves these curves, see Feature Engineering for Tabular Data: Techniques That Actually Work.

from sklearn.metrics import precision_score, recall_score, f1_score, confusion_matrix

# y_true: ground truth, y_pred: thresholded predictions
precision = precision_score(y_true, y_pred, zero_division=0)
recall = recall_score(y_true, y_pred, zero_division=0)
f1 = f1_score(y_true, y_pred, zero_division=0)
tn, fp, fn, tp = confusion_matrix(y_true, y_pred).ravel()

print(f"Precision: {precision:.3f} | Recall: {recall:.3f} | F1: {f1:.3f}")
print(f"Confusion Matrix: TP={tp} FP={fp} FN={fn} TN={tn}")

# Precision-Recall curve across thresholds
from sklearn.metrics import precision_recall_curve
import numpy as np

precisions, recalls, thresholds = precision_recall_curve(y_true, y_scores)
# Choose threshold that gives recall >= 0.9 with max precision
target_idx = np.where(recalls >= 0.90)[0]
best_thresh = thresholds[target_idx[np.argmax(precisions[target_idx])]] if len(target_idx) > 0 else 0.5

F1, F-Beta and the Cost of Compressing Two Numbers Into One

F1 is the harmonic mean of precision and recall: F1 = 2 * (precision * recall) / (precision + recall). It is useful when you need a single number for model selection and you consider false positives and false negatives equally undesirable. The harmonic mean penalizes extreme imbalance more than the arithmetic mean. A model with precision 0.9 and recall 0.1 gets an F1 of 0.18, not 0.5, which correctly signals that it is not useful if you need both.

In practice, I rarely optimize for F1 alone without thinking about beta. The F-beta score generalizes F1: F_beta = (1 + beta^2) * (precision * recall) / (beta^2 * precision + recall). Beta > 1 weights recall higher; beta < 1 weights precision higher. For a support ticket triage model where missed urgent tickets (low recall) meant SLA breaches, I used F2 (beta=2). For a recommendation system where false positives hurt trust, F0.5 was more appropriate. I have seen teams pick F1 out of habit and then wonder why the selected model does not match business incentives. Define beta from the cost ratio before you run the search.

When Averaging Fails in Multiclass Problems

For multiclass evaluation, how you average matters. Micro-averaged precision, recall and F1 aggregate counts globally and are dominated by frequent classes. Macro-averaging computes the metric per class and then averages, giving rare classes equal weight. Weighted averaging accounts for support. I have found that reporting all three exposes problems: if micro-F1 is 0.88 but macro-F1 is 0.54, your model is failing on rare classes even though the headline number looks strong. Do not let a single averaged F1 hide per-class recall collapse. Log per-class precision and recall alongside the aggregated score, and slice by class prevalence.

from sklearn.metrics import fbeta_score, classification_report

# Tune for F2 when recall matters twice as much as precision
f2 = fbeta_score(y_true, y_pred, beta=2.0, zero_division=0)

# Detailed per-class report exposes macro vs micro differences
print(classification_report(y_true, y_pred, digits=3, zero_division=0))
# For multiclass, compare averaging strategies explicitly
from sklearn.metrics import precision_recall_fscore_support
prec_micro, rec_micro, f1_micro, _ = precision_recall_fscore_support(y_true, y_pred, average='micro', zero_division=0)
prec_macro, rec_macro, f1_macro, _ = precision_recall_fscore_support(y_true, y_pred, average='macro', zero_division=0)
print(f"Micro F1: {f1_micro:.3f} vs Macro F1: {f1_macro:.3f}")

ROC-AUC: What It Actually Measures and Where It Misleads

ROC-AUC remains one of the most reported metrics, and also one of the most misunderstood. The ROC curve plots true positive rate (recall) against false positive rate (FP / (FP + TN)) across thresholds. AUC is the area under that curve. Intuitively, AUC is the probability that a randomly chosen positive sample is scored higher than a randomly chosen negative sample. An AUC of 0.5 is random, 1.0 is perfect separation, and 0.0 is perfectly inverted.

What I value about ROC-AUC is its threshold independence. It evaluates ranking quality, not a single operating point. This makes it useful for comparing models during architecture search when you have not yet fixed a threshold. It is also insensitive to class prevalence in the sense that it uses TPR and FPR, both of which are rates within each class. In my experience, this is helpful for model selection in balanced or moderately imbalanced settings where ranking matters, such as learning-to-rank or initial screening stages.

The problem is that ROC-AUC can look optimistic under severe imbalance. Because false positive rate uses the count of true negatives in its denominator, a large TN population dilutes the impact of many false positives. A model can have an AUC of 0.92 while precision is 0.15 at any usable recall. I have seen this repeatedly in ad click prediction and intrusion detection where negatives dominate. The business cares about precision at the operating threshold, but ROC-AUC does not directly show it. This is why I never report ROC-AUC alone for imbalanced tasks.

PR-AUC as a More Sensitive Alternative

The Precision-Recall curve and its area (PR-AUC, also called average precision) is often more informative when positives are rare. PR-AUC directly shows the tradeoff between precision and recall without the dampening effect of true negatives. Baseline PR-AUC is the prevalence (positive rate), not 0.5, so you must interpret it relative to that baseline. A PR-AUC of 0.40 is excellent if prevalence is 0.02 and poor if prevalence is 0.50. When I compare boosting models on imbalanced tabular data, I track both ROC-AUC and PR-AUC side-by-side and select based on PR-AUC if the production threshold will operate in the high-recall or high-precision region. The PyTorch Documentation provides practical implementations for these curves in TorchMetrics that integrate cleanly with training loops, and recent surveys on arXiv Machine Learning (https://arxiv.org/list/cs.LG/recent) consistently emphasize PR-AUC for rare-event evaluation.

from sklearn.metrics import roc_auc_score, average_precision_score, roc_curve, precision_recall_curve
import matplotlib.pyplot as plt

roc_auc = roc_auc_score(y_true, y_scores)
pr_auc = average_precision_score(y_true, y_scores)  # area under PR curve

fpr, tpr, _ = roc_curve(y_true, y_scores)
prec, rec, _ = precision_recall_curve(y_true, y_scores)

print(f"ROC-AUC: {roc_auc:.3f} | PR-AUC (AP): {pr_auc:.3f} | Prevalence: {y_true.mean():.3f}")

# In PyTorch / TorchMetrics (pseudo-code for training loop):
# from torchmetrics.classification import BinaryAUROC, BinaryAveragePrecision
# auroc = BinaryAUROC()
# ap = BinaryAveragePrecision()
# auroc.update(y_scores_tensor, y_true_tensor)

Matthews Correlation Coefficient: The Balanced Metric I Trust for Imbalanced Data

If I had to choose a single scalar metric to summarize binary classification on imbalanced data, I would choose Matthews Correlation Coefficient. MCC is essentially the Pearson correlation between the true and predicted binary labels. It returns a value between -1 and +1, where +1 is perfect prediction, 0 is random, and -1 is total disagreement. The formula uses all four quadrants of the confusion matrix: MCC = (TP * TN - FP * FN) / sqrt((TP+FP)(TP+FN)(TN+FP)(TN+FN)).

What makes MCC valuable is that it only rewards a model when it performs well on both the positive and negative classes. Unlike F1, which ignores true negatives, MCC accounts for TN explicitly. Unlike accuracy, it is not inflated by a large majority class. In a project on equipment failure prediction with 2% positives, we had two candidate models with nearly identical F1 scores of 0.41. Their MCC scores were 0.38 and 0.21, and the lower MCC model had three times the false positives. That difference translated to hundreds of wasted inspections per month. MCC surfaced the distinction; F1 did not.

Interpreting and Using MCC in Practice

MCC is not always intuitive for stakeholders, so I report it alongside precision, recall, and business cost, not instead of them. I have found MCC particularly useful in three situations: first, when comparing models across datasets with different prevalence; second, when doing early pruning in automated hyperparameter search where you need a single balanced objective; and third, when you need to detect when a model has collapsed to majority-class predictions (MCC drops to 0 or near 0 while accuracy stays high). Note that MCC is still threshold-dependent; you must compute it at your operating threshold. I often optimize the threshold for MCC on the validation set and then validate that the resulting precision and recall are acceptable for the use case.

A practical note on multiclass MCC: the generalization (sometimes called R_k or multiclass MCC) exists in scikit-learn and works well for multi-class imbalance. If your classes are heavily imbalanced, track multiclass MCC alongside macro-F1 to get both a per-class balanced view and a correlation-based view. This combination has helped me catch models that over-optimize for head classes while ignoring the tail.

Choosing the Right Metric Stack for Model Selection and Monitoring

No single metric tells the full story. The mistake I see most often is optimizing for one metric in offline evaluation and then being surprised by live performance. In production, you need a metric stack that aligns with business cost and remains sensitive to distribution shift. My default stack for binary classification is: ROC-AUC for ranking quality during model selection, PR-AUC for sensitivity to rarity, precision and recall at the chosen operating threshold, F-beta tuned to cost, and MCC as a balanced summary. I also log calibration error because threshold-dependent metrics are meaningless if probabilities drift.

How you select the operating threshold deserves its own run. I never reuse the threshold that maximized F1 on the test set. I use a separate validation or calibration split to choose the threshold that meets the precision or recall constraint, then lock it before final test evaluation. If you are tuning gradient boosting models, for example, the interaction between class weights, learning rate, and threshold is non-trivial. A thorough comparison such as Gradient Boosting Showdown: XGBoost, LightGBM and CatBoost Compared shows how different frameworks handle weighting and early stopping, but the threshold decision still sits outside the booster. Similarly, capacity decisions interact with your metric choice; a large model that overfits can inflate accuracy while harming calibrated probabilities, a connection explored in Bias-Variance Tradeoff: A Practical Guide to Model Selection.

For monitoring, track the full metric set over time, sliced by cohort. A drop in recall on a minority slice may not move aggregate accuracy at all. I have found that alerting on MCC and precision at fixed recall is more responsive than alerting on accuracy. Also track prevalence. If prevalence shifts, your precision at a fixed threshold will shift even if the model is unchanged. Without that context, you will misattribute a data shift to model decay.

Frequently Asked Questions