Classification Metrics, From the Ground Up: Precision, Recall, F1, ROC and AUC

Almost every evaluation number you’ll ever report, whether for detectors, trackers, segmenters, or even generative models, is built from four counts on a binary decision. Get the confusion matrix and the precision/recall trade-off right once, and the rest of the field stops being a zoo of acronyms. We’ll build all of it on one tiny running example, a backyard cat camera, so every formula comes with a number you can check by hand.

The running example: Whiskers, the backyard cat-cam

Meet Whiskers, a wildlife camera pointed at a backyard. Its one job: snap a photo whenever a cat walks by. Last night it scored 10 photos, each with a confidence that there’s a cat in frame. By morning you check what was really in each shot. Here’s the night’s log, and we’ll compute every metric in this article from this table:

Photo	What was really there	Cat? (truth)	Whiskers’ score
#1	cat under the floodlight	✅ yes	0.95
#2	cat on the fence	✅ yes	0.88
#3	cat by the bins	✅ yes	0.72
#4	cat at the door	✅ yes	0.60
#5	cat in deep shadow	✅ yes	0.45
#6	cat half-out of frame	✅ yes	0.20
#7	a raccoon	❌ no	0.55
#8	empty bush	❌ no	0.40
#9	empty lawn	❌ no	0.15
#10	a swaying shadow	❌ no	0.30

So 6 photos truly have a cat, 4 don’t. To turn scores into yes/no decisions we need a threshold, so start with the usual 0.5: flag a photo as “cat” when the score is ≥ 0.5. That flags photos #1, #2, #3, #4 (real cats) and #7 (the raccoon, a 0.55 false alarm). Photos #5 and #6 are real cats Whiskers slept through. Hold this picture; every metric below is a different question about it.

The confusion matrix

Sort every photo into one of four bins by asking two questions: did we flag it, and was it really a cat? Everything else is arithmetic on those four counts.

For a binary classifier, every prediction lands in one of four cells, comparing the predicted label against the truth:

	Actual positive	Actual negative
Pred. positive	True Positive (TP)	False Positive (FP)
Pred. negative	False Negative (FN)	True Negative (TN)

A false positive is a false alarm; a false negative is a miss. Which one hurts more is a domain question (a spam filter and a cancer screen weight them very differently), and that choice is what every metric below is really about [4].

Accuracy, and why it lies

The obvious question is what fraction of the 10 photos Whiskers got right, which is accuracy:

$$\text{Accuracy} = \frac{TP + TN}{TP + TN + FP + FN}.$$

The trap is class imbalance. Suppose the backyard were quieter and only 1 of 100 photos had a cat. A lazy camera that never flags anything scores 99% accuracy while catching zero cats, because its TN count alone carries the score. Accuracy hides the failure because the easy negatives dominate the sum. You need metrics that ignore the easy negatives: precision and recall.

Precision and recall

These two split “was the model right?” into two different questions.

Precision asks when Whiskers cries “cat!”, how often is it right? It punishes false alarms.

$$\text{Precision} = \frac{TP}{TP + FP}.$$

Recall (also called sensitivity, or true-positive rate) asks of all the cats that walked by, how many did we catch? It punishes misses.

$$\text{Recall} = \frac{TP}{TP + FN}.$$

The picture makes the two denominators obvious: precision divides by the flagged set, recall by the actual cats:

They pull in opposite directions. Flag everything → recall 1.0, precision in the gutter. Flag only your single most-confident shot → precision 1.0, recall near zero. The art is the balance, and it’s set by the threshold (next section).

Two more counts come from the negative column, used in ROC analysis:

$$\text{Specificity (TNR)} = \frac{TN}{TN + FP}, \qquad \text{FPR} = \frac{FP}{FP + TN} = 1 - \text{Specificity}.$$

For Whiskers: specificity $= 3/(3+1) = 0.75$ (it correctly ignored 3 of the 4 non-cats), so FPR $= 0.25$.

F1 and Fβ: one number from two

Sometimes you want a single score that stays low unless precision and recall are both decent, so a terrible recall can’t hide behind a great precision. That’s the harmonic mean, which (unlike the plain average) is dragged toward the smaller of the two:

$$F_1 = 2 \cdot \frac{\text{Precision} \cdot \text{Recall}}{\text{Precision} + \text{Recall}}.$$

When the two errors aren’t equally costly, Fβ weights recall β times as much as precision ($\beta = 2$ favours recall, $\beta = 0.5$ favours precision):

$$F_\beta = (1 + \beta^2)\,\frac{\text{Precision} \cdot \text{Recall}}{\beta^2 \cdot \text{Precision} + \text{Recall}}.$$

For Whiskers, if missing cats is the real sin we’d track F2 ≈ 0.69 (pulled toward the weaker recall); if false 3 a.m. alerts are worse, F0.5 ≈ 0.77 (pulled toward the stronger precision).

The threshold, and the curves that free you from it

Every number above came from one choice: threshold 0.5. Slide that threshold and the whole confusion matrix changes. Here’s Whiskers’ night laid out by score: cats above the line, non-cats below, the threshold as a movable wall (flag everything to its right):

A single F1 only describes one wall position. To judge the model itself, sweep the threshold:

Precision–Recall curve and Average Precision

Plot precision against recall across all thresholds and you get the PR curve. The summary number is Average Precision (AP), the area under it, a sum over the points where recall steps up:

$$\text{AP} = \sum_{n} (R_n - R_{n-1})\, P_n.$$

ROC curve and AUC

Plot true-positive rate (recall) against false-positive rate across thresholds; the area under it is AUC (AUROC) [2]. It has a beautifully concrete meaning:

AUC is the probability that the model scores a random cat higher than a random non-cat.

So 0.5 is random guessing and 1.0 is perfect ranking. AUC is threshold-free and great for ranking quality; the PatchCore vs VLM post on VisA uses it to compare anomaly detectors without picking a threshold for either. But on very imbalanced data it looks optimistically high (FPR has a huge TN denominator), so prefer PR-AUC / AP there.

Now you try

Reading is one thing; moving the numbers is another. The playground below starts on Whiskers’ exact night. Drag the threshold and watch the confusion matrix recompute while the live dot traces the PR and ROC curves, and find the wall position that catches the shadow-cat without letting the raccoon in. Then switch to Edit the counts and push TP/FP/FN/TN around to see how each metric reacts (try making precision 1.0 and watch what happens to recall).

Things worth confirming for yourself: at threshold 0 everything is flagged (recall 1.0, precision in the gutter); push it to 1.0 and you flag nothing (recall 0); and no slider position beats the raccoon, because it always costs you either a false alarm or a missed cat. That stubbornness is the 0.83 AUC.

In scikit-learn

The same Whiskers numbers, computed for real with scikit-learn [5], and the printed values match every hand calculation above:

# scikit-learn==1.9.1
import numpy as np
from sklearn.metrics import (
    confusion_matrix, precision_score, recall_score, f1_score,
    roc_auc_score, average_precision_score, classification_report,
)

# Whiskers' night: photos #1..#10, 1 = a cat is really there.
y_true = np.array([1, 1, 1, 1, 1, 1, 0, 0, 0, 0])
y_prob = np.array([.95, .88, .72, .60, .45, .20, .55, .40, .15, .30])  # scores
y_pred = (y_prob >= 0.5).astype(int)                                   # threshold

tn, fp, fn, tp = confusion_matrix(y_true, y_pred).ravel()
print(f"TP={tp} FP={fp} FN={fn} TN={tn}")     # TP=4 FP=1 FN=2 TN=3
print("precision", precision_score(y_true, y_pred))   # 0.80
print("recall   ", recall_score(y_true, y_pred))      # 0.667
print("f1       ", f1_score(y_true, y_pred))           # 0.727

# Threshold-free, computed from the scores (not the 0/1 predictions):
print("ROC-AUC", roc_auc_score(y_true, y_prob))            # 0.833
print("PR-AUC ", average_precision_score(y_true, y_prob))  # 0.917

print(classification_report(y_true, y_pred, digits=3))

The key habit: pass scores to roc_auc_score / average_precision_score and hard labels to precision/recall/F1. Mixing them up is the most common metrics bug there is.

Caveats & common pitfalls

• Every score sits on top of the labels. If you’d mislabeled the raccoon as a cat, Whiskers’ “precision” would rise to 1.0 while the camera got no better. Noisy ground truth caps and distorts every number. Audit the labels before chasing the last point.
• One threshold = one operating point. A lone precision/recall/F1 describes a single wall position; always report the threshold you used (or the curve).
• ROC-AUC flatters imbalanced data. Its FPR has a huge true-negative denominator, so it stays high even when the model is useless on the minority class, so prefer PR-AUC / AP there [1].
• Discrimination ≠ calibration. AUC and AP measure ranking; they say nothing about whether a “0.8” really means 80%. If you act on probabilities, check calibration separately; Guo et al. found modern networks more accurate than older ones yet worse calibrated, typically overconfident [3].
• Precision and recall are also where generative models get scored. The image-generation chapter borrows them to separate sample fidelity from coverage.

Takeaways

• Everything starts at the confusion matrix. TP/FP/FN/TN, and which error costs more. Whiskers was 4/1/2/3.
• Accuracy is unreliable under imbalance (0.70 here, but 99% for a camera that never fires on a quiet night), reach for precision/recall.
• Precision (0.80) vs recall (0.67) is a trade-off set by the threshold; F1/Fβ combine them at one operating point.
• To judge the model across thresholds, use curves: PR-AUC / AP (≈0.92 here) under imbalance, ROC-AUC (≈0.83) for general ranking quality.
• Feed scores to AUC/AP and labels to P/R/F1.

Next in the series: how these scale to many classes and to multi-label problems → Multiclass & multilabel classification metrics.

Further reading

• Go deeper on assessment: Hastie et al., ESL ch. 7, for bias–variance and how to evaluate a model beyond a single test score.
• Calibration: scikit-learn’s probability calibration guide, on turning raw scores into probabilities you can act on.
• Related on CondadosAI: next, Multiclass & multilabel metrics; then how this AP becomes object-detection mAP.

References

[1] Davis, J., & Goadrich, M. (2006). The Relationship Between Precision-Recall and ROC Curves. Proceedings of the 23rd International Conference on Machine Learning (ICML). DOI:10.1145/1143844.1143874.

[2] Fawcett, T. (2006). An Introduction to ROC Analysis. Pattern Recognition Letters, 27(8), 861–874. DOI:10.1016/j.patrec.2005.10.010.

[3] Guo, C., Pleiss, G., Sun, Y., & Weinberger, K. Q. (2017). On Calibration of Modern Neural Networks. ICML, PMLR 70:1321–1330. arXiv:1706.04599.

[4] Murphy, K. P. (2022). Probabilistic Machine Learning: An Introduction, §5.1.2 (Classification problems), including §5.1.2.2 (Cost-sensitive classification). MIT Press. Free online.

[5] Pedregosa, F., et al. (2011). Scikit-learn: Machine Learning in Python. JMLR, 12, 2825–2830. Docs (scikit-learn 1.9): Classification metrics.