Multiclass and Multilabel Metrics: Micro, Macro, Weighted, and the Difference That Matters

Two words decide whether your multiclass F1 rewards the model for the rare classes or lets the common ones paper over them: “macro” vs “micro.” Pick the wrong average and a model that ignores your minority class can still look great. We’ll watch that happen on one running example — a shelter intake camera — and you can push the numbers around yourself.

This builds directly on the binary foundations — read that first if precision/recall/F1 aren’t second nature yet.

The running example: PetSpotter, the shelter intake camera

PetSpotter photographs each animal arriving at a small shelter and sorts it into one of three classes: Dog, Cat, or Bird. Last batch had 20 animals — and they’re imbalanced, like real intake: 12 dogs, 5 cats, 3 birds. The model loves dogs, is so-so on cats, and — the twist — never once says “bird”; every bird gets called a cat. Here’s the full $3\times3$ confusion matrix (rows = what it really was, columns = what PetSpotter guessed):

Multiclass: one-vs-rest, then average

Intuition: score each class as its own yes/no question — “Dog vs not-Dog” — which turns the big matrix into three familiar binary problems, then combine.

With $C$ mutually exclusive classes you compute precision/recall/F1 per class by treating it as “this class vs everything else,” giving $P_c, R_c, F_1^{(c)}$.

The only real question is how you collapse those three F1s into one number — and the choice changes the verdict completely.

Intuition — macro: every class gets one vote, no matter how rare.

Intuition — micro: every sample gets one vote (pool all the counts first), so big classes dominate.

Intuition — weighted: macro, but each class’s vote is scaled by its support $n_c$.

Now you try

Edit any cell of PetSpotter’s matrix and watch the per-class scores and all three averages react. The lesson to confirm: give the Bird row a real diagonal (say move those 3 into the Bird column) and macro-F1 jumps far more than micro — because you just fixed the class micro was already ignoring.

Top-k accuracy

Intuition: for hard, many-class problems, give credit if the right answer is in the running — among the model’s top $k$ guesses.

Multilabel: each photo can carry many labels

Now move PetSpotter outdoors, where one frame can hold several animals at once — a dog and a cat on the deck. Predictions and truth become binary vectors over $L=3$ labels $\{$dog, cat, bird$\}$, and two metrics are unique to this setting.

Intuition — subset accuracy: all-or-nothing. The whole photo’s tag set must match exactly; one wrong tag fails it.

Intuition — Hamming loss: partial credit. Score each label slot on its own and count the fraction wrong.

Then the same micro / macro / weighted averages apply to P/R/F1 over the $L$ labels, plus a samples average (per-sample, then averaged). For ranking-style multilabel, label-ranking AP / per-label AP are the go-to.

Now you try

Toggle PetSpotter’s predicted tags (click the cells) and watch subset accuracy lurch while Hamming loss moves gently. Try fixing just the missed cat on “Deck” — subset accuracy leaps from 0.33 to 0.67 off a single slot, which is exactly its all-or-nothing temperament.

In scikit-learn

The same PetSpotter numbers, computed for real (labels 0=Dog, 1=Cat, 2=Bird):

import numpy as np
from sklearn.metrics import f1_score, classification_report, top_k_accuracy_score
from sklearn.metrics import hamming_loss, accuracy_score

# --- Multiclass: 20 animals (12 dog, 5 cat, 3 bird); birds all called cat ---
y_true = [0]*12 + [1]*5            + [2]*3
y_pred = [0]*12 + [0,0,1,1,1]      + [1,1,1]   # 2 cats->dog, all birds->cat
print("macro   ", f1_score(y_true, y_pred, average="macro"))      # 0.489
print("micro   ", f1_score(y_true, y_pred, average="micro"))      # 0.75 == accuracy
print("weighted", f1_score(y_true, y_pred, average="weighted"))   # 0.690
print(classification_report(y_true, y_pred, digits=3))            # per-class + all

# --- Multilabel: 3 backyard photos, labels [dog, cat, bird] ---
Y_true = np.array([[1, 0, 0], [1, 1, 0], [0, 0, 1]])
Y_pred = np.array([[1, 0, 0], [1, 0, 0], [0, 1, 1]])
print("subset accuracy", accuracy_score(Y_true, Y_pred))   # 0.333
print("hamming loss   ", hamming_loss(Y_true, Y_pred))     # 0.222
print("micro-F1       ", f1_score(Y_true, Y_pred, average="micro"))   # 0.75
print("macro-F1       ", f1_score(Y_true, Y_pred, average="macro"))   # 0.667

Caveats & common pitfalls

• The averaging choice is a reporting decision, not a detail. Macro, micro, and weighted disagreed by 26 points on PetSpotter (0.49 vs 0.75) — never quote a bare “F1” without saying which average produced it [3].
• Micro-F1 hides minority failure by construction. It pooled away the entire Bird failure into a healthy-looking 0.75. Read the per-class report [3].
• Undefined precision is silent. Bird’s precision is 0/0 (never predicted); libraries quietly fill 0 and may warn — check, because it can swing macro a lot.
• Top-k flatters large label sets. It credits “in the running,” right for shortlists, a vanity number if you only act on the top guess.
• Multilabel thresholds are per-label. Subset accuracy and Hamming loss both swing with each label’s decision threshold; tune and report it [4].

Takeaways

• Multiclass = one-vs-rest per class + an averaging choice. That choice is the whole story — PetSpotter scored 0.49, 0.69, or 0.75 depending on it.
• Macro for fairness to rare classes; micro when every sample should count equally (and micro-F1 = accuracy for single-label multiclass); weighted is the frequency-weighted compromise.
• Always read the per-class report, not just the headline average — it’s where the rare-class failures (Bird = 0.00) hide.
• Multilabel adds subset accuracy (strict, 0.33) and Hamming loss (granular, 0.22); then reuse micro/macro/samples averaging.

Next: leaving pure classification for localization → Object detection metrics: IoU, AP, and mAP.

Further reading

• Imbalanced data, hands-on: the imbalanced-learn user guide — resampling and the metric choices that go with skewed classes.
• Go deeper: Hastie et al., ESL ch. 4 — multiclass classification from a statistical-learning view.
• Related on CondadosAI: the binary foundations; next, object-detection metrics.

References

[1] Sokolova, M., & Lapalme, G. (2009). A Systematic Analysis of Performance Measures for Classification Tasks. Information Processing & Management, 45(4), 427–437. DOI:10.1016/j.ipm.2009.03.002.

[2] Tsoumakas, G., & Katakis, I. (2007). Multi-Label Classification: An Overview. International Journal of Data Warehousing and Mining, 3(3), 1–13.

[3] Hastie, T., Tibshirani, R., & Friedman, J. (2009). The Elements of Statistical Learning (2nd ed.), ch. 4. Springer. Free PDF.

[4] Pedregosa, F., et al. (2011). Scikit-learn: Machine Learning in Python. JMLR, 12, 2825–2830. Docs: Multiclass & multilabel averaging.