Multiclass and Multilabel Metrics: Micro, Macro, Weighted, and the Difference That Matters

Two words decide whether your multiclass F1 rewards the model for the rare classes or lets the common ones paper over them: “macro” vs “micro.” Pick the wrong average and a model that ignores your minority class can still look great. We’ll watch that happen on one running example, a shelter intake camera, and you can push the numbers around yourself.

This builds directly on the binary foundations. Read that first if precision/recall/F1 aren’t second nature yet.

The running example: PetSpotter, the shelter intake camera

PetSpotter photographs each animal arriving at a small shelter and sorts it into one of three classes: Dog, Cat, or Bird. Last batch had 20 animals: and they’re imbalanced, like real intake: 12 dogs, 5 cats, 3 birds. The model loves dogs, is so-so on cats, and, the twist, never once says “bird”; every bird gets called a cat. Here’s the full $3\times3$ confusion matrix (rows = what it really was, columns = what PetSpotter guessed):

Multiclass: one-vs-rest, then average

Intuition: score each class as its own yes/no question, “Dog vs not-Dog”, which turns the big matrix into three familiar binary problems, then combine.

With $C$ mutually exclusive classes you compute precision/recall/F1 per class by treating it as “this class vs everything else,” giving $P_c, R_c, F_1^{(c)}$.

The only real question is how you collapse those three F1s into one number, and the choice changes the verdict completely. The three standard answers, macro, micro and weighted, are defined formally by Sokolova & Lapalme [1].

Intuition, macro: every class gets one vote, no matter how rare.

Intuition, micro: every sample gets one vote (pool all the counts first), so big classes dominate.

Intuition, weighted: macro, but each class’s vote is scaled by its support $n_c$.

Now you try

Edit any cell of PetSpotter’s matrix and watch the per-class scores and all three averages react. The lesson to confirm: give the Bird row a real diagonal (say move those 3 into the Bird column) and macro-F1 jumps far more than micro, because you just fixed the class micro was already ignoring.

Top-k accuracy

Intuition: for hard, many-class problems, give credit if the right answer is in the running, among the model’s top $k$ guesses.

Multilabel: each photo can carry many labels

Now move PetSpotter outdoors, where one frame can hold several animals at once: a dog and a cat on the deck. Predictions and truth become binary vectors over $L=3$ labels $\{$dog, cat, bird$\}$, and two metrics are unique to this setting [2].

Intuition, subset accuracy: all-or-nothing. The whole photo’s tag set must match exactly; one wrong tag fails it.

Intuition, Hamming loss: partial credit. Score each label slot on its own and count the fraction wrong.

Then the same micro / macro / weighted averages apply to P/R/F1 over the $L$ labels, plus a samples average (per-sample, then averaged). For ranking-style multilabel, label-ranking AP / per-label AP are the go-to.

Now you try

Toggle PetSpotter’s predicted tags (click the cells) and watch subset accuracy lurch while Hamming loss moves gently. Try fixing just the missed cat on “Deck”: subset accuracy leaps from 0.33 to 0.67 off a single slot, which is exactly its all-or-nothing temperament.

In scikit-learn

The same PetSpotter numbers, computed for real (labels 0=Dog, 1=Cat, 2=Bird):

import numpy as np
from sklearn.metrics import f1_score, classification_report, top_k_accuracy_score
from sklearn.metrics import hamming_loss, accuracy_score

# --- Multiclass: 20 animals (12 dog, 5 cat, 3 bird); birds all called cat ---
y_true = [0]*12 + [1]*5            + [2]*3
y_pred = [0]*12 + [0,0,1,1,1]      + [1,1,1]   # 2 cats->dog, all birds->cat
print("macro   ", f1_score(y_true, y_pred, average="macro"))      # 0.489
print("micro   ", f1_score(y_true, y_pred, average="micro"))      # 0.75 == accuracy
print("weighted", f1_score(y_true, y_pred, average="weighted"))   # 0.690
print(classification_report(y_true, y_pred, digits=3))            # per-class + all

# --- Multilabel: 3 backyard photos, labels [dog, cat, bird] ---
Y_true = np.array([[1, 0, 0], [1, 1, 0], [0, 0, 1]])
Y_pred = np.array([[1, 0, 0], [1, 0, 0], [0, 1, 1]])
print("subset accuracy", accuracy_score(Y_true, Y_pred))   # 0.333
print("hamming loss   ", hamming_loss(Y_true, Y_pred))     # 0.222
print("micro-F1       ", f1_score(Y_true, Y_pred, average="micro"))   # 0.75
print("macro-F1       ", f1_score(Y_true, Y_pred, average="macro"))   # 0.667

Caveats & common pitfalls

• The averaging choice is a reporting decision, not a detail. Macro, micro, and weighted disagreed by 26 points on PetSpotter (0.49 vs 0.75), so never quote a bare “F1” without saying which average produced it [3].
• Micro-F1 hides minority failure by construction. It pooled away the entire Bird failure into a healthy-looking 0.75. Read the per-class report [3].
• Undefined precision is silent. Bird’s precision is 0/0 (never predicted); libraries quietly fill 0 and may warn, so check, because it can swing macro a lot.
• Top-k flatters large label sets. It credits “in the running,” right for shortlists, a vanity number if you only act on the top guess.
• Multilabel thresholds are per-label. Subset accuracy and Hamming loss both swing with each label’s decision threshold; tune and report it [4].

Takeaways

• Multiclass = one-vs-rest per class + an averaging choice. That choice is the whole story. PetSpotter scored 0.49, 0.69, or 0.75 depending on it.
• Macro for fairness to rare classes; micro when every sample should count equally (and micro-F1 = accuracy for single-label multiclass); weighted is the frequency-weighted compromise.
• Always read the per-class report, not just the headline average. It’s where the rare-class failures (Bird = 0.00) hide.
• Multilabel adds subset accuracy (strict, 0.33) and Hamming loss (granular, 0.22); then reuse micro/macro/samples averaging.

Next: leaving pure classification for localization → Object detection metrics: IoU, AP, and mAP.

Further reading

• Imbalanced data, hands-on: the imbalanced-learn user guide, on resampling and the metric choices that go with skewed classes.
• Go deeper: Hastie et al., ESL ch. 4, for multiclass classification from a statistical-learning view.
• Related on CondadosAI: the binary foundations; next, object-detection metrics.

References

[1] Sokolova, M., & Lapalme, G. (2009). A Systematic Analysis of Performance Measures for Classification Tasks. Information Processing & Management, 45(4), 427–437. DOI:10.1016/j.ipm.2009.03.002.

[2] Tsoumakas, G., & Katakis, I. (2007). Multi-Label Classification: An Overview. International Journal of Data Warehousing and Mining, 3(3), 1–13.

[3] Hastie, T., Tibshirani, R., & Friedman, J. (2009). The Elements of Statistical Learning (2nd ed.), ch. 4. Springer. Free PDF.

[4] Pedregosa, F., et al. (2011). Scikit-learn: Machine Learning in Python. JMLR, 12, 2825–2830. Docs: Multiclass & multilabel averaging.