Multiclass and Multilabel Metrics: Micro, Macro, Weighted, and the Difference That Matters

Two words decide whether your multiclass F1 rewards the model for the rare classes or lets the common ones paper over them: “macro” vs “micro.” Pick the wrong average and a model that ignores your minority class can still look great. We’ll watch that happen on one running example, a shelter intake camera, and you can push the numbers around yourself.

This builds directly on the binary foundations. Read that first if precision/recall/F1 aren’t second nature yet.

The running example: PetSpotter, the shelter intake camera

PetSpotter photographs each animal arriving at a small shelter and sorts it into one of three classes: Dog, Cat, or Bird. Last batch had 20 animals: and they’re imbalanced, like real intake: 12 dogs, 5 cats, 3 birds. The model loves dogs, is so-so on cats, and, the twist, never once says “bird”; every bird gets called a cat. Here’s the full $3\times3$ confusion matrix (rows = what it really was, columns = what PetSpotter guessed):

Multiclass: one-vs-rest, then average

Score each class as its own yes/no question, “Dog vs not-Dog”. That turns the big matrix into three familiar binary problems, which you then combine.

With $C$ mutually exclusive classes you compute precision/recall/F1 per class by treating it as “this class vs everything else” (the one-versus-the-rest construction [3]), giving $P_c, R_c, F_1^{(c)}$.

The remaining question is how you collapse those three F1s into one number, and the choice changes the verdict completely. Macro and micro averaging are defined formally by Sokolova & Lapalme [1]; the support-weighted variant is the one scikit-learn calls average="weighted" [4].

Macro averaging gives every class one vote, however rare it is:

Micro averaging gives every sample one vote: pool all the counts first, so big classes dominate:

Weighted averaging is macro with each class’s vote scaled by its support $n_c$:

Now you try

Edit any cell of PetSpotter’s matrix and watch the per-class scores and all three averages react. The lesson to confirm: give the Bird row a real diagonal (say move those 3 into the Bird column) and macro-F1 jumps far more than micro, because you just fixed the class micro was already ignoring.

Top-k accuracy

For hard, many-class problems, top-$k$ accuracy gives credit when the right answer is in the running, among the model’s top $k$ guesses.

Multilabel: each photo can carry many labels

Now move PetSpotter outdoors, where one frame can hold several animals at once: a dog and a cat on the deck. Predictions and truth become binary vectors over $L=3$ labels $\{$dog, cat, bird$\}$, and two metrics are unique to this setting [2].

Subset accuracy is all-or-nothing: the whole photo’s tag set must match, and one wrong tag fails it.

Hamming loss gives partial credit: score each label slot on its own and count the fraction wrong.

Then the same micro / macro / weighted averages apply to P/R/F1 over the $L$ labels, plus a samples average (per-sample, then averaged). For ranking-style multilabel, label-ranking AP / per-label AP are the go-to.

Now you try

Toggle PetSpotter’s predicted tags (click the cells) and watch subset accuracy lurch while Hamming loss moves gently. Try fixing just the missed cat on “Deck”: subset accuracy leaps from 0.33 to 0.67 off a single slot, which is its all-or-nothing temperament at work.

In scikit-learn

The same PetSpotter numbers, computed for real (labels 0=Dog, 1=Cat, 2=Bird):

# scikit-learn==1.9.1
import numpy as np
from sklearn.metrics import f1_score, classification_report, top_k_accuracy_score
from sklearn.metrics import hamming_loss, accuracy_score

# --- Multiclass: 20 animals (12 dog, 5 cat, 3 bird); birds all called cat ---
y_true = [0]*12 + [1]*5            + [2]*3
y_pred = [0]*12 + [0,0,1,1,1]      + [1,1,1]   # 2 cats->dog, all birds->cat
print("macro   ", f1_score(y_true, y_pred, average="macro"))      # 0.489
print("micro   ", f1_score(y_true, y_pred, average="micro"))      # 0.75 == accuracy
print("weighted", f1_score(y_true, y_pred, average="weighted"))   # 0.690
print(classification_report(y_true, y_pred, digits=3))            # per-class + all

# --- Multilabel: 3 backyard photos, labels [dog, cat, bird] ---
Y_true = np.array([[1, 0, 0], [1, 1, 0], [0, 0, 1]])
Y_pred = np.array([[1, 0, 0], [1, 0, 0], [0, 1, 1]])
print("subset accuracy", accuracy_score(Y_true, Y_pred))   # 0.333
print("hamming loss   ", hamming_loss(Y_true, Y_pred))     # 0.222
print("micro-F1       ", f1_score(Y_true, Y_pred, average="micro"))   # 0.75
print("macro-F1       ", f1_score(Y_true, Y_pred, average="macro"))   # 0.667

Caveats & common pitfalls

• The averaging choice is a reporting decision, not a detail. Macro, micro, and weighted disagreed by 26 points on PetSpotter (0.49 vs 0.75), so never quote a bare “F1” without saying which average produced it [1].
• Micro-F1 hides minority failure by construction. It pooled away the entire Bird failure into a healthy-looking 0.75. Read the per-class report [1].
• Undefined precision is silent. Bird’s precision is 0/0 (never predicted); libraries quietly fill 0 and may warn, so check, because it can swing macro a lot.
• Top-k flatters large label sets. It credits “in the running,” right for shortlists, a vanity number if you only act on the top guess.
• Multilabel thresholds are per-label. Subset accuracy and Hamming loss both swing with each label’s decision threshold; tune and report it [4].

Takeaways

• Multiclass = one-vs-rest per class + an averaging choice. That choice is the whole story. PetSpotter scored 0.49, 0.69, or 0.75 depending on it.
• Macro for fairness to rare classes; micro when every sample should count equally (and micro-F1 = accuracy for single-label multiclass); weighted is the frequency-weighted compromise.
• Always read the per-class report, not just the headline average. It’s where the rare-class failures (Bird = 0.00) hide.
• Multilabel adds subset accuracy (strict, 0.33) and Hamming loss (granular, 0.22); then reuse micro/macro/samples averaging.

Next: leaving pure classification for localization → Object detection metrics: IoU, AP, and mAP.

Further reading

• Imbalanced data, hands-on: the imbalanced-learn user guide, on resampling and the metric choices that go with skewed classes.
• Go deeper: Hastie et al., ESL ch. 4, for multiclass classification from a statistical-learning view.
• Related on CondadosAI: the binary foundations; next, object-detection metrics.

References

[1] Sokolova, M., & Lapalme, G. (2009). A Systematic Analysis of Performance Measures for Classification Tasks. Information Processing & Management, 45(4), 427–437. DOI:10.1016/j.ipm.2009.03.002.

[2] Tsoumakas, G., & Katakis, I. (2007). Multi-Label Classification: An Overview. International Journal of Data Warehousing and Mining, 3(3), 1–13.

[3] Murphy, K. P. (2022). Probabilistic Machine Learning: An Introduction, §17.3.7 (Multi-class classification with SVMs), which describes the one-versus-the-rest construction. MIT Press. Free online.

[4] Pedregosa, F., et al. (2011). Scikit-learn: Machine Learning in Python. JMLR, 12, 2825–2830. Docs (scikit-learn 1.9): From binary to multiclass and multilabel.