Image Generation Metrics: FID, IS, KID, LPIPS, and CLIPScore

Every metric so far compared a prediction to a known answer. Generative models break that: there’s no single “correct” image for “a cat in a spacesuit.” So we measure differently — comparing distributions of real vs generated images, and leaning on learned perceptual networks. We’ll score one toy cat generator every way that matters, with numbers you can move.

Unlike the classification foundations, generation has no per-sample target. A good metric has to capture three things: fidelity (do images look real?), diversity (is the model just memorizing a few?), and for text-to-image, alignment (does it match the prompt?).

The running example: CatGen, a cat-image generator

CatGen invents pictures of cats. We can’t check each one against a “right answer,” so instead we compare the distribution of CatGen’s images to the distribution of real cats. To keep the arithmetic visible, imagine an Inception network squashes each image to a single “cattiness” feature: real cats land around $\mu_r = 5$ with spread $\sigma_r = 1$; CatGen’s outputs land around $\mu_g$, $\sigma_g$. The closer those two bell curves, the better CatGen is.

Inception Score (IS)

Intuition: reward two things at once — each image is confidently one kind of thing (sharp), and across the batch you see many kinds (diverse).

The first widely-used one [1]: the KL divergence between each image’s label distribution $p(y \mid x)$ and the overall marginal $p(y)$:

$$\text{IS} = \exp\!\Big( \mathbb{E}_{x}\, D_{\mathrm{KL}}\big(p(y \mid x)\, \|\, p(y)\big) \Big).$$

Fréchet Inception Distance (FID) — the headline

Intuition: put real and generated features side by side as two bell curves and measure the distance between them. Same place, same spread → distance 0.

FID [2] models each set as a Gaussian and takes the Fréchet distance:

$$\text{FID} = \lVert \mu_r - \mu_g \rVert_2^2 + \operatorname{Tr}\!\Big( \Sigma_r + \Sigma_g - 2(\Sigma_r \Sigma_g)^{1/2} \Big).$$

Now you try

Drag CatGen’s mean and spread and watch the blue curve slide onto the green real-cat curve as FID falls toward 0. Confirm the two ways to score badly: a shifted mean (wrong kind of cats) and a wrong spread (too samey, or too chaotic) each push FID up on their own.

Kernel Inception Distance (KID)

Intuition: measure the same real-vs-generated gap, but in a way that doesn’t cheat on small samples.

KID [3] uses a squared Maximum Mean Discrepancy with a polynomial kernel instead of FID’s Gaussian assumption. It’s unbiased and far more stable on small sample sets, where FID reads artificially low. Also lower-is-better — so the FID playground above is the right mental model; KID just estimates that distance more honestly when you only have a few hundred images.

LPIPS — perceptual distance between two images

Intuition: “do these two specific images look alike to a human?” — measured in deep-feature space, not pixels.

When you have a pair to compare (a reconstruction, a paired translation, or two samples for diversity), LPIPS [4] gives a perceptual distance from network features that tracks human similarity far better than pixel MSE or SSIM. Lower = more similar (illustratively, two near-duplicate cat frames might score ~0.05, a cat vs a couch ~0.7). The common flip: report higher mean LPIPS among samples as a diversity signal.

CLIPScore — does the image match the prompt?

Intuition: alignment is the angle between the image’s embedding and the prompt’s embedding — point the same way, score high.

For text-to-image, none of the above checks prompt fidelity. CLIPScore [6] does, via cosine similarity of CLIP’s image and text embeddings:

$$\text{CLIPScore} = \max\big(\cos(E_{\text{image}}, E_{\text{text}}),\, 0\big).$$

Now you try

Drag the image vector toward the prompt vector and watch the cosine (and CLIPScore) climb to 1; swing it 90° away and alignment falls to 0. That angle is all CLIPScore is.

Beyond these, improved precision & recall for generative models [5] disentangle fidelity (precision: do samples fall on the real manifold?) from coverage (recall: does the model span real diversity?) — useful when FID alone is ambiguous.

In code (torchmetrics)

import torch
from torchmetrics.image.fid import FrechetInceptionDistance
from torchmetrics.image.kid import KernelInceptionDistance
from torchmetrics.multimodal.clip_score import CLIPScore

# FID — feed real and generated separately. normalize=True => float images in [0,1].
fid = FrechetInceptionDistance(feature=2048, normalize=True)
real = torch.rand(64, 3, 299, 299)
fake = torch.rand(64, 3, 299, 299)
fid.update(real, real=True)
fid.update(fake, real=False)
print("FID", fid.compute().item())          # lower is better; fix the sample count

kid = KernelInceptionDistance(subset_size=50, normalize=True)
kid.update(real, real=True); kid.update(fake, real=False)
print("KID mean/std", kid.compute())         # better for small sample sets

# CLIPScore — prompt fidelity (images uint8 in [0,255]).
clip = CLIPScore(model_name_or_path="openai/clip-vit-base-patch16")
imgs = (torch.rand(2, 3, 224, 224) * 255).to(torch.uint8)
print("CLIPScore", clip(imgs, ["a cat in a spacesuit", "a red bicycle"]).item())

Caveats & common pitfalls

• FID is a biased estimator. It shrinks as you add samples, so a number is only meaningful at a stated sample count — compare runs at the same N (10k–50k) [2]. On small sets, prefer KID, which is unbiased [3].
• Implementation details move the number. FID/KID inherit Inception’s ImageNet bias and are sensitive to image resizing, interpolation, and the exact feature extractor. Report the toolkit and settings; don’t compare across papers with different pipelines [2].
• CLIPScore can be gamed. It rewards embedding alignment, not correctness or aesthetics — the six-legged cat still scores well [6].
• Automated metrics are proxies. For generation, human evaluation still settles ties; distribution metrics measure quality+diversity in aggregate, never a single image.

Takeaways

• No ground truth → compare distributions (FID, KID, IS) and use learned perceptual models (LPIPS, CLIPScore).
• FID is the headline (lower better, captures quality + diversity; CatGen = 1.25) — but biased by sample count, so always compare at equal N; KID is the small-sample-safe alternative.
• IS is legacy — no real reference (CatGen’s max was just 2.0 = its class count), ImageNet-biased.
• LPIPS for perceptual pair distance / diversity; CLIPScore (CatGen = 0.80) for text-to-image prompt fidelity.
• Automated metrics are proxies — human evaluation still settles ties.

That closes the series. Back to the start: Classification metrics: the foundations.

Further reading

• Go deeper: Goodfellow et al., Deep Learning ch. 20 — the generative models these metrics are built to evaluate.
• Why your FID may be wrong: Parmar et al., On Aliased Resizing… (2022), clean-fid — how resizing bugs silently corrupt FID scores.
• Related on CondadosAI: back to where precision/recall began — classification foundations.

References

[1] Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., & Chen, X. (2016). Improved Techniques for Training GANs. NeurIPS. arXiv:1606.03498. (Inception Score.)

[2] Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., & Hochreiter, S. (2017). GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium. NeurIPS. arXiv:1706.08500. (FID.)

[3] Bińkowski, M., Sutherland, D. J., Arbel, M., & Gretton, A. (2018). Demystifying MMD GANs. ICLR. arXiv:1801.01401. (KID.)

[4] Zhang, R., Isola, P., Efros, A. A., Shechtman, E., & Wang, O. (2018). The Unreasonable Effectiveness of Deep Features as a Perceptual Metric. CVPR. arXiv:1801.03924. (LPIPS.)

[5] Kynkäänniemi, T., Karras, T., Laine, S., Lehtinen, J., & Aila, T. (2019). Improved Precision and Recall Metric for Assessing Generative Models. NeurIPS. arXiv:1904.06991.

[6] Hessel, J., Holtzman, A., Forbes, M., Le Bras, R., & Choi, Y. (2021). CLIPScore: A Reference-free Evaluation Metric for Image Captioning. EMNLP. arXiv:2104.08718.

[7] Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep Learning, ch. 20 (deep generative models). MIT Press. Free online. Tooling: TorchMetrics — image.