metrics
7 posts · browse all tags
Evaluation How many good parts does it take to beat a System One AI model? Between one and sixteen, on VisA
System One models, the idea TypeSafe launched with Jev in September 2026, answer a question with a decision in a single pass. Jev-Omni, an open one, and the untouched Gemma 4 12B it was built from inspect all 2,162 VisA test images without seeing a good part. PatchCore, the standard industrial anomaly detector, is given k good parts per product: at 256 pixels it passes Jev-Omni at k = 8 and Gemma 4 at k = 16, at 512 pixels at k = 1 and k = 4. Also: the base model beats its System One fine-tune, and a day-0 demo on a lime line.
Evaluation Classification Metrics, From the Ground Up: Precision, Recall, F1, ROC and AUC
The confusion matrix and everything that falls out of it: precision, recall, F1, specificity, the precision/recall trade-off, PR and ROC curves, and AUC, all built on one running toy example (a backyard cat-cam), with the equations, the intuition, and scikit-learn.
Evaluation Image Generation Metrics: FID, IS, KID, LPIPS, and CLIPScore
How do you score images with no ground truth? You compare distributions and use learned perceptual models: Inception Score, Fréchet Inception Distance, KID, LPIPS, and CLIPScore for prompt fidelity, on one running cat-generator example, with the equations, the catches, and torchmetrics.
Evaluation Multiclass and Multilabel Metrics: Micro, Macro, Weighted, and the Difference That Matters
How precision/recall/F1 scale past two classes: one-vs-rest, and the micro/macro/weighted averaging that decides whether rare classes count. Plus multilabel-specific metrics: subset accuracy, Hamming loss, and top-k, all on one running shelter-camera example, with scikit-learn.
Evaluation Object Detection Metrics: IoU, AP, and the mAP Everyone Quotes
How detection is scored: IoU to decide a hit, greedy matching into TP/FP/FN, per-class Average Precision as the area under the precision–recall curve, and the COCO mAP@[.5:.95] you see in every paper, all on one running parking-lot example, with torchmetrics.
Evaluation Object Tracking Metrics: MOTA, MOTP, IDF1, and HOTA
Tracking adds identity over time, so its metrics score two things at once: detecting objects and keeping their IDs consistent. MOTA, MOTP, ID switches, IDF1, and the modern HOTA that balances detection against association, on one running door-cam example, with the equations and motmetrics.
Evaluation Segmentation Metrics: IoU, Dice, mIoU, and Panoptic Quality
Scoring masks instead of boxes: pixel IoU/Jaccard and the Dice coefficient (and why they're related), mIoU for semantic segmentation, mask-AP for instance segmentation, and Panoptic Quality for the unified task, on one running cat-mask example, with the equations and code.