What Is SIFT? Finding the Same Point in Two Photos

SIFT finds points in a photograph that can be found again in another photograph of the same scene, taken in another direction, from somewhere else or at another zoom, and describes each point with 128 numbers so that the two photos can be paired point by point. The animation shows where this unit ends: two photos of Barcelona harbour, their SIFT points, every point paired with its most similar point in the other photo, and then only the pairs SIFT trusts, coloured by whether they are right.

Every SIFT match goes through the same six steps. Lesson 1 sets the yardstick they are scored by; lessons 2 to 5 take the steps in order.

1. Blur at many sizes. A feature can be 2 pixels wide or 80. Blurring the photo by increasing amounts and subtracting neighbouring blurs makes every size of feature stand out at one of the levels. Each halving of the photo along the way starts a new octave, a stack of blurs at one resolution.
2. Keep the extrema. A candidate is a pixel brighter (or darker) than its 26 neighbours: 8 around it at its own blur, and 9 at the blur above and 9 below. The level it wins at is its size.
3. Refine, and drop the weak and edge-like ones. The position is moved to a fraction of a pixel, and candidates with too little contrast, or that lie along an edge where they could slide, are thrown out.
4. Give each point a direction. The strongest gradient direction around the point becomes its own compass; a point with two strong directions becomes two keypoints.
5. Describe it. The descriptor is the point’s summary for matching. The gradients around the point, measured relative to that compass and at that size, are summarised as 4 × 4 small histograms of 8 directions: 128 numbers.
6. Match. Each point is paired with the point of the other photo whose 128 numbers are closest, and the pair is kept only when the second-closest is much farther.

What the matches are used for

You have met SIFT matches, or the matches of its descendants, in:

• Panoramas. A phone’s panorama mode matches points between neighbouring frames and warps them onto one canvas; unit 3.5 stitches this same Barcelona pair.
• 3-D reconstruction from photos, where matches across dozens of photos give the cameras’ positions and the scene’s shape (structure from motion).
• Recognising a landmark, a painting or a product from a phone photo, by matching it against a database of reference photos.
• AR and robot navigation, which follow matched points from frame to frame to know how the camera moved.
• Image registration in medicine, remote sensing and astronomy, where two images of the same thing must be laid on top of each other.

SIFT finds the same point in two photos taken in different directions, at different zooms and turned by any angle, and tells you which pairs to trust. On the Barcelona pair it keeps 440 pairs and 420 are right. A raw patch of pixels compared directly gets 0 right once one photo is turned 20°, where SIFT gets 433. What makes the difference is that each SIFT point comes with its own size and direction: across a 2× zoom, points described at their own size give 294 right matches, and corner points described at one fixed size give 1.

Before this

You need Edge detection, unit 3.1: SIFT is built from image gradients, its scale space is the Laplacian of Gaussian in another form, and the question “which points can be found again” starts at corners. The blurs are 2-D convolutions with a Gaussian.

How to describe a point

To pair points across two photos, every point needs a description that the other photo can be searched for. The first idea is the patch itself: the 11 × 11 pixels around the point, compared with every patch around the points of the other photo, after subtracting each patch’s mean and dividing by its spread so that a brighter photo does not change the score (normalized cross-correlation).

On the Barcelona pair that works, a little. Then turn photo B by 20°, or shrink it to 71%, and compare again, keeping only clear winners in each case:

A patch is a grid of pixels in the photo’s own rows and columns. Turn the photo and the same harbour lands on different rows and columns; zoom out and it covers fewer pixels. The patch changes completely, even though the harbour did not.

SIFT’s description fixes the three things that broke. It measures the patch at the point’s own size, so a zoomed-out copy is measured over the same piece of harbour (lessons 2 and 3). It lays the grid along the point’s own direction, so a turned copy is read in the same order (lesson 3). And it records gradient directions in coarse histograms, so a brighter photo or a pixel of shift barely moves it (lesson 4). Szeliski §7.1.2 sets SIFT among the descriptors that came before and after it [2], and in Mikolajczyk and Schmid’s comparison of ten descriptors, GLOH, an extension of SIFT, came out best in most tests, closely followed by SIFT [9]. Lowe published a first version in 1999 [10] and the form used here in 2004 [1].

Topics

1. What makes a good keypoint? The yardstick every step is scored by: repeatability [3], a point found again after the camera moves. Measured on Harris and SIFT, and why the size SIFT attaches matters more than where it puts the point.
2. Scale space and the difference of Gaussians. Step 1: one detector for every size of feature, built from blurs, and why subtracting two blurs stands in for the Laplacian [4].
3. How SIFT finds keypoints. Steps 2 to 4: the 26-neighbour test, the sub-pixel fit, the contrast and edge tests and the orientation, built from scratch and checked point for point against OpenCV.
4. The SIFT descriptor. Step 5: 4 × 4 × 8 = 128 numbers, and what each design choice buys, measured by taking it away.
5. Lowe’s ratio test. Step 6: nearest neighbour, then comparing the best candidate with the second best.

The order follows the pipeline. Lesson 1 sets the measure, lessons 2 and 3 find the points, lesson 4 describes them, and lesson 5 pairs them.

The scene

Two photos of Barcelona harbour [8], taken from one spot on Montjuïc by turning the camera, the same pair unit 3.5 stitches into a panorama. Every lesson measures on them at 960 × 723 px. Because the camera only turned, one homography (a 3 × 3 map) sends each pixel of photo A to its place in photo B, which gives an answer key for every match. I fit it with ORB [6], a different detector with a different descriptor, so SIFT is never scored against itself: 345 ORB pairs agree on it, with a median error of 0.81 px, and the homography SIFT’s own matches give agrees with it to a median of 0.46 px. A match is right when it lands within 3 px of where that homography says. For changes the pair does not contain, the lessons turn, shrink, blur, darken or add noise to photo A themselves, where the answer is exact.

In the wild: a temple seen from a ring of cameras. The Middlebury templeRing set [7] is 47 photos of a plaster temple taken every 7.66° around it. It is the one test the Barcelona pair cannot give: the camera walking around a 3-D object.

What the unit measures

Every number below is a key in output/sift_numbers.json.

Try it

Every point of photo A that photo B can show, with its two nearest distances. Slide the threshold of step 6, or tap a point to see why it was kept or dropped.

Limitations & caveats

• One pair from one viewpoint. The Barcelona photos differ by a turn of the camera, so every real-pair number here is about a change of direction and of exposure. Walking around a scene is covered only by the temple sweep, on one object.
• The answer key is another detector’s fit. A pair counts as right when it lands within 3 px of the ORB homography. That homography and the one SIFT’s matches give differ by a median of 0.46 px and at most 1.53 px over the points SIFT keeps, well inside the tolerance. It is still a fit to the photos, with no surveyed points to check it against.
• The warps are synthetic. Turning or shrinking a photo in software is a cleaner change than a real camera makes (no new perspective, no motion blur). The rotation and zoom numbers are an upper bound on what a real photo would give.
• OpenCV’s defaults. Every count uses OpenCV’s default thresholds (contrast 0.04, edge ratio 10, three levels per octave). Lowering the contrast threshold returns more points on a dim photo, and more noise.
• Lesson 1’s repeatability depends on the tolerance. With a 1 px tolerance SIFT repeats better than Harris after halving the photo (29% against 17%); with 3 px, Harris does (57% against 38%). The lesson reports the whole table.
• Speed is not measured. The from-scratch detector takes 0.61 s on photo A and OpenCV 0.036 s, on the CPU; neither is a benchmark.

What this means for your pipeline

• Matching photos of the same scene from other directions or zooms: SIFT with the ratio test at 0.75 to 0.8 is a sound default. On this pair, 95.5% of what it keeps is right before any geometric check.
• Always follow it with geometry. The ratio test let 20 wrong pairs through. RANSAC, which keeps the pairs that agree on one camera motion, removes them: with a homography when the camera only turned, or with a fundamental matrix (the map between two photos taken from different places) otherwise.
• Viewpoint changes past about 30° around an object: capture more photos in between. On the temple the consistent pairs fall from 55 at 30.6° to 9 at 38.3°.
• Dim or low-contrast photos: expect far fewer points. Darkening photo A to 40% cut its SIFT points from 2,897 to 426, though 96% of the ones that remained were found again in the same places.
• What this does not tell you: speed on your hardware, how learned features compare (unit 3.4, not written yet), or how SIFT does on repeated texture such as façades of identical windows, which this unit did not measure.

Your turn: mirrors and half turns

Before running anything, predict how many of SIFT’s matches survive against:

1. Photo A turned upside down (180°).
2. Photo A mirrored left to right, as a selfie camera shows you.

Where this lands

SIFT’s pairs go to RANSAC, which keeps those that agree on one camera motion, and then to the homography and the panorama of unit 3.5 or to structure from motion. Unit 3.4, not written yet, continues with what came after SIFT: binary descriptors such as ORB [6] and learned features.

In code, the whole pipeline is a few lines of OpenCV [5]:

import cv2

sift = cv2.SIFT_create()
kp_a, desc_a = sift.detectAndCompute(img_a, None)  # steps 1-5
kp_b, desc_b = sift.detectAndCompute(img_b, None)

knn = cv2.BFMatcher(cv2.NORM_L2).knnMatch(desc_a, desc_b, k=2)
good = [m for m, n in knn if m.distance < 0.75 * n.distance]  # step 6

#include <opencv2/opencv.hpp>

auto sift = cv::SIFT::create();
std::vector<cv::KeyPoint> kpA, kpB;
cv::Mat descA, descB;
sift->detectAndCompute(imgA, cv::noArray(), kpA, descA);  // steps 1-5
sift->detectAndCompute(imgB, cv::noArray(), kpB, descB);

cv::BFMatcher matcher(cv::NORM_L2);
std::vector<std::vector<cv::DMatch>> knn;
matcher.knnMatch(descA, descB, knn, 2);
std::vector<cv::DMatch> good;
for (const auto& m : knn)
  if (m[0].distance < 0.75f * m[1].distance) good.push_back(m[0]);  // step 6

Run it: every code block on this page has a cell in the unit’s notebook, open it in Colab.

Further reading

• The source: Lowe 2004 [1] is unusually readable, and every constant in this unit (three levels per octave, the edge ratio of 10, 36 bins, 4 × 4 × 8, the 0.2 clip, the 0.8 ratio) is justified there with an experiment.
• The book: Szeliski, §7.1 “Points and patches” [2], places SIFT among the detectors and descriptors that came before and after it.
• How descriptors were compared: Mikolajczyk and Schmid [9], which tested SIFT against the other descriptors of its time under the same kinds of change.
• Related on CondadosAI: Edge detection is the prerequisite · RANSAC is what the matches go to · What is a homography? stitches this same pair.

References

[1] Lowe, D. G. (2004). Distinctive Image Features from Scale-Invariant Keypoints. International Journal of Computer Vision, 60(2), 91–110. doi:10.1023/B:VISI.0000029664.99615.94

[2] Szeliski, R. (2022). Computer Vision: Algorithms and Applications (2nd ed.), ch. 7 “Feature detection and matching”: §7.1 “Points and patches”, p. 419, §7.1.1 “Feature detectors”, p. 422, §7.1.2 “Feature descriptors”, p. 434, §7.1.3 “Feature matching”, p. 441. Springer. Free PDF

[3] Schmid, C., Mohr, R., & Bauckhage, C. (2000). Evaluation of Interest Point Detectors. International Journal of Computer Vision, 37(2), 151–172, §3.1 “Repeatability criterion”. doi:10.1023/A:1008199403446

[4] Lindeberg, T. (1998). Feature Detection with Automatic Scale Selection. International Journal of Computer Vision, 30(2), 79–116. doi:10.1023/A:1008045108935

[5] OpenCV 4.11.0 source, modules/features2d/src/sift.dispatch.cpp (defaults: contrast threshold 0.04, edge threshold 10, σ 1.6, lines 90–92) and sift.simd.hpp, commit 31b0eee.

[6] Rublee, E., Rabaud, V., Konolige, K., & Bradski, G. (2011). ORB: An Efficient Alternative to SIFT or SURF. ICCV 2011. doi:10.1109/ICCV.2011.6126544

[7] Seitz, S. M., Curless, B., Diebel, J., Scharstein, D., & Szeliski, R. (2006). A Comparison and Evaluation of Multi-View Stereo Reconstruction Algorithms. CVPR 2006, pp. 519–528. doi:10.1109/CVPR.2006.19. The Middlebury multi-view data sets: vision.middlebury.edu/mview

[8] Pap3rinik. BarcelonaHarbour1.jpg and BarcelonaHarbour2.jpg. Wikimedia Commons, public domain. commons.wikimedia.org

[9] Mikolajczyk, K., & Schmid, C. (2005). A Performance Evaluation of Local Descriptors. IEEE Transactions on Pattern Analysis and Machine Intelligence, 27(10), 1615–1630. doi:10.1109/TPAMI.2005.188

[10] Lowe, D. G. (1999). Object Recognition from Local Scale-Invariant Features. Proceedings of the Seventh IEEE International Conference on Computer Vision (ICCV 1999), vol. 2, pp. 1150–1157. doi:10.1109/ICCV.1999.790410