Before this unit: 2D convolution

Corners: The Structure Tensor, Harris, Shi-Tomasi and Förstner

A contour says where a boundary runs but not where you are along it. Two eigenvalues of a 2x2 matrix fix that: measured on a real photo, flat background scores 0.001/0.000, an edge 0.411/0.019 and a corner 0.442/0.318. Harris, Shi-Tomasi and Förstner all read that same matrix, and on the edge patch Harris comes out positive at k = 0.04 and negative at k = 0.06, so k decides the answer rather than the data.

Luis Condados · August 18, 2026 · Updated October 24, 2026 · 10 min read Intermediate

image-processing edge-detection features fundamentals

The strongest Harris responses on the same frame. They land where two edges meet, which is the one place a contour cannot tell you where you are.

A corner is a point where brightness changes in every direction, so, unlike a point on an edge, it can be pinned to one place. Below, a corner detector runs on photographs of a calibration chessboard, the way a camera-calibration app does.

A person holding a black-and-white chessboard in front of a camera: first a glowing heat map that is bright only where squares meet, then red dots on its strongest peaks, then the board's 54 inner corners joined by coloured lines in order — For each photo: the Shi-Tomasi corner response (bright where brightness changes in both directions), its strongest peaks, and the 54 inner corners of the board that a calibration app keeps. Calibration images from the OpenCV samples, Apache-2.0.

Corners are the points a lot of software holds on to:

Camera calibration apps find the inner corners of a chessboard in every photo and solve for the lens from them (camera calibration).
Video stabilisation and tracking follow corners from frame to frame; OpenCV’s function for picking them is named after the paper, goodFeaturesToTrack.
AR apps that keep a virtual object pinned to a table follow corner-like points on the real surface.
Panoramas and 3-D reconstruction start by matching points like these between photos (SIFT, structure from motion).

A curve cannot tell you where you are along it; asking whether brightness changes in both directions at once finds points instead. On the unit’s frame an edge scores the eigenvalues 0.411 / 0.019 and a corner 0.442 / 0.318. Harris, Shi-Tomasi and Förstner all read that one matrix, and on the edge patch Harris flips sign between k = 0.04 and k = 0.06, so there the constant decides the answer.

Where we are

Lesson 5, the last of the Edge Detection unit, and a branch off its pipeline rather than a step: it leaves from the gradient measurement and returns points instead of an edge map. Canny ended on two limits with one cause. Non-maximum suppression fails at a junction because “the gradient direction” is not defined there, and every contour it produces is a curve, so two photographs of the same temple give two sets of curves with no way to pair up points on them.

Both come from asking a one-directional question. A column’s silhouette is an edge because brightness changes across it. Along it nothing changes at all, which is why you cannot say how far up the column you are. So ask about both directions. The gradients this needs are the Sobel $I_x$ and $I_y$ from lesson 2.

Flat, edge, or corner: two eigenvalues decide

Slide the window one pixel in every direction and ask how much the content changes. A patch is matchable when it changes a lot in every direction. The object that captures this is built from the image gradients $I_x$ and $I_y$ (how fast brightness changes horizontally and vertically), averaged over the window $W$ :

M \;=\; \frac{1}{|W|}\sum_{(x,y)\in W} \begin{bmatrix} I_x^2 & I_x I_y \\ I_x I_y & I_y^2 \end{bmatrix}

The two eigenvalues $\lambda_1 \ge \lambda_2$ of $M$ measure the gradient energy along the two principal directions of the patch. The classification, due to Harris and Stephens [1] and covered in Szeliski ch. 7 (Feature Detection and Matching) [5]:

$\lambda_1$	$\lambda_2$	patch	on the temple
small	small	flat	the black backdrop
large	small	edge	a column’s silhouette
large	large	corner	where the base meets a column

Worked example · Three 4×4 toy patches, by hand

Work on 4×4 patches of intensities in $\{0, 1\}$ , with central-difference gradients $I_x = (I_{right} - I_{left})/2$ on the four interior pixels.

Edge (dark left half, bright right half; every row is 0 0 1 1): each interior pixel has $I_x = (1-0)/2 = 0.5$ and $I_y = 0$ , so

M = \begin{bmatrix} 0.25 & 0 \\ 0 & 0 \end{bmatrix} \;\Rightarrow\; \lambda_1 = 0.25,\; \lambda_2 = 0

Corner (bright bottom-right quadrant): the four interior gradients are $(0,0)$ , $(0,0.5)$ , $(0.5,0)$ , $(0.5,0.5)$ . Averaging their outer products:

M = \begin{bmatrix} 0.125 & 0.0625 \\ 0.0625 & 0.125 \end{bmatrix} \;\Rightarrow\; \lambda_{1,2} = 0.125 \pm 0.0625 = 0.1875,\; 0.0625

Flat (all zeros): $M = 0$ , so $\lambda_1 = \lambda_2 = 0$ .

What it means: the edge has all its energy on one axis ( $\lambda_2 = 0$ says sliding along the edge costs nothing, the aperture problem in one number). The corner resists sliding on both axes. The flat patch resists nothing.

The same computation on the real photo [6], with Sobel gradients and a 21×21 window (these three spots are the presets in the lab below; artifact output/sift_numbers.json):

spot on the temple photo	$\lambda_1$	$\lambda_2$	verdict
background (50, 60)	0.001	0.000	flat
column silhouette (277, 240)	0.411	0.019	edge
base corner (150, 320)	0.442	0.318	corner

The ratio between the cases is enormous: the edge carries 22× more energy in its first direction than its second (0.411 / 0.019), while the corner is within a factor of 1.4 of isotropic. No tuned threshold is needed to tell these apart.

The three cases as gradient pictures. An edge’s gradients all point across it, so one eigenvalue collects everything and sliding along the edge is free. A corner’s gradients disagree, and that disagreement is what makes it findable. The numbers under each panel are measured on the temple photo.

Now you try

Drag the window over the photo and watch the eigenvalues respond; the three dots mark the preset spots from the table. The right panel scatters every pixel’s $(I_x, I_y)$ pair: an edge collapses the scatter onto a line, a corner spreads it, and the ellipse (axes $\sqrt{\lambda_1}, \sqrt{\lambda_2}$ ) summarizes the shape.

Try it: drag the window — flat, edge, or corner?

Try the silhouette of a column, then slide along it. The verdict stays “edge” and the ellipse stays needle-shaped the whole way down, which is the aperture problem made visible: the patch cannot tell you where along the edge it came from.

Three responses, one matrix

Computing two eigenvalues per pixel was expensive in 1988, and Harris and Förstner avoid it by scoring the matrix through its determinant and trace. Shi and Tomasi, six years later, take the eigenvalue anyway; for a 2×2 symmetric matrix it has a closed form, $\lambda_{2} = \tfrac{1}{2}\left(\operatorname{tr} M - \sqrt{(\operatorname{tr} M)^2 - 4\det M}\right)$ , so the cost is one square root. The three differ in what they consider a corner, and all three are still the standard entry points into feature extraction [4].

Harris and Stephens [1] use the determinant and trace, which are $\lambda_1\lambda_2$ and $\lambda_1 + \lambda_2$ and cost no eigendecomposition:

R = \det M - k\,(\operatorname{tr} M)^2

Shi and Tomasi [2] take the smaller eigenvalue on its own, $\min(\lambda_1, \lambda_2)$ , on the grounds that a patch is only as trackable as its weakest direction.

Förstner and Gülch [3] refuse to collapse the two into one number. They report a strength, $w = \det M / \operatorname{tr} M$ , and separately a roundness, $q = 4\det M / (\operatorname{tr} M)^2$ , which is 1 for a perfectly isotropic patch and 0 for a pure edge.

Worked example · The three measured spots, scored three ways

The eigenvalues measured above, put through each response:

spot	$\lambda_1$	$\lambda_2$	$R$ (k=0.04)	$R$ (k=0.06)	Shi-Tomasi	$w$	$q$
background	0.001	0.000	-0.000000	-0.000000	0.000	0.00000	0.0000
column edge	0.411	0.019	+0.000413	-0.003285	0.019	0.01816	0.1689
base corner	0.442	0.318	0.117452	0.105900	0.318	0.18494	0.9734

Substituting for the corner at k = 0.04:

R = (0.442)(0.318) - 0.04\,(0.442 + 0.318)^2 = 0.140556 - 0.023104 = 0.117452

What it means: all three agree the corner is a corner and the background is nothing. They disagree about the edge, and the disagreement is instructive.

Harris separates corner from edge by a factor of 284, far more than Shi-Tomasi’s 16.7, so as a ranking it is the sharper of the two. But its edge score at k = 0.04 is positive. Any pipeline that keeps pixels where $R > 0$ accepts that edge patch as a corner. Raise k to 0.06 and the same patch scores -0.003285 and is rejected. The constant is doing the deciding, on real numbers, which is why k is quoted as a range and not a value.

Shi-Tomasi has no constant to get wrong; 0.019 against 0.318 needs no interpretation. Förstner’s $q$ says the same thing on a scale that is free of contrast entirely: 0.1689 for the edge against 0.9734 for the corner, where 1 would be a perfectly round error ellipse.

How close is this to OpenCV?

The structure tensor is three products of the gradients, each summed over the window. OpenCV computes it inside cv2.cornerMinEigenVal (Shi-Tomasi) and cv2.cornerHarris, with Sobel gradients scaled by $1 / (4 \times \text{window})$ and an unnormalised box sum. Built that way in NumPy, the three spots on a white square have answers known in advance: nothing in the middle, one direction on a side, two at a corner.

The structure tensor in NumPy

import numpy as np
import cv2

img = np.zeros((160, 160), np.float32)
img[50:110, 50:110] = 1  # white square
win = 5                  # window, pixels

s = 1 / (4 * win)        # OpenCV's scale
ix = cv2.Sobel(img, cv2.CV_32F, 1, 0, scale=s)
iy = cv2.Sobel(img, cv2.CV_32F, 0, 1, scale=s)

def wsum(a):  # sum over the window
    return cv2.boxFilter(a, -1, (win, win),
                         normalize=False)

xx, xy, yy = wsum(ix*ix), wsum(ix*iy), wsum(iy*iy)
tr, det = xx + yy, xx * yy - xy * xy
lam2 = tr / 2 - np.sqrt((xx - yy)**2 / 4 + xy**2)
harris = det - 0.04 * tr**2

Output

Against cornerMinEigenVal and cornerHarris

st = cv2.cornerMinEigenVal(img, win, 3)
h = cv2.cornerHarris(img, win, 3, 0.04)
print("max diff, Shi-Tomasi:",
      np.abs(lam2 - st).max())
print("max diff, Harris:   ",
      np.abs(harris - h).max())

spots = {"flat": (80, 80), "edge": (80, 50),
         "corner": (50, 50), "near": (56, 50)}
for name, (r, c) in spots.items():
    M = np.array([[xx[r, c], xy[r, c]],
                  [xy[r, c], yy[r, c]]])
    l2, l1 = np.linalg.eigvalsh(M)
    print(f"{name:6} l1={l1:.3f} l2={l2:.3f} "
          f"R={h[r, c]:+.4f}")

Output

max diff, Shi-Tomasi: 0.0
max diff, Harris:    3.7252903e-09
flat   l1=0.000 l2=0.000 R=+0.0000
edge   l1=0.400 l2=0.000 R=-0.0064
corner l1=0.250 l2=0.170 R=+0.0354
near   l1=0.400 l2=0.000 R=-0.0064

Shi-Tomasi agrees with OpenCV to the last bit on every pixel, borders included, and Harris to within 4e-9, which is float32 rounding (on OpenCV 5.0 both differences are 0.0). Both cells use OpenCV’s own Sobel and box sum, so what this checks is the eigenvalue and the two responses built on top. The spots sort themselves as the table at the top of the lesson says: the middle of the square has both eigenvalues at 0, a point on its left side has $\lambda_1 = 0.400$ and $\lambda_2 = 0$ , and the corner has 0.250 and 0.170. Harris is negative on the side and positive at the corner, with no constant to argue about, because a drawn edge has $\lambda_2 = 0$ . The temple’s column edge had $\lambda_2 = 0.019$ , and that small number is what let $k$ decide its sign.

In the wild

A detector that finds six hundred corners is worth nothing if they are six hundred different corners each time the camera moves. The property that matters is repeatability, and it can be measured on any photograph by rotating it and asking how many points come back.

Detecting Harris corners on a market stall, rotating the image by a known angle, detecting again, mapping those detections back, and counting a corner as repeated when some detection lands within three pixels of it:

rotation	corners in the interior	repeated within 3 px	repeatability
5°	374	319	85.3%
15°	374	331	88.5%
45°	374	318	85.0%

A market fruit stall with several hundred detected corners marked, clustered on the fruit boundaries and the stall structure — Where the detector puts its points: on the boundaries between fruit, on the crate edges, on the lettering. Source: “Marrakech souk fruit vendor” by Mustang Joe (Wikimedia Commons), CC0.

What it means: between 85% and 88% of interior corners survive a rotation, and the figure barely moves between 5° and 45°. That is the rotation invariance the structure tensor predicts, checked on a photograph nobody prepared: rotating a patch rotates its eigenvectors and leaves the eigenvalue pair unchanged, so the corner measure does not care which way the camera was held.

The one-in-seven that does not come back is, I believe, mostly interpolation: the rotation resamples the image, which can move weak corners by more than three pixels or soften them below the detector’s quality threshold. I did not separate those two effects. And 15° scoring higher than 5° is noise. Three rotations of one image cannot resolve a three-point difference, and I would not read a mechanism into it.

Where this breaks

The window is a fixed size. Every number above came from a 21x21 patch, and that choice is baked into the answer.

A real corner is never a mathematical point. Moulded plaster is rounded and chipped, and at the distance this photograph was taken that rounding is too small to see in a 21x21 window centred on the corner. Move the camera closer and everything in the window grows: the rounding becomes a gently curving edge that fills the window, and $\lambda_2$ falls toward the edge case. Move it farther away and the same window takes in the base, the column and whatever sits next to them, and the response blends several structures into one number. The corner did not change; its size relative to the window did. I have not measured how far the camera has to move before the verdict flips.

No response function fixes it, because all three read a matrix that was already built at one scale. A detector tied to one window size finds features of one size, and two photographs taken from different distances will not agree on a single point.

The second limit is subtler and shows in the lab above: a corner is found to the pixel, and a pixel is not a description. Two corners that look identical to the structure tensor are indistinguishable, so even a repeatable detector gives you nothing to match with.

Exercises

Shi-Tomasi without an eigensolver. Write smaller_eig(M), the smaller eigenvalue of a 2x2 symmetric matrix, from the closed form in “Three responses, one matrix”. It must match np.linalg.eigvalsh on the worked example’s edge and corner matrices (0 and 0.0625).

Your turn

def smaller_eig(M):
    return 0.0  # the smaller eigenvalue of M

edge = np.array([[0.25, 0.0], [0.0, 0.0]])
corner = np.array([[0.125, 0.0625],
                   [0.0625, 0.125]])
print(smaller_eig(edge), smaller_eig(corner))

Find where the constant takes over. The temple’s column edge measured $\lambda_1 = 0.411$ and $\lambda_2 = 0.019$ . Harris called it a corner at $k = 0.04$ and an edge at $k = 0.06$ . Solve $R = 0$ for $k$ to find where it flips.
Move the window, not the camera. The comparison cell also prints near, the point (56, 50) on the square’s left side, six pixels below the corner. Set win to 9, 11, 13 and 15 in turn and rerun both cells. Before running, say at which size its verdict changes from edge to corner.

What you should see

These come from the corner cells of the unit notebook, edge_detection.ipynb, and are the same on OpenCV 4.11 (this page) and 5.0.

0 and 0.0625.
$k = \lambda_1 \lambda_2 / (\lambda_1 + \lambda_2)^2 = 0.007809 / 0.1849 = 0.0422$ . Any $k$ below that accepts this edge as a corner, and 0.04 sits 0.002 under the line.
$\lambda_2$ is 0 for windows of 5, 9 and 11, then 0.039 at 13 and 0.067 at 15. A 13-pixel window reaches six pixels out, far enough to take in the corner, and the same point becomes one. That is “Where this breaks” on a drawn shape: the point stays where it was while the window grows around it.

Check yourself

Four questions on this lesson. Pick an answer to see why it is right or wrong.

A window gives λ₁ = 0.411 and λ₂ = 0.019. What kind of patch is it?
With those eigenvalues, Harris scores +0.000413 at k = 0.04 and −0.003285 at k = 0.06. What does that tell you?
What does Shi-Tomasi score a patch by?
A corner found with a 21×21 window. The camera moves much closer. What happens to the verdict?

Both are where the next unit starts. Searching for the same feature across a range of window sizes is what a scale space is for (built from the difference-of-Gaussians approximation to the Laplacian of Gaussian), and summarising a patch as a vector of numbers is what a descriptor is. SIFT does both [7], and its first lesson measures these corners against SIFT’s own points.

That closes the Edge Detection unit: from what counts as an edge, through the two derivatives that find one, to Canny’s repair, and finally to the points that survive being looked at twice.

References

[1] Harris, C. G., & Stephens, M. (1988). A Combined Corner and Edge Detector. Proceedings of the Alvey Vision Conference 1988, pp. 1-6. doi:10.5244/C.2.23

[2] Shi, J., & Tomasi, C. (1994). Good features to track. CVPR 1994, pp. 593-600. doi:10.1109/CVPR.1994.323794

[3] Förstner, W., & Gülch, E. (1987). A Fast Operator for Detection and Precise Location of Distinct Points, Corners and Centres of Circular Features. Proceedings of the ISPRS Conference on Fast Processing of Photogrammetric Data, Interlaken, pp. 281-305. No DOI is published for this paper.

[4] Gonzalez, R. C., & Woods, R. E. (2018). Digital Image Processing (4th ed.), ch. 12 “Feature Extraction”. Pearson. Chapter title verified against the publisher’s detailed table of contents; no section-level locator is quoted here because none was checked.

[5] Szeliski, R. (2022). Computer Vision: Algorithms and Applications (2nd ed.), ch. 7 “Feature Detection and Matching”. Springer. Free PDF

[6] Seitz, S. M., Curless, B., Diebel, J., Scharstein, D., & Szeliski, R. (2006). A Comparison and Evaluation of Multi-View Stereo Reconstruction Algorithms. CVPR 2006, pp. 519-528. doi:10.1109/CVPR.2006.19. The Middlebury multi-view datasets: vision.middlebury.edu/mview

[7] Lowe, D. G. (2004). Distinctive Image Features from Scale-Invariant Keypoints. International Journal of Computer Vision, 60(2), 91-110. doi:10.1023/B:VISI.0000029664.99615.94

Tagged