Lens Distortion: Where the Straight-Line Model Actually Fails

Real lenses bend straight lines, most of all near the edges of the picture, and software can straighten them again once the bend has been measured. Below, one photograph eases between the way it was taken and its corrected version.

Distortion is corrected more often than you notice:

• Phone and action cameras correct it before you see the photo; action cameras let you choose a “linear” mode that does it for the very wide lens.
• Lens profiles in photo editors apply a correction measured for each lens model.
• Measurement from images: a camera is calibrated and its distortion removed before any distance is read off a picture (camera calibration).

A real lens bends rays more the further from its axis they arrive, so the pinhole model is right in the middle of the frame and wrong at the edges, and a few measured coefficients undo it. On the unit’s camera the top-left pixel sits 51.22 px from where the straight-line model puts it, and five coefficients take one view’s mean error from 2.55 px to 0.170 px.

Where we are

Lesson 3 of image formation, and step 3 of its pipeline: what the glass does to each ray. Lesson 1 left a residual it could not explain: projecting board corners with the two scalar equations was accurate to 0.29 px near the middle of the frame and off by 5.75 px near its edges. Lesson 2 explained what the lens buys, assuming it bends every ray by the right amount. It does not, and the difference is what lesson 1 measured.

The error is radial, so the model is a function of radius

In a real lens the magnification is not constant across the field. A ray arriving far off-axis is imaged at a slightly different scale from one near the axis, so a straight line that crosses the frame comes out bowed. A lens is built rotationally symmetric about its axis, so that change of scale can depend only on how far off-axis the ray lands and not on the direction it came from, which is why the correction is written as a function of radius alone [2].

Work in coordinates centred on the optical axis and scaled by focal length, so the image is described independently of its resolution:

The Brown–Conrady model [1] then multiplies by an even polynomial in $r$ and adds two small terms for a lens that is not perfectly parallel to the sensor:

Only even powers appear because the factor has to depend on radius alone and still be a smooth function of position, and the smooth functions of $x$ and $y$ that depend only on radius are functions of $r^2 = x^2 + y^2$. The polynomial equals 1 at $r = 0$, so the centre of the image never moves no matter how bad the lens is.

The whole behaviour is in that $r^6$: the displacement is negligible for most of the frame and then climbs steeply.

Now you try

The lab opens on this camera’s measured coefficients. Drag $k_1$ toward zero and the red grid straightens; push it positive and the bow reverses into pincushion. The tangential sliders are worth a moment too: they have to be pushed a hundred times past this lens’s values before they do anything visible.

Undistorting costs field of view

With the coefficients known, the correction is one OpenCV call, and for video the map is worth precomputing once [3]:

import cv2

undistorted = cv2.undistort(img, K, dist)          # one frame

h, w = img.shape[:2]                                # for video, build the map once
new_k, _ = cv2.getOptimalNewCameraMatrix(K, dist, (w, h), 0)
map1, map2 = cv2.initUndistortRectifyMap(K, dist, None, new_k, (w, h), cv2.CV_16SC2)
undistorted = cv2.remap(img, map1, map2, cv2.INTER_LINEAR)

#include <opencv2/opencv.hpp>

cv::Mat undistorted;
cv::undistort(img, undistorted, K, dist);           // one frame

cv::Mat map1, map2;                                  // for video, build the map once
cv::Mat new_k = cv::getOptimalNewCameraMatrix(K, dist, img.size(), 0);
cv::initUndistortRectifyMap(K, dist, cv::noArray(), new_k, img.size(),
                            CV_16SC2, map1, map2);
cv::remap(img, undistorted, map1, map2, cv::INTER_LINEAR);

The usual warning is that undistortion crops the image. On this lens it does not. Counting output pixels with no source pixel to read from:

Keeping the same focal length wastes nothing and shows 61.67°, while the lens actually collected 67.32°. Barrel distortion squeezes the outer scene inward, so the pixels are there; correcting at the same $f$ pushes them off the edge. Asking OpenCV for the widest view that is still fully covered recovers those 5.65° for 0.03% empty output. The choice is how much field of view to give up against how much black to allow in the corners.

The same correction pays off downstream. What is a homography fits a one-parameter radial model to the painted lines of a court camera, straightens a baseline that bowed by 7.7 px to 0.16 px, and cuts its far-sideline error from 5.95 cm to 1.96 cm without touching the homography itself.

How close is this to OpenCV?

The worked example in cells, against cv2.projectPoints, which applies the same five coefficients, and cv2.undistortPoints, which runs them backwards. Then the whole photo, straightened.

import numpy as np

fx, fy = 536.0735, 536.0164
cx, cy = 342.3705, 235.5369
k1, k2, k3 = -0.26509, -0.04674, 0.25231
p1, p2 = 0.00183, -0.00031

# the top-left pixel, centred and scaled by f
x, y = (0 - cx) / fx, (0 - cy) / fy
r2 = x * x + y * y
radial = 1 + k1 * r2 + k2 * r2**2 + k3 * r2**3
xd = x * radial + 2 * p1 * x * y + p2 * (r2 + 2 * x * x)
yd = y * radial + p1 * (r2 + 2 * y * y) + 2 * p2 * x * y
ud, vd = cx + fx * xd, cy + fy * yd
print(f"radial factor {radial:.5f}")
print(f"lands at ({ud:.2f}, {vd:.2f}), "
      f"{np.hypot(ud, vd):.2f} px from (0, 0)")

radial factor 0.87857
lands at (41.89, 29.48), 51.22 px from (0, 0)

Against projectPoints and undistortPoints Reset ▶ Run

import cv2

K = np.array([[fx, 0, cx], [0, fy, cy], [0, 0, 1]])
dist = np.array([k1, k2, p1, p2, k3])

# OpenCV's forward model on the same ray
p, _ = cv2.projectPoints(np.array([[x, y, 1.0]]),
                         np.zeros(3), np.zeros(3), K, dist)
print("cv2.projectPoints:", p.ravel().round(2))

# and back: undistortPoints inverts it iteratively
back = cv2.undistortPoints(p.reshape(1, 1, 2), K, dist, P=K)
print("cv2.undistortPoints:", back.ravel().round(2))

img = cv2.imread("left01.webp")
show(np.hstack([img, cv2.undistort(img, K, dist)]),
     "left01: as taken, and straightened")

Output

cv2.projectPoints: [41.89 29.48]
cv2.undistortPoints: [0. 0.]

OpenCV lands the corner at the same (41.89, 29.48), and undistorting that pixel returns (0, 0). The model has no inverse formula; undistortPoints iterates until it gets back.

In the wild

Vignetting is easy to assert and awkward to measure, because you need a subject that is close to uniform. A brick wall photographed square-on is close enough: it fills the frame, it is flat, and it was lit by the sky.

The camera is a Sony DSC-W530 compact at 4.7 mm and f/2.7, which its EXIF records; that is the wide end of its zoom, wide open. Sampling a patch at the centre and one in each corner of the frame, as the JPEG’s own code values:

The worst corner sits at 86% of the centre in code values, and $\log_2(133.4/114.7)$ is 0.22 stops. That figure is in the file’s encoded values, though, and a JPEG’s values are gamma-encoded rather than proportional to light (the camera response curve lesson measures that curve). Decoding both numbers through the standard sRGB curve first:

Treat that as an estimate of about half a stop in linear light rather than a measurement. The decode was applied to patch means rather than to each pixel, the camera’s tone curve is not guaranteed to be sRGB, and a compact may already apply some shading correction before it writes the file.

What it means: about half a stop is modest, and the sign of the error never changes: the corners are always darker. Stack a hundred frames for a panorama and the seams show; measure reflectance across a field and the edges read dark; skip flat-field correction on a microscope image and every measurement inherits the same bias. Some of the falloff could also be the light on the wall rather than the lens, which a single photograph cannot separate.

Where this breaks

The polynomial is a fit. $k_3$ especially is often noise: with a board that does not reach the corners of the frame there is nothing constraining the $r^6$ term, and fitting it anyway can make predictions worse outside the region that was measured. OpenCV provides CALIB_FIX_K3 for this reason, and a calibration whose coefficients change a lot when you add images has not converged.

Everything above assumes the distortion is radial about one centre. Vignetting, the darkening toward the corners, is also radial but acts on brightness rather than position, so none of these coefficients touch it. Chromatic aberration is worse for this model. Glass bends blue light more than red, so a lens brings the colours to focus at slightly different distances (longitudinal aberration, which shows as coloured blur) and at slightly different magnifications (lateral aberration, which shifts each channel radially by its own amount). The lateral part means the correct displacement differs per channel, and a single set of coefficients fitted on greyscale corners splits the difference.

And the model has a hard ceiling. Everything here is a correction to $x = X/Z$, which is $\tan\theta$ for a ray arriving at angle $\theta$. As $\theta$ approaches 90° that tangent runs to infinity, and well before then the stretch toward the edges grows as $1/\cos^2\theta$: four times at the edge of a 120° field. Past about 120° a polynomial in $r$ is no longer a practical description of the lens, and adding terms does not help, because the quantity being corrected is the wrong one for a fisheye.

Exercises

1. How far does the lens move a pixel? Write shift_px(u, v), the distance in pixels between where the straight-line model puts pixel (u, v) and where this lens puts it.

def shift_px(u, v):
    return 0.0  # how far the lens moves pixel (u, v)

print(shift_px(0, 0))

2. Predict, then run. Rank three pixels by how far they move: the left edge of the middle row, a point halfway to the top-left corner, and the corner itself.
3. Drop the tangential terms. How much of the corner’s 51.22 px is left with $k_1$, $k_2$ and $k_3$ alone?

Next

Past that ceiling the correction stops being a correction and the projection itself has to change. Wide-angle and fisheye models fits both models to one very wide lens and measures how badly the one from this lesson does.

Run it: every code block on this page has a cell in the unit’s notebook: open it in Colab.

References

[1] Brown, D. C. (1971). Close-range camera calibration. Photogrammetric Engineering, 37(8), 855–866.

[2] Hartley, R., & Zisserman, A. (2004). Multiple View Geometry in Computer Vision (2nd ed.), §7.4 “Radial distortion” (p. 189). Cambridge University Press.

[3] OpenCV Documentation (5.0). Camera calibration and 3D reconstruction. docs.opencv.org/5.0