What Is Camera Calibration? Finding K from Photos of a Chessboard

Camera calibration measures the numbers that describe a camera, its focal length, its image centre and how its lens bends straight lines, from photographs of an object whose shape is known. Below, a photograph straightened with those numbers once they are known.

You have met it wherever a camera is trusted to measure:

• Phone makers calibrate every camera module on the production line, and the phone’s AR and portrait features read those numbers.
• Robots, cars and drones calibrate their cameras before any distance they report means anything.
• Photogrammetry and 3-D scanning calibrate first or estimate the same numbers along the way.
• cv2.calibrateCamera, the OpenCV call this lesson runs on the page.

Calibration finds a camera’s own numbers by fitting the camera model to many photos of a flat board whose corners are known, and the fit is only as good as the photos it is given. Thirteen photos fit OpenCV’s sample camera to 0.41 px; any three of them leave the focal length anywhere from 520 to 568 px.

Where we are

Lesson 3 of camera geometry, and step 6 of its pipeline with many photos: compare the corners the model predicts with the ones the detector found, and adjust the camera until they agree. Lesson 2 used a $K$ and a lens that were given. This lesson is where they come from [1] [2].

Why one photo is not enough

A flat board in one photo gives a homography between the board and the image: eight numbers. The camera needs eleven, five in $K$ and six for where it stood, so one photo leaves three unknown. Zhang’s method, the one behind cv2.calibrateCamera, shows that every photo of the board adds two equations about $K$ alone, because the board’s two axes are at right angles in the world. Three photos at different angles fix $K$; more of them average out the noise of corner detection, and the lens coefficients come from the corners near the edges [1].

What calibration minimises

The fit has one score, the reprojection error: push every known board corner through the camera model, measure how far each lands from where the detector found it, and take the root mean square over every corner of every photo. Calibration adjusts $K$, the five lens coefficients and one $[R \mid t]$ per photo until that number is as small as it gets [2].

How close is this to OpenCV?

The page calibrates for real with cv2.calibrateCamera [3], from the corners the detector found in each of the thirteen photos (served as a 12 KB file, because detecting them takes the full images). Then it asks the question the reprojection error cannot answer on its own: how does the calibration do on a photo it never saw?

import json
import numpy as np
import cv2

d = json.load(open("corners.json"))
# the board's own coordinates, one unit per square
obj = np.zeros((54, 3), np.float32)
obj[:, :2] = np.mgrid[0:9, 0:6].T.reshape(-1, 2)
img = [np.float32(c).reshape(-1, 1, 2)
       for c in d["corners"]]
size = tuple(d["image_size"])

rms, K, dist, rvecs, tvecs = cv2.calibrateCamera(
    [obj] * len(img), img, size, None, None)
print(len(img), "views, RMS", round(rms, 4), "px")
print("fx, fy:", K[0, 0].round(2), K[1, 1].round(2))
print("cx, cy:", K[0, 2].round(2), K[1, 2].round(2))
print("k1, k2, p1, p2, k3:", dist.ravel().round(4))

13 views, RMS 0.4087 px
fx, fy: 536.07 536.02
cx, cy: 342.37 235.54
k1, k2, p1, p2, k3: [-0.2651 -0.0466  0.0018 -0.0003  0.252 ]

# a view the fit never saw: calibrate on the other 12,
# then place the first board with that camera
K12, d12 = cv2.calibrateCamera(
    [obj] * 12, img[1:], size, None, None)[1:3]
ok, rv, tv = cv2.solvePnP(obj, img[0], K12, d12)
p, _ = cv2.projectPoints(obj, rv, tv, K12, d12)
e = np.sqrt(np.mean(np.sum(
    (p - img[0]).reshape(-1, 2) ** 2, axis=1)))
print("first view, held out:", round(float(e), 3), "px")

# the same 13 views with no lens model at all
flags = (cv2.CALIB_FIX_K1 | cv2.CALIB_FIX_K2
         | cv2.CALIB_FIX_K3 | cv2.CALIB_ZERO_TANGENT_DIST)
rms0, K0 = cv2.calibrateCamera(
    [obj] * 13, img, size, None, np.zeros(5),
    flags=flags)[:2]
print("pinhole only: RMS", round(rms0, 3),
      "px, fx", K0[0, 0].round(2))

first view, held out: 0.2 px
pinhole only: RMS 1.555 px, fx 557.45

The page’s OpenCV 4.11 returns the same 0.4087 px and $f_x = 536.07$ as the repository’s 5.0. Held out, the first photo scores 0.20 px, about what it scored when it was part of the fit: over all thirteen, leaving each out in turn gives a median of 0.204 px against 0.194 in-sample. Dropping the lens model costs a factor of almost four, 1.555 px, and pushes the focal length 21 px off to absorb the bend.

In the wild

The same fit on a different lens shows what a wrong model looks like. Unit 1.1’s fisheye, fifteen photos of a board, calibrated with the model above, fits to 7.326 px; with a fisheye model, cv2.fisheye.calibrate, the same corners fit to 0.644 px (lesson 4 of unit 1.1). A reprojection error several times larger than the corner detector’s own accuracy is a model that does not describe the lens, whatever the number of photos.

Where this breaks

Too few photos. Every one of the 286 ways to choose three of the thirteen photos gives a calibration, and their focal lengths run from 519.6 to 567.9 px; the middle 90% from 530.2 to 544.3. Each of those fits has a small reprojection error on its own photos. The error says how well the model fits what it was shown, not whether what it was shown was enough.

Photos that all look alike. The equations about $K$ come from the board being seen at different angles. A set of photos taken straight on, or all tilted the same way, leaves the focal length and the board’s distance trading off against each other.

Board squares, not metres. The square size of OpenCV’s board is not published, so $K$ and the lens are right and every distance is in squares. A real calibration measures the square.

Exercises

1. The board’s coordinates. Write board(cols, rows), the 3-D positions of a board’s inner corners in squares, in the order the detector returns them.

def board(cols, rows):
    return np.zeros((cols * rows, 3), np.float32)

print(board(9, 6)[:3])

2. Predict, then run. Calibrate on three photos at a time: views (0, 1, 2), (0, 5, 10) and (3, 7, 11). How close do they come to the thirteen-photo $f_x$?
3. Read the table above. Which is the larger risk on this set: too few photos, or no lens model?

Next

With $K$ and the lens known, one photo of the board is enough to find where the camera stood: lesson 4, PnP.

References

[1] Zhang, Z. (2000). A flexible new technique for camera calibration. IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(11), 1330–1334. doi:10.1109/34.888718

[2] Szeliski, R. (2022). Computer Vision: Algorithms and Applications (2nd ed.), §11.1 “Geometric intrinsic calibration” (p. 685). Springer. Free PDF

[3] OpenCV Documentation (5.0). Camera Calibration and 3D Reconstruction (calib3d): calibrateCamera, findChessboardCorners, cornerSubPix. docs.opencv.org/5.0