Image Formation: How a Camera Flattens the World Onto a Sensor

How a three-dimensional world ends up flattened onto a sensor, and what the piece of glass in front of that sensor does to it on the way.

That is the unit. Four lessons: the geometry that makes a picture sharp, the lens that makes it bright and charges a focus plane for it, the ways real glass departs from that model, and the point past which the departure is the design rather than a defect.

Before this

Nothing. This is the first unit on the Fundamentals track and it assumes only that you have seen an image as an array of numbers. If you want that part first, read the image as data, which covers the array, its grid and its colour axes.

What comes after it is 2D convolution and then edge detection, both of which assume the array in front of them came from a camera that behaves like the model below.

The photographs everything is measured on

Three public sets, one per problem, and every number on this page comes out of them rather than out of a textbook.

Source	Used for	What it gives
OpenCV’s own chessboard photographs, samples/data/left*.jpg	lessons 1 and 3	13 views of a board with 9×6 inner corners, 640×480, Apache-2.0
One published scene from the DPDD dataset [7]	lesson 2	the same bench at f/4 and f/22, 93 mm, focused at about 1.5 m
15 fisheye views of a board with 6×8 inner corners, from the py-OCamCalib test set	lesson 4	a lens wide enough to break the model taught in lessons 1 to 3

The chessboard set is the spine. It is thirteen ordinary photographs of a printed board held at different angles, and calibrating on it [5] yields the camera every lesson refers back to:

Parameter	Value
$f_x$, $f_y$	536.07 px, 536.02 px
$c_x$, $c_y$	342.37 px, 235.54 px
$k_1$, $k_2$, $k_3$	−0.2651, −0.0467, 0.2523
RMS reprojection error	0.4087 px over 13 views

Two things in that table are worth pausing on before any lesson starts. The principal point sits 22 px right of centre and 4 px above it, so the optical axis does not pass through pixel (320, 240) even on a camera nobody dropped. And $k_1$ is negative, which is the signature of barrel distortion: the lens pulls the corners of the scene inward.

Topics

1. Pinhole and perspective projection. Why does a small hole produce a sharp picture at all, and what does the resulting projection do to parallel lines and to size with distance? The similar-triangles model, $x = fX/Z$, and how far it lands from where the corner actually is [1].
2. Lenses, focal length and depth of field. A hole small enough to be sharp is too dark to use. A lens fixes the light and introduces a plane of focus, so what exactly is in focus and what is not [1]?
3. Distortion, vignetting and aberration. Real glass bends rays more at the edges than the thin-lens model allows. How far does that move a pixel [3], and what does straightening it back cost you [2]?
4. Wide-angle and fisheye models. Past roughly 120° the correction stops working, because $\tan\theta$ runs to infinity and the field of view does not. What replaces it [4]?

The order is a ramp. Each lesson ends at the thing the next one repairs, so lesson 2 exists because lesson 1’s model is unusable in practice, and lesson 4 exists because lesson 3’s correction runs out.

How they fit together

What the unit measures

Result	Measurement
A real lens moves a pixel a long way	The measured coefficients displace the worst image corner by 51.22 px on a 640×480 frame
Undistorting does not crop this lens, it re-frames it	Keeping the same $f$, 0.00% of the output has no source pixel, and the view stays at 61.67° while the lens actually captured 67.32°
Stopping down deepens focus by the aperture ratio	Depth of field goes from 0.050 m at f/4 to 0.277 m at f/22, a factor of 5.54 against an aperture ratio of 5.50
The wrong camera model costs an order of magnitude	On one fisheye lens, pinhole plus Brown–Conrady fits to 7.326 px RMS and Kannala–Brandt to 0.644 px, a factor of 11.4

Lesson 4 works the last row out in detail. Out to 45° the two models stay within 17% of each other and both grow smoothly. At 80° the pinhole prediction is −35,572 px, a sign flip rather than a large error, while the fisheye model keeps climbing to 458.8 px and then 502.1 px at 89°.

Run it: every code block on this page has a cell in the unit’s notebook — open it in Colab.

Reproducibility

Parameter	Value
CPU	12th Gen Intel Core i7-12700H, 20 threads
GPU	none used; every fit here runs on the CPU
RAM / OS	31 GB · Ubuntu 22.04.5 LTS, kernel 6.8.0-136
Key versions	Python 3.12.9, OpenCV 5.0.0 [6], NumPy 2.5.2
Data	OpenCV samples/data/left*.jpg (13 views, Apache-2.0) · py-OCamCalib test_images/fish_1 (15 views, GPL-2.0, downloaded not redistributed) · capture settings read off DPDD’s figures/data_example.png [7]
Commands	uv run formation-download, then uv run formation-calibrate, formation-distort, formation-dof, formation-fisheye; the photographs come from uv run data-experiments
In the wild	Four Wikimedia Commons files, all CC0 — a temple cloister, a market stall, a brick wall and a circular fisheye frame. Registered in CondadosAI/cv-assets with the licence read from the Commons API and a sha256 pinning the bytes measured; downloaded, never redistributed
Runs	every result is a deterministic fit, run once; no averaging, no seed
Excluded	no timing is reported in this unit; every number is a geometric quantity or a reprojection error

Every number above is a key in output/formation_numbers.json; the “in the wild” figures are keys in output/image_data_numbers.json under in_the_wild_other_units. The lessons themselves carry no reproducibility table — this one covers all four, which is how unit 3.1 has always worked and how the whole track now works.

Each lesson also repeats its measurement on a photograph, because these four were measured on calibration targets and a calibration target is not a picture anybody takes. Lesson 1 fits a vanishing point to 53 lines of a temple cloister and lands within 1.6% of the frame diagonal; lesson 2 reads a Nikon’s own EXIF and predicts 0.59 m of depth at 300 mm and f/4; lesson 3 measures 0.22 stops of falloff on a brick wall; lesson 4 predicts a fisheye’s image circle at 1910 px and measures 1752.

Limitations & caveats

• Two cameras, not a survey. The intrinsics, the distortion and the field-of-view results describe one lens, and the fisheye comparison describes one other. A different lens has different coefficients, and nothing here says how common these values are.
• The board’s physical square size is not published, so the calibration runs in board units. That leaves $f_x$, $f_y$, the principal point and every distortion coefficient unchanged, and it means no translation in this unit carries metres.
• Depth of field is computed, not observed. The f/4 interval is reproduced from the thin-lens formula and matches the published figure, which is a consistency check rather than a measurement of blur in the pixels.
• One row of the source figure does not reconcile. DPDD’s caption gives a f/22 depth of field of 0.70–5.80 m, which its own 93 mm focal length cannot produce; reproducing that interval would need a 29.1 mm lens at 1.249 m. Whether the row describes a different scene, a dataset-wide range or an error is not something the figure settles, so lesson 2 states the discrepancy rather than choosing for it.
• Reprojection error is not accuracy. It measures how well a model fits the points it was fitted to. A model with more free parameters will always score better on its own data, which is part of why the fisheye comparison uses two models with a comparable number of them.

Where this lands

This unit has no measured closer yet. The natural one is what undistortion costs per frame on real hardware, comparing cv2.remap with a precomputed map against cv2.undistort per frame against fisheye rectification, at 1080p on an integrated GPU. That post does not exist, and the unit ships without it rather than waiting.

What does depend on this unit is everything downstream that assumes a rectilinear image: SIFT matching points across views, RANSAC fitting a model to those matches, and structure from motion, which needs a calibrated camera before it can triangulate anything.

Further reading

• Go deeper: Forsyth & Ponce, Computer Vision: A Modern Approach (2nd ed.), ch. 1 “Geometric Camera Models” [1]. Sections 1.1.1 and 1.1.3 cover lessons 1 and 2 in twelve pages, and its split between §1.1 and §1.2 is the same boundary this unit draws.
• For the algebra: Hartley & Zisserman, ch. 6 “Camera Models” [2], once you want the projective form rather than the two scalar equations.
• Related on CondadosAI: camera models, calibration and PnP is the sequel that turns this geometry into a matrix · the image as data is what the sensor’s output turns into.

References

[1] Forsyth, D. A., & Ponce, J. (2012). Computer Vision: A Modern Approach (2nd ed.), ch. 1 “Geometric Camera Models”, §1.1.1 “Pinhole Perspective” (p. 4) and §1.1.3 “Cameras with Lenses” (p. 8). Pearson.

[2] Hartley, R., & Zisserman, A. (2004). Multiple View Geometry in Computer Vision (2nd ed.), ch. 6 “Camera Models” (p. 153) and §7.4 “Radial distortion” (p. 189). Cambridge University Press.

[3] Brown, D. C. (1971). Close-range camera calibration. Photogrammetric Engineering, 37(8), 855–866.

[4] Kannala, J., & Brandt, S. S. (2006). A Generic Camera Model and Calibration Method for Conventional, Wide-Angle, and Fish-Eye Lenses. IEEE Transactions on Pattern Analysis and Machine Intelligence, 28(8), 1335–1340. doi:10.1109/TPAMI.2006.153

[5] Zhang, Z. (2000). A flexible new technique for camera calibration. IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(11), 1330–1334. doi:10.1109/34.888718

[6] OpenCV Documentation (5.0). Camera calibration and 3D reconstruction and Fisheye camera model. docs.opencv.org/5.0

[7] Abuolaim, A., & Brown, M. S. (2020). Defocus Deblurring Using Dual-Pixel Data. European Conference on Computer Vision (ECCV) 2020. arXiv:2005.00305