The Pinhole Model: Why a Hole Makes a Picture, and Where the Prediction Fails

The pinhole model says where any point in front of a camera lands in the picture, and that is what lets software draw an object into a photograph so that it looks as if it is standing there. Below, a cube is drawn onto a chessboard in each of thirteen photographs.

Where the same projection is at work:

• AR apps and filters that place a virtual object on a table or a pair of glasses on a face.
• Sports broadcasts, where lines and graphics drawn onto the playing field follow the camera as it pans and zooms.
• Robots and drones that turn a point in the image back into a direction in the world.

A pinhole camera sends every point along a straight ray, so where it lands depends only on its sideways offset divided by its distance. On one of OpenCV’s calibration photos that prediction is within 0.29 px of the corners near the middle of the frame and off by 5.75 px near its edges, which is the lens bending the rays.

Where we are

Lesson 1 of image formation, and steps 2 and 4 of its pipeline: a point’s ray, and the pixel it lands on. Nothing on the track comes before it. The unit’s job is to explain how a three-dimensional scene becomes a grid of numbers, and the first question is why any of it is sharp.

A hole is a filter on rays

Point a bare sensor at a room and every photosite receives light from everywhere. Each cell sums the whole room, so the result is a uniform grey with no picture in it. Put a barrier with a small hole in front, and each point on the sensor can only be reached from one direction. That restriction is what makes an image: the hole turns “light from the whole room” into “light from one direction per pixel” [1].

Once each pixel corresponds to one ray, the geometry is a pair of similar triangles. A point sitting $X$ to the side at depth $Z$ projects onto the image plane at distance $f$ behind the hole, and the two triangles share an angle:

Sensors are indexed in pixels rather than millimetres, and the optical axis does not necessarily meet the sensor at its middle. Writing the focal length in pixels as $f_x, f_y$ and the point where the axis crosses the sensor as $(c_x, c_y)$:

Two scalar equations, one division each. Every camera model in computer vision is built on top of these [2], and the rest of this unit is about the ways a real camera departs from them.

Checking it on a real corner

The unit’s camera comes from 13 photographs of a printed chessboard [4], which give $f_x = 536.07$, $f_y = 536.02$ and a principal point at (342.37, 235.54). Note that the principal point is 22 px right of the frame’s centre. The optical axis and the middle of the sensor are different places, and assuming otherwise would start every prediction up to 22 px off before anything else went wrong.

Calibration also recovers where the board was in each shot. Take that as given for now, since recovering pose is the subject of PnP, and look only at what happens after: a corner whose position in the camera’s own frame is known, projected by the two equations above.

Averaged over all 54 corners in that photograph, the pinhole equations miss by 2.55 px. Splitting the corners by how far they sit from the principal point shows what that average is hiding:

Near the middle of the frame the model is right to about a quarter of a pixel. Out at the edges it is twenty times worse. Whatever is wrong grows with distance from the centre, which is a strong hint about its cause and the subject of lesson 3.

Written out, the prediction is two lines of arithmetic:

# (X, Y, Z) is the corner in the camera's own frame; fx, fy, cx, cy from calibration.
u = cx + fx * X / Z
v = cy + fy * Y / Z

// (X, Y, Z) is the corner in the camera's own frame; fx, fy, cx, cy from calibration.
const double u = cx + fx * X / Z;
const double v = cy + fy * Y / Z;

What perspective does to a scene

Because the division is by $Z$, two things follow that the human eye takes for granted. An object twice as far away covers half as many pixels, so size on the sensor carries distance. And lines that are parallel in the world stop being parallel in the image unless they are parallel to the sensor, because each point along them is divided by a different $Z$.

The same division is why one photograph cannot tell you how big anything is: doubling both an object’s size and its distance leaves $X/Z$ unchanged, so a small near object and a large far one produce identical pixels.

Now you try

Move the ball and the camera and watch the projected position follow the two equations. Then switch to the second panel, which holds $X/Z$ fixed while changing both: two very different scenes, one photograph.

How close is this to OpenCV?

The two equations are one line each, and cv2.projectPoints is OpenCV’s version of the same step, with or without the lens’s bend. The cells use the worked corner as printed above.

import numpy as np

# the calibrated camera, from the unit's 13 photos
fx, fy = 536.0735, 536.0164
cx, cy = 342.3705, 235.5369

# one board corner in the camera's own frame
X, Y, Z = 4.68658, -4.06741, 13.83412

u = cx + fx * X / Z
v = cy + fy * Y / Z
print(f"pinhole: ({u:.2f}, {v:.2f})")
print("measured: (513.77, 86.53)")

pinhole: (523.98, 77.94)
measured: (513.77, 86.53)

import cv2

K = np.array([[fx, 0, cx], [0, fy, cy], [0, 0, 1]])
dist = np.array([-0.26509, -0.04674, 0.00183,
                 -0.00031, 0.25231])
pt = np.array([[X, Y, Z]])
zero = np.zeros(3)

# no distortion: the same two equations
p, _ = cv2.projectPoints(pt, zero, zero, K, None)
print("cv2, pinhole only:", p.ravel().round(2))
# with the lens's measured bend
q, _ = cv2.projectPoints(pt, zero, zero, K, dist)
print("cv2, with distortion:", q.ravel().round(2))
for name, a in (("pinhole", p), ("with bend", q)):
    e = np.hypot(*(a.ravel() - [513.77, 86.53]))
    print(f"{name:9} misses the corner by {e:.2f} px")

cv2, pinhole only: [523.98  77.94]
cv2, with distortion: [514.05  86.72]
pinhole   misses the corner by 13.34 px
with bend misses the corner by 0.34 px

With no distortion terms OpenCV returns the same pixel to the hundredth. Give it the five coefficients lesson 3 measures and the miss falls from 13.34 px to 0.34 px.

In the wild

The pinhole model’s central claim is that parallel lines in the world meet at a single point in the image. That is testable on any photograph with architecture in it, and it does not need a calibration target. How to compute that point from two lines is one cross product, which is the subject of homogeneous coordinates.

A temple cloister, shot straight down its axis. Detect line segments with OpenCV’s own Hough transform [3], keep the 60 longest that are neither vertical nor horizontal (those are the receding ones, since a building’s own uprights meet only at infinity), and find the point that is closest to all of them at once.

What it means: 53 lines from stone that was cut by hand, photographed on a phone, agree on one point to within 1.6% of the frame. The residual has ordinary causes: masonry that is not perfectly straight, a lens that is not perfectly rectilinear (the subject of lesson 3), and segments fitted to a few hundred pixels each. The projection this lesson derives from a hole in a box survives contact with a real building.

Where this breaks

The error is radial, and it is not small. The table above is the failure: 0.29 px in the middle of the frame against 5.75 px at the edges, with a worst corner 13.34 px out. Adding the lens terms that lesson 3 covers drops the same view’s mean error from 2.55 px to 0.170 px, with a worst corner of 0.404 px. More care in calibration would not have closed that gap: the pinhole equations are missing a term that only matters away from the axis.

Depth is gone and cannot be recovered from one image. The projection maps a three-dimensional world onto two dimensions, and the information lost is not retrievable by looking harder. Everything the geometry branch of this track does later, from stereo to structure from motion, exists to put it back.

And a pinhole sharp enough for this is too dark to use. The whole model above assumes each pixel sees exactly one ray, which requires a hole of essentially no size. A hole of no size passes no light, so the exposure time runs to minutes and anything that moves is a smear. Open the hole to get light back and each sensor point starts collecting a cone of rays instead of one, which is blur. That trade is the reason no camera you own is a pinhole camera.

Exercises

1. Project a point. Write project(X, Y, Z), the pixel the pinhole equations give for a point in the camera’s frame, with fx, fy, cx, cy from the cells above.

def project(X, Y, Z):
    return (0.0, 0.0)  # pixel (u, v), pinhole only

print(project(X, Y, Z))

2. Predict, then run. Move the same corner twice as far from the camera. How far from the image centre does it land now?
3. Where is the centre? How far is the optical centre $(c_x, c_y)$ from the middle of the 640×480 frame?

Next

A lens collects a wide cone of light and bends it back to a point, buying the brightness a pinhole cannot have. It charges for it with a plane of focus, and lenses, focal length and depth of field is about what is in focus once you have one.

Run it: every code block on this page has a cell in the unit’s notebook: open it in Colab.

References

[1] Forsyth, D. A., & Ponce, J. (2012). Computer Vision: A Modern Approach (2nd ed.), ch. 1 “Geometric Camera Models”, §1.1.1 “Pinhole Perspective” (p. 4). Pearson.

[2] Hartley, R., & Zisserman, A. (2004). Multiple View Geometry in Computer Vision (2nd ed.), ch. 6 “Camera Models” (p. 153). Cambridge University Press.

[3] OpenCV Documentation (5.0). Image processing (imgproc): Feature detection, cv::HoughLinesP and cv::Canny. docs.opencv.org/5.0

[4] Zhang, Z. (2000). A flexible new technique for camera calibration. IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(11), 1330–1334. doi:10.1109/34.888718