Camera Models, End to End: Pinhole, Lens Distortion, Calibration & PnP — with OpenCV

A camera is a function that turns a 3-D point into a pixel. Master that function and its inverse and you can measure the world from images: where things are, how big, how far, and where the camera itself sits. This post walks the whole chain on one running scene — pinhole projection and the intrinsic matrix $K$, the extrinsics $[R\,|\,t]$ that place the camera, the lens distortion that bends straight lines (a corner in our scene moves 62 px), the calibration that recovers all of it to 0.35 px reprojection error, and solvePnP, which takes a pose that is 68 px wrong and drives it to essentially zero. Math, three things you can drag, and runnable OpenCV in Python and C++.

This is the geometry sequel to From Photons to Pixels: Image Formation and Color Spaces, which ended where the pinhole equation begins. Here we take that one matrix and build the rest of the camera around it.

The running example: the patio cam

One scene for the whole post. A fixed camera watches Luis’s patio. Its sensor is 640 × 480, its focal length is $f = 500$ px, and its principal point sits dead-centre at $(c_x, c_y) = (320, 240)$. We use the OpenCV camera frame: $+X$ right, $+Y$ down, $+Z$ straight out into the scene, so the projection of a camera-frame point $(X_c, Y_c, Z_c)$ is

Two props live in the scene, and we’ll reuse their numbers everywhere:

Everything below is computed on these numbers, so you can check every step by hand.

1. The pinhole model and the intrinsic matrix K

Intuition first: a pinhole camera is just similar triangles. A point twice as far away projects half as large; the focal length $f$ is the constant of proportionality that turns “metres sideways per metre deep” into pixels [3].

The two scalar equations stack into one matrix. Writing pixels in homogeneous coordinates, the intrinsic matrix $K$ packs the focal length and principal point [3]:

The "$\sim$" means up to scale: you divide by the third coordinate ($Z_c$) at the end, and that single division is the entire reason distant things look small. $f_x$ and $f_y$ differ only if your pixels aren’t square; for the patio cam $f_x = f_y = 500$.

A useful companion number is the field of view, which $f$ and the sensor width fix together:

Short focal length → wide FOV → small objects; long focal length → narrow FOV → “zoomed in.” Same sensor, same equation.

In OpenCV you never hand-roll the division — projectPoints applies $K$ (and, as we’ll see, distortion) for you:

import cv2
import numpy as np

K = np.array([[500, 0, 320],
              [0, 500, 240],
              [0,   0,   1]], dtype=np.float64)
dist = np.zeros(5)                       # no distortion yet

obj = np.array([[0.6, -0.5, 5.0]])       # cactus apex, camera frame
rvec = np.zeros(3); tvec = np.zeros(3)   # camera == origin → identity pose
pix, _ = cv2.projectPoints(obj, rvec, tvec, K, dist)
print(pix.ravel())                       # [380. 190.]

#include <opencv2/opencv.hpp>
#include <iostream>

int main() {
    cv::Matx33d K(500, 0, 320,  0, 500, 240,  0, 0, 1);
    cv::Mat dist = cv::Mat::zeros(5, 1, CV_64F);   // no distortion yet

    std::vector<cv::Point3d> obj{{0.6, -0.5, 5.0}}; // cactus apex, camera frame
    cv::Vec3d rvec(0, 0, 0), tvec(0, 0, 0);         // identity pose
    std::vector<cv::Point2d> pix;
    cv::projectPoints(obj, rvec, tvec, K, dist, pix);
    std::cout << pix[0] << "\n";                     // [380, 190]
}

Now you try

Drag the cactus around in real 3-D and watch its pixel and on-screen radius track $f\,X/Z$ and $f\,R/Z$. Push the focal-length slider and feel the field of view narrow. (This lab renders with $+Y$ up, so lifting the ball makes $v$ decrease — the same geometry as our $+Y$-down table, with the vertical axis flipped.)

2. Extrinsics: placing the camera in the world

So far the cactus was already in camera coordinates. In real life your points live in a world frame — a corner of the patio, say — and the camera is rotated and translated relative to it. The extrinsics are that rigid transform: a rotation $R$ (3×3) and a translation $t$ (3×1) that carry a world point into the camera frame.

That product $P = K\,[R \mid t]$ is the full 3×4 projection matrix — the entire camera in one object. Intrinsics are who the camera is (fixed by the lens and sensor); extrinsics are where it stands (changes every time it moves).

Rotation is the part worth respecting. A general $R$ has three angles (yaw, pitch, roll); OpenCV passes it compactly as a 3-vector rvec (axis × angle, the Rodrigues form) and converts with cv2.Rodrigues. We won’t grind a 3×3 rotation by hand here — instead, the PnP lab in §5 lets you spin the camera’s yaw and watch every pixel respond.

3. Lens distortion: why straight lines bend

The pinhole model assumes light travels in perfect straight lines through a single point. Real lenses don’t: they bend rays more toward the edges, so straight lines in the world become curved in the image. Intuition: a wide cheap lens “wraps” the scene, pulling the corners inward (barrel) or pushing them out (pincushion).

OpenCV models this with the Brown–Conrady coefficients [4]: radial terms $k_1, k_2, k_3$ (the dominant, symmetric bowing) and tangential terms $p_1, p_2$ (from a lens not perfectly parallel to the sensor). On normalized coordinates $x = (u - c_x)/f$, $y = (v - c_y)/f$ with $r^2 = x^2 + y^2$:

The radial factor depends only on $r$, so it does nothing at the centre ($r = 0$) and grows toward the corners — which is why distortion is an edge phenomenon.

When distortion is extreme (action cams, 180°+ lenses), the polynomial model breaks down and you switch to a dedicated fisheye model (cv2.fisheye), which parameterizes distortion by the angle $\theta$ from the optical axis rather than by $r$ [5]. Same idea, a function fit for very wide angles.

Now you try

Slide $k_1, k_2, p_1, p_2$ and watch the grid bow. Hit Barrel for the worked numbers above, then Show undistorted to see calibration straighten it back — that green grid is what every downstream algorithm actually wants.

Correcting it in OpenCV, once you have the coefficients, is one call:

import cv2

# K, dist come from calibration (next section). Straighten the whole image:
undistorted = cv2.undistort(img, K, dist)

# For video, precompute the remap once and reuse it every frame (much faster):
h, w = img.shape[:2]
newK, _ = cv2.getOptimalNewCameraMatrix(K, dist, (w, h), alpha=0)
map1, map2 = cv2.initUndistortRectifyMap(K, dist, None, newK, (w, h), cv2.CV_16SC2)
undistorted = cv2.remap(img, map1, map2, cv2.INTER_LINEAR)

#include <opencv2/opencv.hpp>

// K, dist come from calibration (next section). Straighten the whole image:
cv::Mat undistorted;
cv::undistort(img, undistorted, K, dist);

// For video, precompute the remap once and reuse it every frame:
cv::Mat map1, map2, newK;
newK = cv::getOptimalNewCameraMatrix(K, dist, img.size(), 0);
cv::initUndistortRectifyMap(K, dist, cv::noArray(), newK, img.size(),
                            CV_16SC2, map1, map2);
cv::remap(img, undistorted, map1, map2, cv::INTER_LINEAR);

4. Calibration: recovering K and the distortion

Calibration is the inverse of §1–§3: instead of using $K$ and the coefficients to make pixels, you show the camera an object of known geometry, observe where its features land, and solve for the parameters that explain those observations. The standard recipe is Zhang’s method [6]: print a checkerboard (known square size), photograph it in ~15–20 poses, detect the inner corners, and let least squares recover $K$, the distortion coefficients, and a pose $[R\,|\,t]$ per image.

How do you know the calibration is good? The reprojection error: take the recovered parameters, project the known 3-D corners back to pixels, and measure how far they land from the detected ones. OpenCV reports the RMS over all corners.

The whole pipeline is a handful of calls:

import cv2
import numpy as np

PATTERN = (9, 6)          # inner corners
SQUARE = 0.025            # 25 mm squares → results in metres

# One template of the board's 3-D corners (Z=0 plane), reused for every view.
objp = np.zeros((PATTERN[0] * PATTERN[1], 3), np.float32)
objp[:, :2] = np.mgrid[0:PATTERN[0], 0:PATTERN[1]].T.reshape(-1, 2) * SQUARE

obj_points, img_points = [], []
for img in images:                       # ~15–20 photos of the board
    gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
    found, corners = cv2.findChessboardCorners(gray, PATTERN)
    if found:
        corners = cv2.cornerSubPix(
            gray, corners, (11, 11), (-1, -1),
            (cv2.TERM_CRITERIA_EPS + cv2.TERM_CRITERIA_MAX_ITER, 30, 1e-3))
        obj_points.append(objp)
        img_points.append(corners)

rms, K, dist, rvecs, tvecs = cv2.calibrateCamera(
    obj_points, img_points, gray.shape[::-1], None, None)
print(f"RMS reprojection error: {rms:.3f} px")   # want < ~0.5
print(K)                                          # the recovered intrinsics

#include <opencv2/opencv.hpp>
#include <iostream>

cv::Size pattern(9, 6);          // inner corners
const float square = 0.025f;     // 25 mm squares → metres

// One template of the board's 3-D corners (Z=0 plane), reused per view.
std::vector<cv::Point3f> objp;
for (int y = 0; y < pattern.height; ++y)
    for (int x = 0; x < pattern.width; ++x)
        objp.emplace_back(x * square, y * square, 0.f);

std::vector<std::vector<cv::Point3f>> objPoints;
std::vector<std::vector<cv::Point2f>> imgPoints;
cv::Size imgSize;
for (const cv::Mat& img : images) {           // ~15–20 photos
    cv::Mat gray; cv::cvtColor(img, gray, cv::COLOR_BGR2GRAY);
    imgSize = gray.size();
    std::vector<cv::Point2f> corners;
    if (cv::findChessboardCorners(gray, pattern, corners)) {
        cv::cornerSubPix(gray, corners, {11, 11}, {-1, -1},
            {cv::TermCriteria::EPS + cv::TermCriteria::MAX_ITER, 30, 1e-3});
        objPoints.push_back(objp);
        imgPoints.push_back(corners);
    }
}

cv::Mat K, dist;
std::vector<cv::Mat> rvecs, tvecs;
double rms = cv::calibrateCamera(objPoints, imgPoints, imgSize, K, dist, rvecs, tvecs);
std::cout << "RMS reprojection error: " << rms << " px\n";  // want < ~0.5

5. PnP: where is the camera, given a known object?

Calibration recovered the intrinsics. PnP (Perspective-n-Point) recovers the extrinsics: given a calibrated $K$, a set of known 3-D points, and where they appear in one image, find the pose $[R\,|\,t]$. It’s the engine behind augmented reality, marker tracking, and robot/drone localization — anywhere you know an object and want to know where the camera sits relative to it.

Intuition: there is exactly one pose that makes all the known corners project onto the pixels you actually observed. PnP searches pose space for it by minimizing reprojection error — the same scorecard as calibration [3][7]:

where $M_i$ are the known 3-D points, $m_i$ the observed pixels, and $\pi$ the divide-by-$Z$ projection.

Now you try

You are the optimizer. Drag yaw, $t_x$, $t_z$ to slide the red reprojected tag onto the green observed one and shrink the error. Then press Solve (LM) and watch Levenberg–Marquardt do it for you — that’s solvePnP in ~30 lines of arithmetic.

In practice it’s one call — and with outlier correspondences, the RANSAC variant [8] is the robust default:

import cv2
import numpy as np

# K, dist from calibration. obj_pts: known 3-D points (N,3).
# img_pts: where they appear in THIS image (N,2).
ok, rvec, tvec = cv2.solvePnP(obj_pts, img_pts, K, dist,
                              flags=cv2.SOLVEPNP_ITERATIVE)

# With noisy/outlier matches, prefer the robust RANSAC version:
ok, rvec, tvec, inliers = cv2.solvePnPRansac(obj_pts, img_pts, K, dist)

R, _ = cv2.Rodrigues(rvec)        # 3×3 rotation; tvec is the translation
# Camera centre in world coords: C = -Rᵀ t
C = (-R.T @ tvec).ravel()
print("camera position:", C)

#include <opencv2/opencv.hpp>

// K, dist from calibration. objPts: known 3-D points. imgPts: their pixels.
cv::Mat rvec, tvec;
cv::solvePnP(objPts, imgPts, K, dist, rvec, tvec, false, cv::SOLVEPNP_ITERATIVE);

// With noisy/outlier matches, prefer the robust RANSAC version:
cv::Mat inliers;
cv::solvePnPRansac(objPts, imgPts, K, dist, rvec, tvec);

cv::Mat R; cv::Rodrigues(rvec, R);       // 3×3 rotation
cv::Mat C = -R.t() * tvec;               // camera centre in world coords

A small but common gotcha: solvePnP needs an undistorted measurement model. Either pass the real dist (as above) so OpenCV accounts for it, or feed already undistorted points with dist = 0 — but never mix the two.

A note on versions

This is an explainer; the code is illustrative rather than benchmarked. It targets OpenCV 4.x (validated against the 4.10 calib3d API). The functions shown — projectPoints, calibrateCamera, solvePnP/solvePnPRansac, undistort, fisheye — are stable across the 4.x line. There are no measured timings to reproduce here.

Limitations & caveats

• A single image has no absolute scale. PnP needs known 3-D geometry; from pixels alone, a small near object and a big far one are identical (the scale–depth ambiguity) — exactly the failure we made draggable in image processing vs vision vs graphics. Monocular pose is metric only because the object’s size is given.
• Calibration generalizes only as far as your captures. Too few views, all at similar angles, or a board that’s nearly coplanar in every shot, and $K$ is poorly constrained — the low RMS lies. Cover the frame, including the corners where distortion lives, and vary tilt [6].
• The Brown–Conrady model is a polynomial, not physics. It fits normal lenses well but breaks for very wide/fisheye optics — use cv2.fisheye past ~120° [5]. More $k$ terms can overfit: $k_3$ often hurts unless you truly need it.
• A low reprojection error is necessary, not sufficient. It can be small while the focal length and distortion trade off against each other. Sanity-check $K$ against the physical sensor/lens, and test on held-out views.
• Zoom, focus, and temperature change the intrinsics. Calibration is valid for the configuration you calibrated. Autofocus or a zoom lens means $K$ drifts — recalibrate or lock the lens.

Takeaways

• A camera is one matrix product, $P = K\,[R \mid t]$: extrinsics place the camera, intrinsics turn camera-frame metres into pixels, and a divide-by-$Z$ makes far things small.
• $K$ is who the camera is; $[R \mid t]$ is where it stands. Calibration recovers the first, PnP the second.
• Distortion is an edge effect that the centre never reveals — a corner in our scene moved 62 px. Always undistort before you measure.
• Reprojection error is the universal scorecard. Sub-pixel (~0.35 px) means a trustworthy calibration; PnP turns a 68-px-wrong pose into ~0 by minimizing it.
• Calibrate before you PnP. Pose estimation inherits every error in $K$.

Further reading

• Go deeper: Hartley & Zisserman, Multiple View Geometry (2nd ed.), ch. 6 (camera models) and ch. 7 (recovering $K$) — the definitive treatment of everything above.
• Practical depth: Szeliski, Computer Vision (2nd ed.), §2.1 (projection) and §11 (calibration & pose) — free PDF, very readable.
• The original method: Zhang’s calibration paper [6] — short, and the basis of calibrateCamera.
• Related on CondadosAI: prerequisite — image formation & color spaces; the framing — image processing vs vision vs graphics.

References

[1] OpenCV. Camera calibration and 3D reconstruction (calib3d). Version 4.10. Docs.

[2] Szeliski, R. (2022). Computer Vision: Algorithms and Applications (2nd ed.), §2.1 (geometric image formation) and §11 (structure from motion, calibration & pose). Springer. Free PDF.

[3] Hartley, R., & Zisserman, A. (2004). Multiple View Geometry in Computer Vision (2nd ed.), ch. 6 (camera models) & ch. 7 (camera matrix estimation). Cambridge University Press.

[4] Brown, D. C. (1971). Close-range camera calibration. Photogrammetric Engineering, 37(8), 855–866. (The radial + tangential distortion model.)

[5] Kannala, J., & Brandt, S. S. (2006). A generic camera model and calibration method for conventional, wide-angle, and fish-eye lenses. IEEE TPAMI, 28(8), 1335–1340. doi:10.1109/TPAMI.2006.153.

[6] Zhang, Z. (2000). A flexible new technique for camera calibration. IEEE TPAMI, 22(11), 1330–1334. doi:10.1109/34.888718.

[7] Lepetit, V., Moreno-Noguer, F., & Fua, P. (2009). EPnP: An accurate O(n) solution to the PnP problem. International Journal of Computer Vision, 81(2), 155–166. doi:10.1007/s11263-008-0152-6.

[8] Fischler, M. A., & Bolles, R. C. (1981). Random sample consensus (RANSAC). Communications of the ACM, 24(6), 381–395. doi:10.1145/358669.358692.