Camera Models, End to End: K, the Extrinsics, Calibration & PnP, with OpenCV

A camera is a function that turns a 3-D point into a pixel. Master that function and its inverse and you can measure the world from images: where things are, how big, how far, and where the camera itself sits. This post walks the whole chain on one running scene: pinhole projection and the intrinsic matrix $K$, the extrinsics $[R\,|\,t]$ that place the camera, the calibration that recovers both to 0.35 px reprojection error, and solvePnP, which takes a pose that is 68 px wrong and drives it to essentially zero. Math, two things you can drag, and runnable OpenCV in Python and C++.

This is the geometry sequel to how a camera forms an image, which covers the optics: the pinhole equations, what a lens adds, and the distortion coefficients that appear in the calibration below. Here we take those two scalar equations and build the rest of the camera around them.

The running example: the patio cam

One scene for the whole post. A fixed camera watches Luis’s patio. Its sensor is 640 × 480, its focal length is $f = 500$ px, and its principal point sits dead-centre at $(c_x, c_y) = (320, 240)$. We use the OpenCV camera frame: $+X$ right, $+Y$ down, $+Z$ straight out into the scene, so the projection of a camera-frame point $(X_c, Y_c, Z_c)$ is

Two props live in the scene, and we’ll reuse their numbers everywhere:

Everything below is computed on these numbers, so you can check every step by hand.

1. From the two scalar equations to the intrinsic matrix K

A pinhole camera is similar triangles, and the two scalar equations that come out of them are worked through in the pinhole lesson:

Those two equations stack into one matrix, and that is where this post starts. Writing pixels in homogeneous coordinates, the intrinsic matrix $K$ packs the focal length and the principal point [3]:

The "$\sim$" means up to scale: you divide by the third coordinate ($Z_c$) at the end, and that single division is the entire reason distant things look small. $f_x$ and $f_y$ differ only if your pixels aren’t square; for the patio cam $f_x = f_y = 500$.

A useful companion number is the field of view, which $f$ and the sensor width fix together:

Short focal length → wide FOV → small objects; long focal length → narrow FOV → “zoomed in.” Same sensor, same equation.

In OpenCV you never hand-roll the division. projectPoints applies $K$ (and, as we’ll see, distortion) for you:

import cv2
import numpy as np

K = np.array([[500, 0, 320],
              [0, 500, 240],
              [0,   0,   1]], dtype=np.float64)
dist = np.zeros(5)                       # no distortion yet

obj = np.array([[0.6, -0.5, 5.0]])       # cactus apex, camera frame
rvec = np.zeros(3); tvec = np.zeros(3)   # camera == origin → identity pose
pix, _ = cv2.projectPoints(obj, rvec, tvec, K, dist)
print(pix.ravel())                       # [380. 190.]

#include <opencv2/opencv.hpp>
#include <iostream>

int main() {
    cv::Matx33d K(500, 0, 320,  0, 500, 240,  0, 0, 1);
    cv::Mat dist = cv::Mat::zeros(5, 1, CV_64F);   // no distortion yet

    std::vector<cv::Point3d> obj{{0.6, -0.5, 5.0}}; // cactus apex, camera frame
    cv::Vec3d rvec(0, 0, 0), tvec(0, 0, 0);         // identity pose
    std::vector<cv::Point2d> pix;
    cv::projectPoints(obj, rvec, tvec, K, dist, pix);
    std::cout << pix[0] << "\n";                     // [380, 190]
}

Now you try

Drag the cactus around in real 3-D and watch its pixel and on-screen radius track $f\,X/Z$ and $f\,R/Z$. Push the focal-length slider and feel the field of view narrow. (This lab renders with $+Y$ up, so lifting the ball makes $v$ decrease, the same geometry as our $+Y$-down table, with the vertical axis flipped.)

2. Extrinsics: placing the camera in the world

So far the cactus was already in camera coordinates. In real life your points live in a world frame, a corner of the patio, say, and the camera is rotated and translated relative to it. The extrinsics are that rigid transform: a rotation $R$ (3×3) and a translation $t$ (3×1) that carry a world point into the camera frame.

That product $P = K\,[R \mid t]$ is the full 3×4 projection matrix, the entire camera in one object [2]. Intrinsics are who the camera is (fixed by the lens and sensor); extrinsics are where it stands (changes every time it moves).

Rotation is the part worth respecting. A general $R$ has three angles (yaw, pitch, roll); OpenCV passes it compactly as a 3-vector rvec (axis × angle, the Rodrigues form) and converts with cv2.Rodrigues. We won’t grind a 3×3 rotation by hand here. Instead, the PnP lab in §4 lets you spin the camera’s yaw and watch every pixel respond.

3. Calibration: recovering K and the distortion

Calibration is the inverse of §1–§2: instead of using $K$ and the coefficients to make pixels, you show the camera an object of known geometry, observe where its features land, and solve for the parameters that explain those observations. The standard recipe is Zhang’s method [4]: print a checkerboard (known square size), photograph it in ~15–20 poses, detect the inner corners, and let least squares recover $K$, the distortion coefficients, and a pose $[R\,|\,t]$ per image.

How do you know the calibration is good? The reprojection error: take the recovered parameters, project the known 3-D corners back to pixels, and measure how far they land from the detected ones. OpenCV reports the RMS over all corners.

The whole pipeline is a handful of calls:

import cv2
import numpy as np

PATTERN = (9, 6)          # inner corners
SQUARE = 0.025            # 25 mm squares → results in metres

# One template of the board's 3-D corners (Z=0 plane), reused for every view.
objp = np.zeros((PATTERN[0] * PATTERN[1], 3), np.float32)
objp[:, :2] = np.mgrid[0:PATTERN[0], 0:PATTERN[1]].T.reshape(-1, 2) * SQUARE

obj_points, img_points = [], []
for img in images:                       # ~15–20 photos of the board
    gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
    found, corners = cv2.findChessboardCorners(gray, PATTERN)
    if found:
        corners = cv2.cornerSubPix(
            gray, corners, (11, 11), (-1, -1),
            (cv2.TERM_CRITERIA_EPS + cv2.TERM_CRITERIA_MAX_ITER, 30, 1e-3))
        obj_points.append(objp)
        img_points.append(corners)

rms, K, dist, rvecs, tvecs = cv2.calibrateCamera(
    obj_points, img_points, gray.shape[::-1], None, None)
print(f"RMS reprojection error: {rms:.3f} px")   # want < ~0.5
print(K)                                          # the recovered intrinsics

#include <opencv2/opencv.hpp>
#include <iostream>

cv::Size pattern(9, 6);          // inner corners
const float square = 0.025f;     // 25 mm squares → metres

// One template of the board's 3-D corners (Z=0 plane), reused per view.
std::vector<cv::Point3f> objp;
for (int y = 0; y < pattern.height; ++y)
    for (int x = 0; x < pattern.width; ++x)
        objp.emplace_back(x * square, y * square, 0.f);

std::vector<std::vector<cv::Point3f>> objPoints;
std::vector<std::vector<cv::Point2f>> imgPoints;
cv::Size imgSize;
for (const cv::Mat& img : images) {           // ~15–20 photos
    cv::Mat gray; cv::cvtColor(img, gray, cv::COLOR_BGR2GRAY);
    imgSize = gray.size();
    std::vector<cv::Point2f> corners;
    if (cv::findChessboardCorners(gray, pattern, corners)) {
        cv::cornerSubPix(gray, corners, {11, 11}, {-1, -1},
            {cv::TermCriteria::EPS + cv::TermCriteria::MAX_ITER, 30, 1e-3});
        objPoints.push_back(objp);
        imgPoints.push_back(corners);
    }
}

cv::Mat K, dist;
std::vector<cv::Mat> rvecs, tvecs;
double rms = cv::calibrateCamera(objPoints, imgPoints, imgSize, K, dist, rvecs, tvecs);
std::cout << "RMS reprojection error: " << rms << " px\n";  // want < ~0.5

4. PnP: where is the camera, given a known object?

Calibration recovered the intrinsics. PnP (Perspective-n-Point) recovers the extrinsics: given a calibrated $K$, a set of known 3-D points, and where they appear in one image, find the pose $[R\,|\,t]$. It’s the engine behind augmented reality, marker tracking, and robot/drone localization, anywhere you know an object and want to know where the camera sits relative to it.

Intuition: there is exactly one pose that makes all the known corners project onto the pixels you actually observed. PnP searches pose space for it by minimizing reprojection error, the same scorecard as calibration [3][5]:

where $M_i$ are the known 3-D points, $m_i$ the observed pixels, and $\pi$ the divide-by-$Z$ projection.

Now you try

You are the optimizer. Drag yaw, $t_x$, $t_z$ to slide the red reprojected tag onto the green observed one and shrink the error. Then press Solve (LM) and watch Levenberg–Marquardt do it for you. That’s solvePnP in ~30 lines of arithmetic.

In practice it’s one call, and with outlier correspondences, the RANSAC variant [6] is the robust default:

import cv2
import numpy as np

# K, dist from calibration. obj_pts: known 3-D points (N,3).
# img_pts: where they appear in THIS image (N,2).
ok, rvec, tvec = cv2.solvePnP(obj_pts, img_pts, K, dist,
                              flags=cv2.SOLVEPNP_ITERATIVE)

# With noisy/outlier matches, prefer the robust RANSAC version:
ok, rvec, tvec, inliers = cv2.solvePnPRansac(obj_pts, img_pts, K, dist)

R, _ = cv2.Rodrigues(rvec)        # 3×3 rotation; tvec is the translation
# Camera centre in world coords: C = -Rᵀ t
C = (-R.T @ tvec).ravel()
print("camera position:", C)

#include <opencv2/opencv.hpp>

// K, dist from calibration. objPts: known 3-D points. imgPts: their pixels.
cv::Mat rvec, tvec;
cv::solvePnP(objPts, imgPts, K, dist, rvec, tvec, false, cv::SOLVEPNP_ITERATIVE);

// With noisy/outlier matches, prefer the robust RANSAC version:
cv::Mat inliers;
cv::solvePnPRansac(objPts, imgPts, K, dist, rvec, tvec);

cv::Mat R; cv::Rodrigues(rvec, R);       // 3×3 rotation
cv::Mat C = -R.t() * tvec;               // camera centre in world coords

A small but common gotcha: solvePnP needs an undistorted measurement model. Either pass the real dist (as above) so OpenCV accounts for it, or feed already undistorted points with dist = 0, but never mix the two.

A note on versions

This is an explainer; the code is illustrative rather than benchmarked. It targets OpenCV 4.x (validated against the 4.10 calib3d API) [1]. The functions shown (projectPoints, calibrateCamera, solvePnP/solvePnPRansac, undistort, fisheye) are stable across the 4.x line. There are no measured timings to reproduce here.

Limitations & caveats

• A single image has no absolute scale. PnP needs known 3-D geometry; from pixels alone, a small near object and a big far one are identical (the scale–depth ambiguity), exactly the failure we made draggable in image processing vs vision vs graphics. Monocular pose is metric only because the object’s size is given.
• Calibration generalizes only as far as your captures. Too few views, all at similar angles, or a board that’s nearly coplanar in every shot, and $K$ is poorly constrained, and the low RMS lies. Cover the frame, including the corners where distortion lives, and vary tilt [4].
• The distortion coefficients calibration returns are a fit, not physics. What they mean, how far they move a pixel and where the model gives out are covered in lens distortion and fisheye models. The short version for this post: more $k$ terms can overfit, and $k_3$ often hurts unless you truly need it.
• A low reprojection error is necessary, not sufficient. It can be small while the focal length and distortion trade off against each other. Sanity-check $K$ against the physical sensor/lens, and test on held-out views.
• Zoom, focus, and temperature change the intrinsics. Calibration is valid for the configuration you calibrated. Autofocus or a zoom lens means $K$ drifts, so recalibrate or lock the lens.

Takeaways

• A camera is one matrix product, $P = K\,[R \mid t]$: extrinsics place the camera, intrinsics turn camera-frame metres into pixels, and a divide-by-$Z$ makes far things small.
• $K$ is who the camera is; $[R \mid t]$ is where it stands. Calibration recovers the first, PnP the second.
• Undistort before you measure. solvePnP and every geometric estimate below assume a rectilinear image; feeding them raw pixels imports the lens’s error into the pose.
• Reprojection error is the universal scorecard. Sub-pixel (~0.35 px) means a trustworthy calibration; PnP turns a 68-px-wrong pose into ~0 by minimizing it.
• Calibrate before you PnP. Pose estimation inherits every error in $K$.

Further reading

• Go deeper: Hartley & Zisserman, Multiple View Geometry (2nd ed.), ch. 6 “Camera Models” and ch. 7 “Computation of the Camera Matrix P”, for the definitive treatment of everything above.
• Practical depth: Szeliski, Computer Vision (2nd ed.), a free PDF and very readable; the projection and calibration material is in its early chapters.
• The original method: Zhang’s calibration paper [4], which is short and the basis of calibrateCamera.
• Related on CondadosAI: the prerequisite unit is how a camera forms an image, which covers the optics this post assumes, including lens distortion; the framing is in image processing vs vision vs graphics.

References

[1] OpenCV. Camera calibration and 3D reconstruction (calib3d). Version 4.10. Docs.

[2] Szeliski, R. (2022). Computer Vision: Algorithms and Applications (2nd ed.). Springer. Free PDF.

[3] Hartley, R., & Zisserman, A. (2004). Multiple View Geometry in Computer Vision (2nd ed.), ch. 6 “Camera Models” (p. 153) and ch. 7 “Computation of the Camera Matrix P” (p. 178). Cambridge University Press.

[4] Zhang, Z. (2000). A flexible new technique for camera calibration. IEEE TPAMI, 22(11), 1330–1334. doi:10.1109/34.888718.

[5] Lepetit, V., Moreno-Noguer, F., & Fua, P. (2009). EPnP: An accurate O(n) solution to the PnP problem. International Journal of Computer Vision, 81(2), 155–166. doi:10.1007/s11263-008-0152-6.

[6] Fischler, M. A., & Bolles, R. C. (1981). Random sample consensus (RANSAC). Communications of the ACM, 24(6), 381–395. doi:10.1145/358669.358692.