What Is PnP? Finding Where the Camera Is from Known Points

PnP finds where a camera stood and which way it was facing, from a few points whose positions you know and where they appear in one photograph. Below, for each photo of a chessboard, the board’s axes drawn from the pose PnP recovers, and every camera position seen from above.

You have met it wherever a camera finds itself against something it recognises:

• AR markers such as ArUco and AprilTag: four corners of known size give the camera’s pose every frame.
• Drone landing pads and robot docking stations carry printed patterns for the same reason.
• Robot arms with a camera locate the part they pick from its known corners.
• Structure from motion adds each new photo to a 3-D model by PnP against the points already built.

Given a calibrated camera and a few points of known position, PnP returns where the camera stood and which way it faced; a robust version survives points that were found wrong. On OpenCV’s chessboard photos it matches the calibration’s own pose to 0.00002°, and with five of 54 corners wrong it moves the camera 0.008 squares where a plain fit moves it 2.5.

Where we are

Lesson 4, the last, of camera geometry, and step 6 of its pipeline with one photo: $K$ and the lens are known from lesson 3, and only $[R \mid t]$ is left. Lesson 2 pushed points through a given pose; this lesson runs it the other way [1].

Six unknowns, two equations per point

A pose is six numbers: three for the rotation and three for the position. Every point of known position seen in the photo gives two equations, one per pixel coordinate, so three points are the minimum. Three points give up to four candidate poses; a fourth point picks the right one, and for a flat target four is the usual minimum. With more points, OpenCV’s default solvePnP starts from a closed-form guess and adjusts $R$ and $t$ until the reprojection error, the same score as calibration’s, is as small as it gets [1] [3].

The rotation comes back as a three-number rotation vector; cv2.Rodrigues turns it into the 3×3 $R$. The camera’s position on the board is not $t$ itself: $t$ is the board’s origin seen from the camera, and the camera seen from the board is $C = -R^\top t$.

Now you try

A square tag with known corners and their observed pixels. Drag the pose by hand to bring the predicted corners onto the observed ones and watch the error fall, then press solve and let the iterative fit do it.

How close is this to OpenCV?

The cells run solvePnP on the first photo’s corners with the calibrated camera, then break five corners on purpose and compare the plain fit with solvePnPRansac, which fits on small random subsets and keeps the pose most corners agree with [2].

import json
import numpy as np
import cv2

d = json.load(open("corners.json"))
obj = np.zeros((54, 3), np.float32)
obj[:, :2] = np.mgrid[0:9, 0:6].T.reshape(-1, 2)
seen = np.float32(d["corners"][0]).reshape(-1, 1, 2)

# the camera, from calibration
K = np.array([[536.0735, 0, 342.3705],
              [0, 536.0164, 235.5369], [0, 0, 1]])
dist = np.array([-0.26509, -0.04674, 0.00183,
                 -0.00031, 0.25231])

ok, rvec, tvec = cv2.solvePnP(obj, seen, K, dist)
R, _ = cv2.Rodrigues(rvec)
C = (-R.T @ tvec).ravel()

def rms(rv, tv, o=obj, s=seen):
    p, _ = cv2.projectPoints(o, rv, tv, K, dist)
    return float(np.sqrt(np.mean(np.sum(
        (p - s).reshape(-1, 2) ** 2, axis=1))))

print("camera centre, in squares:", C.round(3))
print("distance:", round(float(np.linalg.norm(C)), 2))
print("misses the corners by", round(rms(rvec, tvec), 3),
      "px RMS")

camera centre, in squares: [  7.371   1.647 -15.059]
distance: 16.85
misses the corners by 0.193 px RMS

# five corners moved 25 to 50 px, as a bad detection would
bad = seen.copy().reshape(-1, 2)
idx = [5, 17, 23, 38, 50]
bad[idx] += np.float32([[40, -25], [-35, 30], [30, 35],
                        [-40, -20], [25, -40]])
bad = bad.reshape(-1, 1, 2)
good = np.setdiff1d(np.arange(54), idx)

def shift(rv, tv):
    Rm, _ = cv2.Rodrigues(rv)
    return float(np.linalg.norm((-Rm.T @ tv).ravel() - C))

ok, rv1, tv1 = cv2.solvePnP(obj, bad, K, dist)
print(f"solvePnP:       moves {shift(rv1, tv1):.3f}, "
      f"{rms(rv1, tv1, obj[good], seen[good]):.3f} px")
ok, rv2, tv2, inl = cv2.solvePnPRansac(
    obj, bad, K, dist, reprojectionError=3.0)
print(f"solvePnPRansac: moves {shift(rv2, tv2):.4f}, "
      f"{rms(rv2, tv2, obj[good], seen[good]):.3f} px, "
      f"{len(inl)} inliers")

solvePnP:       moves 2.517, 2.987 px
solvePnPRansac: moves 0.0079, 0.198 px, 49 inliers

The camera comes back at $(7.371, 1.647, -15.059)$ squares and the corners are missed by 0.19 px, the calibration’s own pose: over all thirteen photos the two never differ by more than 0.00002° or 0.00001 squares. With five corners broken, the plain fit pulls the camera 2.517 squares off and misses even the good corners by 3 px; RANSAC finds the 49 good ones and stays within 0.008 squares.

On the other twelve photos

The thirteen poses put the camera between 11.9 and 16.85 squares from the board’s first corner, all on the same side of it and spread across its width, which is what the plot above shows: the person moved the board, not the camera, and PnP reads that as the camera moving around a still board.

Where this breaks

Noise grows with distance. One pixel of random error on every corner, 200 times, moves the recovered camera by a median of 0.145 squares at 16.85 squares away, under 1%. The farther the target or the smaller it is in the photo, the larger that share, because the same pixel covers more of the world.

Few points, or points in a line. The four outer corners alone give a pose that reproduces all 54 to 0.269 px, 0.073 squares from the full fit. Three points have up to four answers, and points that all lie on one line leave the rotation about that line undetermined.

A wrong $K$ is a wrong pose. PnP trusts the calibration. A focal length that is off moves the camera along its line of sight to compensate, and the reprojection error barely shows it.

Exercises

1. Where the camera stood. Write centre(rvec, tvec), the camera’s position in the board’s frame, from the rotation vector and shift solvePnP returns.

def centre(rvec, tvec):
    return np.zeros(3)  # camera centre, board units

print(centre(rvec, tvec))

2. Predict, then run. Solve from the four outer corners only. How far does the camera move, and how well does that pose fit all 54 corners?
3. Noise. Add 1 px of random noise to every corner, 50 times. What is the median move?

Next

This closes the unit. Camera geometry puts the four lessons and their numbers side by side. Structure from motion runs PnP on photo after photo to build a 3-D model, and adjusts every pose and point together at the end.

References

[1] Szeliski, R. (2022). Computer Vision: Algorithms and Applications (2nd ed.), §11.2 “Pose estimation” (p. 693), including §11.2.1 “Linear algorithms” and §11.2.2 “Iterative non-linear algorithms”. Springer. Free PDF

[2] Fischler, M. A., & Bolles, R. C. (1981). Random Sample Consensus: A Paradigm for Model Fitting with Applications to Image Analysis and Automated Cartography. Communications of the ACM, 24(6), 381–395. doi:10.1145/358669.358692

[3] OpenCV Documentation (5.0). Camera Calibration and 3D Reconstruction (calib3d): solvePnP, solvePnPRansac, Rodrigues. docs.opencv.org/5.0