Fisheye Models: When Correcting the Lens Stops Working

Lesson 4 of the Image Formation unit. The pinhole model puts a ray at f·tan θ, which runs to infinity at 90 degrees, and past about 120 degrees of field the stretch toward the edges makes it impractical. Fitted to the same 15 photographs of one fisheye lens, pinhole plus Brown-Conrady reaches 7.326 px RMS and Kannala-Brandt reaches 0.644 px. At 80 degrees the first model predicts a radius of -35,572 px, a sign flip; the second predicts 458.8 px, which is on the sensor.

Luis Condados · August 19, 2026 · Updated October 4, 2026 · 9 min read Intermediate

camera-calibration opencv optics fundamentals

Both camera models answering the same question. One of them leaves the plot before 90 degrees.

A fisheye squeezes half of everything around the camera into one circle, and a different projection model gets ordinary, straight-lined views back out of it. Below, a virtual camera moves across one fisheye photograph, and the right-hand side shows what it sees.

Left: a circular fisheye photo of a brick courtyard with an orange outline moving over it. Right: a normal-looking view of the part inside the outline, with straight walls and gate bars — Left: the photograph as taken, with the outline of the virtual camera’s view. Right: that view, computed through the lens’s equisolid model, with straight bars and walls where the fisheye bent them. “Fisheye lens Photography In Iran-Kashan City-Mostafa Meraji-2016-free Pictures-Canon EF 8-15mm lens 03.jpg” by Mostafameraji (Wikimedia Commons), CC0.

Where fisheye images get turned back into ordinary ones:

Security-dome viewers let you pan and zoom inside one fisheye image as if steering a normal camera.
Car surround view combines several fisheye cameras into one picture of the car from above.
Action cameras’ “linear” mode turns their very wide frame into one with straight lines.
Consumer 360° cameras record two fisheye images, back to back, and join them.

A wide enough lens cannot be described as a pinhole with corrections, because the pinhole’s $f\tan\theta$ runs to infinity at 90°; a fisheye needs a curve of its own. On 15 photographs of one fisheye lens, the corrected pinhole fits to 7.326 px and the Kannala–Brandt fisheye model to 0.644 px.

Where we are

Lesson 4 of image formation, the last in the unit, and step 3 of its pipeline for lenses too wide for lesson 3’s model. Lesson 3 corrected a real lens to 0.170 px with five coefficients, and ended on the ceiling built into that correction: the thing being corrected is $\tan\theta$ .

The unit’s second lens is where that ceiling is visible. It is a fisheye, and the measurements below come from 15 photographs of a board with 6×8 interior corners at 1032×778.

The tangent is the problem

The pinhole model from lesson 1 puts a flat image plane in front of the camera, and asks where a ray pierces it. A ray arriving at 89° is very nearly parallel to that plane, so it pierces it a very long way out. At exactly 90° it never does. That follows from the flat image plane itself, whatever the lens, and it puts a hard limit under 180° on any camera that uses one [2].

The practical limit arrives much earlier. The radius grows as $f\tan\theta$ , so each degree of the scene is stretched across $1/\cos^2\theta$ times as many pixels as a degree at the centre: 4 times at 60° off-axis, which is the edge of a 120° field, and 33 times at 80°. Past about 120° a rectilinear image spends most of its pixels on a thin ring of the scene, which is why the number is a rule of thumb rather than a mathematical boundary.

A fisheye lens is built to a different rule. Rather than preserving straight lines, it maps angle to radius directly, and the common designs differ only in how:

r = f\theta \quad\text{(equidistant)}, \qquad r = 2f\sin(\theta/2) \quad\text{(equisolid)},

r = 2f\tan(\theta/2) \quad\text{(stereographic)}, \qquad r = f\sin\theta \quad\text{(orthographic)}.

Every one of them is finite at 90° and beyond. Rather than pick one, Kannala and Brandt [1] fit an odd polynomial in $\theta$ , which contains all of them as special cases:

\theta_d = \theta\,(1 + k_1\theta^2 + k_2\theta^4 + k_3\theta^6 + k_4\theta^8), \qquad r = f\,\theta_d.

This is what cv2.fisheye implements, with four coefficients rather than the five of Brown–Conrady [3]. The count of terms is incidental; what matters is that the polynomial multiplies $\theta$ instead of $\tan\theta$ .

Four answers to one question, drawn at one shared focal length. The red curve sits above the others from the start, because tan θ is never smaller than θ, and leaves the sensor near 66°. The three fisheye designs differ in how they compress the outer field, and all of them still have somewhere to put a ray arriving from the side.

One ray, two models

Worked example · A ray arriving at 80° on the fisheye lens

Fitting the pinhole model to these 15 photographs gives $f = 372.22$ px with $k_1 = -0.1904280$ , $k_2 = 0.02420791$ , $k_3 = -0.001105085$ (seven significant figures, because at this angle $r^6$ is over 33,000 and a rounded $k_3$ moves the answer by hundreds of pixels). Take $\theta = 80°$ :

r = \tan 80° = 5.6713, \qquad r^2 = 32.163

k_1 r^2 = -6.1248, \qquad k_2 r^4 = +25.0428, \qquad k_3 r^6 = -36.7691

The correction factor is $1 - 6.1248 + 25.0428 - 36.7691 = -16.8511$ , so

r_{\text{pinhole}} = 372.22 \times 5.6713 \times (-16.8511) = -35{,}572\ \text{px}.

Fitting Kannala–Brandt to the same corners gives $f = 336.86$ px and coefficients $k_1 = -0.002644$ , $k_2 = -0.000295$ , $k_3 = -0.003123$ , $k_4 = 0.00034$ . The same ray, at $\theta = 1.39626$ rad:

\theta_d = 1.36205, \qquad r_{\text{fisheye}} = 336.86 \times 1.36205 = 458.8\ \text{px}.

What it means: the second answer is a position on a sensor whose half-diagonal is 646 px. The first is negative, which is not a large error but a meaningless one: the polynomial has overshot so far that it has folded the image through the origin. And notice how small the fitted fisheye coefficients are. They move $\theta_d$ by −0.24% at 45° and −2.45% at 80°, which says this lens is very nearly an equidistant design and the polynomial is only trimming it. That is a property of this lens; the circular fisheye in the next section is built to a different rule.

Across the range the two predictions stay within 11% of each other out to 15° and 17% out to 45°, both still climbing, and then one of them stops being a position at all:

θ	pinhole + Brown–Conrady	Kannala–Brandt
15°	98.4 px	88.2 px
45°	309.9 px	263.9 px
60°	397.6 px	350.3 px
70°	477.2 px	406.1 px
80°	−35,572 px	458.8 px
89°	−8.3 × 10¹¹ px	502.1 px

The consequence for the fit is the number this lesson exists for. On the same 15 views, with the same corners:

Model	RMS reprojection error	Worst view
Pinhole + Brown–Conrady	7.326 px	10.434 px
Kannala–Brandt	0.644 px	1.247 px

A factor of 11.4. Both models were given identical data and identical corner locations; the only difference is what they assume happens between the world and the sensor.

Now you try

Start at 60°, where the models still look like variations on a theme, then drag $\theta$ toward 89 and watch the perspective readout leave the sensor and then the plot entirely. The fitted Kannala–Brandt curve sits almost on top of the equidistant line, which is what a well-behaved fisheye looks like.

One difference from the table above, so the numbers do not look contradictory: the lab draws all five models at one shared focal length, because the question it answers is what the five shapes do. The table gives each model the focal length it was actually fitted with, 372.22 px for the pinhole and 336.86 px for the fisheye, so its perspective row is smaller than the lab’s at the same angle.

Try it: push the ray past 90°

Fitting one, in practice

import cv2, numpy as np

h, w = gray.shape
# Without a starting guess the solver converges to a focal length it cannot
# recover from. f = max(w, h) / pi is the equidistant estimate for a 180 degree lens.
k = np.array([[max(w, h) / np.pi, 0, w / 2], [0, max(w, h) / np.pi, h / 2], [0, 0, 1]])
flags = cv2.CALIB_USE_INTRINSIC_GUESS | cv2.CALIB_RECOMPUTE_EXTRINSIC | cv2.CALIB_FIX_SKEW
rms, k, d, rvecs, tvecs = cv2.fisheye.calibrate(
    obj_points, img_points, (w, h), k, np.zeros((4, 1)), None, None, flags
)

#include <opencv2/opencv.hpp>

// Without a starting guess the solver converges to a focal length it cannot
// recover from. f = max(w, h) / pi is the equidistant estimate for a 180 deg lens.
const double f0 = std::max(size.width, size.height) / CV_PI;
cv::Mat K = (cv::Mat_<double>(3, 3) << f0, 0, size.width / 2.0,
                                       0, f0, size.height / 2.0,
                                       0, 0, 1);
cv::Mat D = cv::Mat::zeros(4, 1, CV_64F);
const int flags = cv::CALIB_USE_INTRINSIC_GUESS |
                  cv::CALIB_RECOMPUTE_EXTRINSIC | cv::CALIB_FIX_SKEW;
const double rms = cv::fisheye::calibrate(objPoints, imgPoints, size, K, D,
                                          cv::noArray(), cv::noArray(), flags);

Two details in that snippet cost an afternoon each. The intrinsic guess is not optional: left to initialise $K$ itself, the solver settles 200 px from the corners on this lens, and seeding it with $f = \max(w, h)/\pi$ is the difference between a fit and a failure. And in OpenCV 5 the flags moved out of the cv2.fisheye namespace into cv2, so every tutorial written against 4.x raises AttributeError on that line.

How close is this to OpenCV?

The worked example’s two curves in NumPy, against OpenCV’s two projection functions on the same rays. The fisheye photographs are not served from this site (their licence is GPL-2.0), so the cells use the fitted coefficients.

Both models by hand

import numpy as np

# pinhole + Brown-Conrady, fitted to the 15 fisheye photos
f_pin = 372.2176
k1, k2, k3 = -0.1904280, 0.02420791, -0.001105085
# Kannala-Brandt, fitted to the same corners
f_fish = 336.8584
kb = [-0.0026441, -0.00029514, -0.0031234, 0.00034040]

def r_pinhole(deg):
    r = np.tan(np.radians(deg))
    r2 = r * r
    return f_pin * r * (1 + k1*r2 + k2*r2**2 + k3*r2**3)

def r_fisheye(deg):
    t = np.radians(deg)
    td = t * (1 + kb[0]*t**2 + kb[1]*t**4
              + kb[2]*t**6 + kb[3]*t**8)
    return f_fish * td

for deg in (15, 45, 60, 70, 80):
    print(f"{deg:2} deg  pinhole {r_pinhole(deg):10.1f}"
          f"   fisheye {r_fisheye(deg):6.1f} px")

Output

15 deg  pinhole       98.4   fisheye   88.2 px
45 deg  pinhole      309.9   fisheye  263.9 px
60 deg  pinhole      397.6   fisheye  350.3 px
70 deg  pinhole      477.2   fisheye  406.1 px
80 deg  pinhole   -35572.0   fisheye  458.8 px

Against cv2.projectPoints and cv2.fisheye.projectPoints

import cv2

zero = np.zeros(3)
for deg in (45, 80):
    t = np.radians(deg)
    ray = np.array([[[np.sin(t), 0, np.cos(t)]]])
    K = np.array([[f_fish, 0, 0], [0, f_fish, 0],
                  [0, 0, 1.0]])
    q, _ = cv2.fisheye.projectPoints(
        ray, zero, zero, K, np.array(kb))
    Kp = np.array([[f_pin, 0, 0], [0, f_pin, 0],
                   [0, 0, 1.0]])
    d = np.array([k1, k2, 0, 0, k3])
    p, _ = cv2.projectPoints(ray[0], zero, zero, Kp, d)
    print(f"{deg} deg  cv2.projectPoints "
          f"{p.ravel()[0]:10.1f}   "
          f"cv2.fisheye {q.ravel()[0]:6.1f} px")

Output

45 deg  cv2.projectPoints      309.9   cv2.fisheye  263.9 px
80 deg  cv2.projectPoints   -35572.0   cv2.fisheye  458.8 px

Both OpenCV functions return the hand-computed radii, including the pinhole’s −35,572 px at 80°.

In the wild

A circular fisheye makes a testable promise: the whole 180° field lands inside a disc whose radius follows from the focal length and the projection alone. At the edge of the field $\theta = \pi/2$ , so the equidistant rule predicts $r = f\pi/2$ and the equisolid rule predicts $r = 2f\sin(\pi/4) = \sqrt{2}\,f$ .

A Canon EOS 6D with the EF 8–15 mm at 8 mm, the frame cropped square, so its height is the sensor’s 24 mm short side and the pixel pitch is fixed at 6.579 µm:

r_{\text{equidistant}} = 8 \times 1.5708 = 12.57\ \text{mm} = \frac{12.57}{0.006579} = 1910\ \text{px}

r_{\text{equisolid}} = 8 \times 1.4142 = 11.31\ \text{mm} = \frac{11.31}{0.006579} = 1720\ \text{px}

Thresholding the exposed disc against the black corners and taking its bounding box gives a measured radius of 1752 px: 91.7% of the equidistant prediction and 101.9% of the equisolid one. Canon’s own product pages for this lens do not state its projection. Coverage of Canon’s launch material for the later RF 7–14 mm reports that the EF 8–15 mm uses equisolid-angle projection [4], and the photograph agrees with that.

A circular fisheye photograph with two circles drawn on it: the larger equidistant prediction and the smaller measured disc — The equidistant prediction in red, the measured disc in green. The equisolid prediction, 1720 px, would sit about 30 px inside the green circle. Source: “Fisheye lens Photography In Iran-Kashan City” by Mostafameraji (Wikimedia Commons), CC0.

What it means: the photograph favours equisolid (1.9% off) over equidistant (8.3% off), but only weakly. Measuring the same disc by its area instead of its bounding box gives 1567 px, 11% smaller, so the measurement’s own spread is larger than the gap between the two predictions. Part of that is the disc’s edge, which is a soft vignette rather than a line; part may be the square crop, whose disc centroid sits 193 px left of the frame centre. Separating those needs a calibration target, which is where calibration picks up. What the photograph does settle is this lesson’s point: the lens puts a whole hemisphere into a disc 23 mm across, and no $\tan\theta$ model can do that at any focal length.

Where this breaks

Rectifying a fisheye to a pinhole gives back the problem you started with. The obvious next step is to undistort into a rectilinear image, and it is available. But the output is a flat image plane again, so the field past about 120° is stretched by the same $1/\cos^2\theta$ as before: either it is cropped away, or the corners are stretched across so many pixels that nothing is left of the detail. Real fisheye pipelines keep the fisheye model and work in it.

CALIB_CHECK_COND rejects most of a usable dataset. Turned on, OpenCV’s conditioning check throws out 12 of these 15 views, while the fit over all 15 converges to 0.644 px. The check is not measuring what it sounds like it measures, and a pipeline that silently drops views on its advice is calibrating on a third of its data.

The polynomial is not guaranteed to be monotonic. Nothing in the fit forces $\theta_d$ to keep increasing with $\theta$ , and a badly conditioned set of coefficients can fold the outer field back on itself, as the pinhole model does here at 80°. Sanity-check the fitted curve over the whole range rather than trusting a low RMS, which is an average over the corners you happened to observe.

And the coefficients describe this lens only. These four numbers came from one lens photographed 15 times. They are not “the fisheye correction”, and using them on another camera is worse than using no correction at all.

Exercises

The plainest fisheye. Write equidistant(deg, f), the radius on the sensor of a ray arriving deg degrees off the axis under $r = f\theta$ .

Your turn

def equidistant(deg, f=336.8584):
    return 0.0  # radius on the sensor, px

print(equidistant(80))

Predict, then run. How far is the fitted lens from a pure equidistant design at 45° and at 80°?
The stretch. How many times more pixels does a degree of scene get at 60° and at 80° off the axis than at the centre, under $f\tan\theta$ ?

What you should see

These come from the “Lesson 4” cells of the unit notebook, image_formation.ipynb, and are the same on OpenCV 4.11 (this page) and 5.0.

470.34 px at 80°.
−0.24% at 45° and −2.45% at 80°: the polynomial is only trimming an equidistant lens.
4.0 times at 60° and 33.2 times at 80°, which is $1/\cos^2\theta$ .

Check yourself

Four questions on this lesson. Pick an answer to see why it is right or wrong.

Why can no lens that forms a flat rectilinear image reach 180°?
Under f tan θ, how much more is a degree of scene stretched at 60° off-axis than at the centre?
At 80°, the fitted pinhole model with distortion predicts −35,572 px. What does that mean?
What does the Kannala–Brandt model multiply its polynomial by, where Brown–Conrady uses tan θ?

The unit ends here

Four lessons, one thread: a hole makes a picture, a lens makes it bright, real glass bends the geometry, and past a certain field of view the geometry is a different one. The unit’s hub has the summary and the shared references. The next unit, image sensing, follows the light past the lens into the sensor, and the geometry sequel is camera geometry, which takes these two scalar equations and turns them into the matrix that geometry is written in.

Run it: every code block on this page has a cell in the unit’s notebook: open it in Colab.

References

[1] Kannala, J., & Brandt, S. S. (2006). A Generic Camera Model and Calibration Method for Conventional, Wide-Angle, and Fish-Eye Lenses. IEEE Transactions on Pattern Analysis and Machine Intelligence, 28(8), 1335–1340. doi:10.1109/TPAMI.2006.153

[2] Hartley, R., & Zisserman, A. (2004). Multiple View Geometry in Computer Vision (2nd ed.), ch. 6 “Camera Models”, §6.4 “Other camera models” (p. 174). Cambridge University Press.

[3] OpenCV Documentation (5.0). Fisheye camera model (cv::fisheye). docs.opencv.org/5.0

[4] Cameralabs. Canon RF 7-14mm f2.8-3.5L Fisheye review, 2026. Reports Canon’s statement that the RF 7–14 mm uses equidistant projection “versus the EF 8-15mm which used Equisolid-angle projection”. cameralabs.com

Tagged

camera-calibration opencv optics fundamentals