Quantization: How Many Bits a Pixel Deserves

TL;DR — rounding to fewer levels costs $\Delta/\sqrt{12}$ of error, and on this camera’s mid-tones that formula predicts the measurement to within 2%. Set it equal to the read noise unit 1.2 measured and the crossing lands at 9.3 to 9.5 bits — below that, quantization is the dominant error; above it, the converter is writing down noise. A 12-bit file on this sensor therefore carries about two and a half bits of nothing.

Where we are

Part of The Image as Data. Lesson 2 put the samples in the right places. This one asks how finely each sample is allowed to be written down — and, for the first time in this unit, the answer depends on something the sensor already did.

Rounding, and what it costs

Intuition: a quantizer with step $\Delta$ replaces every value with the nearest multiple of $\Delta$, so no value moves by more than half a step, and on average it moves by rather less.

If values are spread evenly inside a step, the RMS error of that rounding is

The NEF here is 12-bit (its white level is 4095), so reducing to $b$ bits means $\Delta = 2^{12-b}$. Measured on the green photosites inside the lit chart’s patches:

What it means: the textbook formula lands within 6% at 10 bits and within 2% everywhere below. That is a formula predicting a photograph, which is rarer than it sounds — and it works here because chart patches are mid-tones, so values really are spread fairly evenly inside each step.

Run the same reduction over the whole frame and the agreement collapses: measured RMS at 4 bits is 23.1 DN against the predicted 73.9, three times optimistic. Nothing is wrong with the formula. This frame is a dark room, most of its values pile up in the first couple of steps rather than spreading evenly inside them, and the assumption the formula rests on is simply false there. Both rows are worth publishing: the assumption is the interesting part.

The question bit depth actually answers

Here is where unit 1.2 pays off. That unit measured the dark-frame noise of every channel on this sensor. If the quantization step is much smaller than that noise, the extra bits are recording the noise rather than the scene.

What is lost when the levels go

The chart’s patches make the loss countable. At each bit depth, how many of the 24 patches still land on distinct codes?

What it means: down to 8 bits almost everything survives — 20 of 22. At 4 bits the chart has 7 distinguishable tones left, the neutral pair 23 and 24 collapse into one code, and the black patch reaches zero, which means it is no longer dark, it is absent. Shadows go first, every time, because the codes are evenly spaced and the dark tones are the ones packed closest together.

import numpy as np

def requantize(values, bits, source_bits=12):
    """Round to `bits` levels, then put it back on the original scale."""
    step = 2 ** (source_bits - bits)
    return np.clip(np.rint(values / step) * step, 0, 2**source_bits - 1)

error = requantize(green, 8) - green
rms = np.sqrt((error ** 2).mean())     # 4.630 DN, against 16/sqrt(12) = 4.619

// Rounding, not truncation: truncating biases every value down by half a step and
// costs twice the RMS error for nothing.
cv::Mat requantize(const cv::Mat& green, int bits, int source_bits = 12) {
    const double step = std::pow(2.0, source_bits - bits);
    cv::Mat scaled, rounded;
    green.convertTo(scaled, CV_32F, 1.0 / step);
    // No cv::round on a Mat: add a half and truncate towards zero, which is the
    // same thing for the non-negative values a raw plane holds.
    cv::add(scaled, cv::Scalar::all(0.5), scaled);
    scaled.convertTo(rounded, CV_32F, step, 0.0);
    cv::Mat out;
    cv::min(rounded, std::pow(2.0, source_bits) - 1, out);
    return out;
}

In the wild

The lab scene could not show banding. A dark room with one lamp has no broad smooth gradient in it — every wide tonal range in that frame is an edge — which is why this lesson has measured levels rather than shown them. A photograph fixes that in one step.

Twilight over La Silla: the top third of the frame is sky and nothing else. Across 1449 pixels of height, that sky travels just 51 code values. A gradient that gentle is where bit depth is spent, and where it runs out first.

What it means: a band 28 pixels tall is invisible and a band 121 pixels tall is a contour line across your sky. The reason skies band before anything else is not that skies are special — it is that they spread very few code values over very many pixels, so each step has to stretch a long way before the next one arrives.

One thing the numbers say that the pictures do not. Counting those bands took a smoothing pass first, because the photograph’s own grain flips a code on nearly every row — 1240 changes at 8 bits before smoothing, 51 after. That is not a nuisance, it is the reason banding is rarer than this table suggests: noise dithers the gradient, and a clean synthetic ramp bands far worse than a real photograph of one.

Clipping is a different loss

Quantization loses precision everywhere. Clipping loses the value outright, and no bit depth brings it back. On this frame, 16,218 red photosites (0.53%) and 5,029 blue sit at the ceiling, while the two greens have essentially none. Tungsten light is red-heavy, so red saturates first — and the two greens top out near 3878 DN rather than 4095, which is the per-channel ceiling unit 1.2 measured rather than the one the file declares.

Now you try

Fill the well, then drop the bit depth and watch two dark tones become one number.

Where this breaks

The crossing is this camera at ISO 100. Raise the ISO and the noise rises with it, so the crossing moves down and fewer bits are justified — unit 1.2 measured one ISO setting and said plainly that it could not generalise, and that limit is inherited here whole.

The uniform-error formula also assumes the quantizer is uniform, which raw converters are and display encodings are not: an 8-bit JPEG spends its codes unevenly on purpose, which is why 8 bits looks better than this lesson’s arithmetic suggests it should. That is the tone curve, and it belongs to unit 1.2 [4].

And this lesson has said nothing about what the three numbers at a pixel mean. It has treated them as three independent scalars to be rounded, which is exactly what a quantizer does and exactly what a colour is not.

Next

Colour spaces: choosing axes you can threshold — what coordinate systems those three numbers can be written in, and why moving between them makes some questions easy and others impossible.

Further reading

• Go deeper: Gonzalez & Woods, Digital Image Processing (4th ed.), §2.4 (Image Sampling and Quantization, p. 47) [1]; Spatial and Intensity Resolution (p. 55) is the textbook treatment of what these bits buy.
• The noise this lesson spends: sensor noise and dynamic range, where the 1.9 DN was measured [2].
• Related on CondadosAI: the camera response curve — why a display encoding does not quantize uniformly.

References

[1] Gonzalez, R. C., & Woods, R. E. (2018). Digital Image Processing (4th ed.), §2.4 (Image Sampling and Quantization, p. 47), incl. Spatial and Intensity Resolution (p. 55). Pearson. Detailed table of contents.

[2] European Machine Vision Association (2021). EMVA Standard 1288, Release 4.0 Linear: Standard for Characterization of Image Sensors and Cameras, effective 21 June 2021. emva.org — the characterisation standard behind the gain and read-noise figures this lesson spends.

[3] Fairchild, M. D. (2007). The HDR Photographic Survey. Proceedings of the IS&T 15th Color and Imaging Conference, pp. 233–238. doi:10.2352/CIC.2007.15.1.art00044 — the scene this unit measures. Used for research and non-commercial publication, as its terms require; images are downloaded, never redistributed.

[4] International Electrotechnical Commission (1999). IEC 61966-2-1: Multimedia systems and equipment — Colour measurement and management — Part 2-1: Default RGB colour space — sRGB. The non-uniform encoding an 8-bit display file uses. Standard.