An Image Is an Array: Shape, dtype, and Why OpenCV Hands You BGR

TL;DR — one photograph, held three ways: a 24.7 MB mosaic of 12-bit counts, a 74.2 MB 16-bit development, and the 37.1 MB 8-bit array imread would hand you. In that last one, two thirds of the numbers were never measured by anything — they were interpolated. Indexing is (row, column). The channels come back B, G, R. And they are 8-bit integers, so adding 60 to a value of 207 gives you 11.

Where we are

Part of The Image as Data. Unit 1.2 finished with a number in a pixel and what it counts. This lesson picks that number up and asks a duller question with a longer tail: what is the container, and what does the container do to you?

Everything below is one frame — _MDF0005.NEF, the 1/45 s exposure of the lit chart in “Luxo Double Checker” — held three different ways.

Three files, one photograph

Intuition first: a camera measures one number per photosite, and everything else you have ever seen from that camera is a reconstruction of the other two thirds.

The pixel is the same place in all three rows: row 1910, column 2474, the centre of the lit chart’s white patch. The NEF on disk is 19.5 MB, which is smaller than the mosaic it decodes to, because raw files are losslessly compressed.

What it means: the mosaic is the only one of the three that is a measurement. It holds one number per photosite and nothing else. The other two hold three numbers per pixel, and since the sensor only ever measured one of them, two thirds of every developed image is interpolated — a number no instrument produced. Green gets half the photosites and red and blue a quarter each, which is why green survives demosaicing best.

The array in memory

An image is stored row by row, and within a row the three channels of a pixel sit next to each other. That layout has a name — interleaved — and it is why img[100, 200] gives you three numbers rather than one.

Two conventions follow from that picture, and both of them bite.

Indexing is (row, column), which is (y, x). Gonzalez & Woods spend a whole subsection on this — §2.4, Linear vs. Coordinate Indexing (p. 54) [1] — because the array convention and the geometric convention disagree, and every drawing function in OpenCV takes the geometric one. img[100, 200] and cv2.circle(img, (100, 200), …) refer to two different places.

The channels come back B, G, R — imread decodes into that order and every OpenCV call assumes it [2]. In this frame, red and blue differ at 7,960,554 pixels — 64% of the image — so getting the order wrong is not a subtle tint, it is a different photograph.

import cv2

img = cv2.imread("frame.png")     # BGR, dtype uint8
print(img.shape, img.dtype)       # (2868, 4312, 3) uint8
b, g, r = img[1910, 2474]         # row first, column second
print(int(b), int(g), int(r))     # 160 189 193

#include <opencv2/opencv.hpp>

cv::Mat img = cv::imread("frame.png");        // BGR, type CV_8UC3
cv::Vec3b px = img.at<cv::Vec3b>(1910, 2474); // (row, col), BGR order
uchar b = px[0], g = px[1], r = px[2];        // 160 189 193

The numbers are integers, and integers run out

Intuition: a uint8 holds 0 to 255 and nothing else, so “make it brighter” has to decide what happens at the top. NumPy and OpenCV decide differently.

import numpy as np

patch = img[1883:1971, 2448:2481]
brightest = patch.max()                             # 207
np.uint8(brightest) + np.uint8(60)                  # 11  — wraps
cv2.add(patch, np.full_like(patch, 60)).max()       # 255 — saturates
np.clip(patch.astype(np.float32) + 60, 0, 255).max()  # 255 — no surprises

# Note the trap in the obvious version: (patch + 60).max() is 255, not 11, because
# some pixel sat at exactly 195 and landed on the ceiling without going over it.
# The maximum of the wrapped array is not the wrap of the maximum.

// cv::Mat arithmetic saturates by default: this is cv::add's behaviour, and the
// wrap only appears if you index the buffer and do the arithmetic yourself.
cv::Mat brighter;
cv::add(patch, cv::Scalar::all(60), brighter);   // 255, clamped

uchar raw = patch.at<cv::Vec3b>(0, 0)[2];
uchar wrapped = static_cast<uchar>(raw + 60);    // 11, if raw was 207

A slice is not a copy

One more property of the container. Slicing an array gives you a view — a window onto the same memory — so writing into the slice writes into the original. Fancy indexing gives you a copy, and writing into that changes nothing.

Measured on this frame at row 1883, column 2448, the pixel reads (167, 194, 199) in imread order. Write zero into a slice of it and the original reads (0, 0, 0). Write zero into a fancy-indexed copy and the original still reads (167, 194, 199).

region = img[1883:1947, 2448:2512]
region[0:8, 0:8] = 0          # a view — img is now modified
copy = img[[1883, 1884, 1885]]
copy[:] = 0                   # a copy — img is untouched

What it means: neither behaviour is wrong, but they are opposite, and the code that mixes them up fails silently in both directions — a “non-destructive” preview that destroys the original, or a fix that never lands.

Now you try

Hover a tile to read its index and its three numbers. Then flip the channel order and watch blue and green trade places, and push the addition up with wrap on until the bright pixels go dark.

In the wild

The chart above is a bench setup. Here is the same fact on a photograph, and it is the version you will actually meet: a 3608 × 5509 picture of a red telephone box outside St Paul’s. On disk it is a 14.6 MB JPEG. Decoded, it is a 59.6 MB array — four times the file, because a JPEG is compressed and an array is not.

Red and blue differ at 19.8 million pixels — 99.7% of the frame. Compare that with the 64% in the lab scene above: that frame is a dark room, and in near-black pixels red and blue are both near zero, so they agree by accident. In a daylight photograph almost nothing agrees, and reading the channels in the wrong order is not a tint. The box itself, found by hue rather than by hand, has a mean BGR of (45, 40, 215). Read those same bytes as RGB and it becomes (215, 40, 45) — and the median hue of the box moves from 350°, which is red, to 242°, which is blue.

What it means: the lab patch made the point arithmetically — 64% of pixels differ. The photograph makes it unmissable, and it also shows the failure mode you should expect: not a subtle cast, but a scene that is confidently, cheerfully wrong.

Where this breaks

The array tells you what the numbers are and nothing about where they came from. Row 1910 and row 1911 are adjacent in memory; whether they are adjacent in the scene depends on how far apart the photosites are and what fell between them. Nothing in the shape, the dtype or the channel order answers that.

This lesson also quietly assumed that reading one pixel is a sensible thing to do. The figure above says otherwise: the “flat” white patch runs from 180 to 189. A single pixel is a sample of a noisy process, and the grid it was sampled on is the subject of the next lesson.

Next

Sampling: what one pixel actually covers — the grid underneath the array, what falls between its points, and why shrinking an image the obvious way invents patterns that were never in the room.

Further reading

• Go deeper: Gonzalez & Woods, Digital Image Processing (4th ed.), §2.4 (Image Sampling and Quantization, p. 47) [1] — the subsection Representing Digital Images (p. 49) is this lesson in textbook form.
• The API: the OpenCV Mat documentation for the container itself, and what at<> costs against a raw pointer.
• Related on CondadosAI: what the number in a pixel counts is where these integers come from · 2D convolution is the first thing anyone does to the array once they have it.

References

[1] Gonzalez, R. C., & Woods, R. E. (2018). Digital Image Processing (4th ed.), §2.4 (Image Sampling and Quantization, p. 47), incl. Representing Digital Images (p. 49) and Linear vs. Coordinate Indexing (p. 54). Pearson. Detailed table of contents.

[2] OpenCV. cv::Mat — the basic image container and imread, OpenCV 5.0.0 documentation. Docs.

[3] Fairchild, M. D. (2007). The HDR Photographic Survey. Proceedings of the IS&T 15th Color and Imaging Conference, pp. 233–238. doi:10.2352/CIC.2007.15.1.art00044 — the scene this unit measures. Used for research and non-commercial publication, as its terms require; images are downloaded, never redistributed.