Image processing, computer vision, and computer graphics: one picture, three directions of the arrow

Three fields, one shared object — the digital image. Graphics turns a model into an image (forward), vision turns an image into a model (inverse), and image processing turns an image into a better image. Worked on a single red-ball scene, with a 1957→2026 timeline of how they split apart and re-merged.

Luis Condados · June 6, 2026 · Updated June 7, 2026

computer-vision computer-graphics image-processing fundamentals history

Image processing, computer vision, and computer graphics: one picture, three directions of the arrow

TL;DR — image processing, computer vision, and computer graphics are not three unrelated fields; they are the three directions you can point one arrow between a model and an image. Graphics goes model → image (forward synthesis). Vision goes image → model (the inverse problem). Image processing stays image → image. The forward direction is a well-posed function; the inverse is one-to-many and ill-posed, which is why vision is “harder” than graphics. For ~60 years the three drifted into separate conferences and textbooks — and since ~2020 (NeRF, diffusion, Gaussian splatting) they have collapsed back together, because a modern generative model is a denoiser, a renderer, and a scene-understander at once.

One mental model: a model ↔ image arrow with three directions.
Why vision ≠ graphics: rendering is a function; un-rendering is ambiguous.
The shared substrate: the same sampled image, the same camera, the same linear algebra.
Where it’s going: inverse graphics — render forward, differentiate, recover the scene.

If you’ve read how an image is formed and color spaces, this is the map that sits one level above it: the same pixel array, viewed as the meeting point of three disciplines.

The running example: a red ball on a table

We’ll use one scene for the entire article so you can compare the three fields on numbers you already understand. The scene is a model; the photo is an image.

Thing	Value
Sphere center $C$	$(0,\,0,\,5)$ metres (camera at the origin, looking down $+Z$ )
Sphere radius $R$	$1$ metre
Sphere colour	red (high red channel, low green/blue)
Light direction $L$	straight up, $(0,\,1,\,0)$
Camera	pinhole, focal length $f=500$ px, principal point $(c_x,c_y)=(320,240)$ , image $640\times480$

Everything below is this scene, viewed three ways:

The whole article in one picture. Three fields = three directions of one arrow between a model and an image.

Before we walk the three arrows one at a time, here is that scene in real 3-D — the same red ball, camera, light, and table you’ll meet below. Orbit the left panel to feel the geometry; the right panel is the actual photo the pinhole camera takes, rendered live. The crosshair and ring on the photo are the article’s own formulas ( $u = c_x + f\,X/Z$ and image radius $f\,R/Z$ ) drawn on top — so the picture and the numbers always agree.

Try it — the red-ball scene in 3-D

Scene — drag to orbit · scroll to zoom

What the camera sees — the render (640×480)

Switch tabs as you read on. Graphics moves the ball, light, and focal length and watches the render follow (the forward problem, next). Vision stages the scale–depth ambiguity in 3-D — a small, near ball and a big, far one that collapse to the exact same disc. Inverse recovers the ball by gradient descent. Each tab is one of the three sections below; come back and orbit it once you’ve read the math.

Computer graphics: model → image (the forward problem)

Intuition first: graphics answers “given a complete description of the world and a camera, what does the photo look like?” It is a function — feed in the scene, turn the crank, get exactly one image out. That makes it the well-behaved direction.

Two crank-turns matter for our ball: where a 3-D point lands on the sensor (projection) and how bright it is (shading).

Projection uses the pinhole model. The intuition: a point twice as far away looks half as big, so screen position scales with $X/Z$ .

$u = c_x + f\,\frac{X}{Z}, \qquad v = c_y - f\,\frac{Y}{Z}$

Shading uses Lambert’s cosine law. The intuition: a surface facing the light is bright; one tilted away is dim, falling off with the cosine of the tilt angle — which is just the dot product of the surface normal $N$ and the light direction $L$ .

$I = \rho\,\max(0,\; N \cdot L)$

Worked example — rendering one surface point of the red ball

Take the point on the ball whose outward unit normal is $N=(0.6,\,0.8,\,0)$ . It sits at $P = C + R\,N = (0.6,\,0.8,\,5)$ .

Where does it land? Substitute into the pinhole equations:

$u = 320 + 500\cdot\frac{0.6}{5} = 320 + 60 = 380$ $v = 240 - 500\cdot\frac{0.8}{5} = 240 - 80 = 160$

So the point lands at pixel (380, 160) — 60 px right of, and 80 px above, the image centre.

How bright is it? With the light straight up, $L=(0,1,0)$ and albedo $\rho=1$ :

$N \cdot L = (0.6)(0) + (0.8)(1) + (0)(0) = 0.8 \;\Rightarrow\; I = 0.8 \times 255 \approx 204$

What it means: that pixel is bright but not blown out. $\arccos(0.8)\approx 37°$ — the surface is tilted 37° away from the light, so it gets 80% of full brightness. Gut check: a point facing the light directly ( $N\cdot L=1$ ) would be pure white (255); a point on the shadowed side ( $N\cdot L<0$ ) clamps to 0. 204 sits sensibly in between.

Pinhole projection: similar triangles give

u/f = X/Z

. Double the depth

Z

and the same object covers half the pixels — the foundation graphics and vision share.

Try it — render one surface point

Move the surface point around the ball, push it deeper, or swing the light — and watch the pixel position and brightness follow the two formulas above.

The pinhole and shading models here are the bare minimum; real renderers integrate the full rendering equation [4] over every light path. But the shape is the same: scene in, image out, no ambiguity. For the geometry behind projection, see image formation and color spaces.

Digital image processing: image → image

Intuition first: image processing doesn’t try to understand the scene or invent it — it takes the pixels you already have and returns better (or differently-useful) pixels: denoise, sharpen, equalize, blur, compress. Input is an image; output is an image of the same kind.

The workhorse is convolution: slide a small kernel over the image and replace each pixel with a weighted sum of its neighbours. A $3\times3$ box blur uses equal weights — the intuition is “replace each pixel with the local average,” which smooths noise but also smears edges.

$I'(x,y) = \frac{1}{9}\sum_{i=-1}^{1}\sum_{j=-1}^{1} I(x+i,\,y+j)$

Worked example — blurring the bright edge of the ball

Take a $3\times3$ patch straddling the ball’s right edge (bright ball, dark table), brightness values:

\begin{bmatrix} 204 & 210 & 90 \\ 200 & 205 & 88 \\ 198 & 100 & 85 \end{bmatrix}

The box blur replaces the centre value ( $205$ ) with the mean of all nine:

$I' = \frac{204+210+90+200+205+88+198+100+85}{9} = \frac{1380}{9} \approx 153$

What it means: the centre pixel fell from a crisp 205 to a muddy 153. Averaging a bright ball pixel with its dark table neighbours pulls it toward the middle — which is exactly why blur softens edges. Gut check: the result must lie between the patch’s min (85) and max (210), and 153 does. The blur didn’t understand there was a ball; it just pushed pixels around.

A box blur is a pure pixel-in, pixel-out operation. For the full mechanics — strides, padding, separable kernels — see 2-D convolution on images.

The hand-computed 153 is one pixel; here is the same operation on a whole image. Switch the preset to Box blur (or edit the nine weights yourself) and watch edges soften — then try Sharpen or Sobel X to see other pixel-in, pixel-out kernels.

Kernel

÷ grayscale input

Upload image

Original

Filtered

Image processing is the oldest of the three (the first digital image was scanned in 1957) and the most closed: it never leaves pixel space. Convolution is also, not by accident, the operation that deep vision later built itself on top of — the same sliding window, now with learned weights instead of a fixed $\tfrac19$ .

Computer vision: image → model (the inverse problem)

Intuition first: vision runs the arrow backwards. Given the pixels, recover something about the scene — where the ball is, how big, what it is. It’s “inverse graphics”: undo the rendering. The catch is that rendering threw information away (depth, occluded surfaces, lighting), so the inverse has many possible answers. That’s what makes vision hard in a way graphics isn’t.

A first, classic pipeline: threshold the red channel to get a binary mask, then compute the blob’s centroid — the average position of the “on” pixels.

$\bar{x} = \frac{1}{N}\sum_{k} x_k, \qquad \bar{y} = \frac{1}{N}\sum_{k} y_k$

Worked example — locating the ball from the pixels

Threshold the red channel ( $\text{red} > 128$ ). For a $5\times5$ crop we get this mask (1 = ball):

\begin{bmatrix} 0&0&0&0&0\\ 0&1&1&1&0\\ 0&1&1&1&0\\ 0&0&1&0&0\\ 0&0&0&0&0 \end{bmatrix}

There are $N=7$ ball pixels. Summing their (0-indexed) column and row coordinates:

$\bar{x} = \frac{(1{+}2{+}3)+(1{+}2{+}3)+(2)}{7} = \frac{14}{7} = 2.0$ $\bar{y} = \frac{(1{\cdot}3)+(2{\cdot}3)+(3{\cdot}1)}{7} = \frac{12}{7} \approx 1.71$

We can even guess an apparent radius from the area, treating the blob as a disc ( $A=\pi r^2$ ):

$r = \sqrt{A/\pi} = \sqrt{7/\pi} \approx 1.49\ \text{px}$

What it means: vision compressed 25 numbers (the pixels) into 3 — centre $(2.0,\,1.71)$ and size. The arrow points the opposite way from graphics: image in, a compact model out. Gut check: the centroid sits in the upper-middle of the blob, which matches a shape that’s heavier in the top two rows. ✔

Vision turns the pixel blob into a few numbers about the scene. Modern detectors do the same job — image in, structured description out — with learned features instead of a hand-set threshold.

Try it — locate the ball

Click cells to change the mask and confirm the centroid moves the way you’d expect — toward wherever you add pixels — and that the area always equals the number of “on” cells.

That threshold-and-centroid trick is 1960s vision. Everything since — SIFT features, Viola–Jones faces, convolutional detectors, segmentation, depth estimation — is the same arrow (image → description), just with progressively better answers to the ill-posed inverse. For how we score those answers once we have them, see classification metrics and object-detection mAP.

Similarities and differences

The clean way to see it: all three share the image; they differ in which way the arrow points, and whether that direction is well-posed.

	Image processing	Computer vision	Computer graphics
Input	image	image	scene / model
Output	image	description / model	image
Arrow	image → image	image → model (inverse)	model → image (forward)
Core task	enhance, restore, transform	analyse, recognise, measure	synthesise, render
Well-posed?	usually	rarely (ill-posed)	yes (it’s a function)
Math core	signal processing, filtering	geometry, statistics, optimisation, ML	geometry, physics of light
Canonical text	Gonzalez & Woods [1]	Szeliski [2]	Foley et al. [3] / PBRT [5]

The deep similarity is the substrate. All three stand on the same sampled, quantised image (see image formation), the same pinhole camera, and the same linear algebra. Convolution is a filter in image processing, a feature extractor in vision, and a reconstruction filter in graphics — one operation wearing three hats.

The deep difference is direction. Forward (graphics) is a function: one scene → one image. Inverse (vision) is one-to-many, and here’s the proof on our own ball:

Why vision is harder than graphics — the scale–depth ambiguity

The image radius of a sphere is $\approx f\,R/Z$ . Consider two completely different scenes:

Scene A: $R=1$ m at $Z=5$ m $\;\Rightarrow\; 500\cdot\frac{1}{5} = 100$ px radius.
Scene B: $R=2$ m at $Z=10$ m $\;\Rightarrow\; 500\cdot\frac{2}{10} = 100$ px radius.

What it means: a small ball up close and a big ball far away render to the exact same image. Graphics had no trouble — each scene gives one image. But vision, handed that 100-px disc, cannot recover absolute size or depth from a single view. The forward map collapsed two scenes into one picture, and no amount of cleverness un-collapses it without extra assumptions (a second camera, a known size, a learned prior). That one-to-many collapse is why “un-rendering” is the hard direction.

Try it — the scale–depth ambiguity

Try to make the two scenes produce the same image radius (the article’s $1\text{ m}@5\text{ m}$ vs $2\text{ m}@10\text{ m}$ is one such pair). Any time $R/Z$ matches, the photo is identical — which is exactly the information a single view throws away.

How the three fields split — and re-merged (1957 → 2026)

For the first few decades the three arrows were studied by different people, in different labs, with different conferences (SIGGRAPH for graphics from 1974; CVPR for vision from 1983). Then deep learning, and then generative models, pulled them back into one room.

Three tracks, born within a few years of each other, drifting into separate disciplines and then converging in the 2020s.

The detailed chronology, by field:

Year	Field	Milestone	Why it matters
1957	DIP	First digitally scanned image (Kirsch, NBS), $176\times176$ [6]	The digital image exists at all
1963	CG	Sketchpad (Sutherland) [7]	Interactive computer graphics is born
1963	CV	”Machine Perception of 3-D Solids” (Roberts)	The blocks-world; vision as a research program
1966	CV	MIT Summer Vision Project (Papert)	The famous underestimate — “solve vision over a summer”
1969	DIP	CCD invented (Boyle & Smith, Bell Labs)	The sensor that made digital imaging practical
1971–72	DIP	CT scanner (Hounsfield)	Image reconstruction — DIP saving lives
1971–75	CG	Gouraud (1971), Phong (1975) shading [8]; z-buffer & texture mapping (Catmull, 1974)	Smooth, lit, textured surfaces
1981	CV	Optical flow (Horn–Schunck; Lucas–Kanade)	Motion from images
1982	CV	Marr’s Vision (posthumous) [9]	Vision as a layered computational theory
1986	CV	Canny edge detector	The classical edge operator still taught today
1986	CG	The Rendering Equation (Kajiya) [4]	One integral unifies all of light transport
1992	DIP	JPEG standard	Image processing meets the whole world
1998	CV	LeNet-5 (LeCun et al.)	Convolutional nets read digits — the seed of deep vision
1999	CV	SIFT (Lowe)	Robust local features; a decade of “features + classifier”
2001	CV	Viola–Jones face detection	Real-time vision on commodity hardware
2009	CV	ImageNet (Deng et al.)	The dataset that made deep learning possible
2012	CV	AlexNet	Deep CNNs win ImageNet; the modern era begins
2014	CV/CG	GANs (Goodfellow et al.) [10]	Networks generate images — vision starts doing graphics
2015	CV	ResNet [11], U-Net [12], FCN [13]	Very deep nets; dense prediction & segmentation
2015	CV/CG	Neural style transfer (Gatys et al.) [14]	Optimising an image — analysis and synthesis fuse
2017	ML	Transformers [15]	The architecture that later ate vision too
2020	CV	Vision Transformer (ViT) [16]	Attention replaces convolution for recognition
2020	CV+CG	NeRF [17]	A network is a renderable 3-D scene — inverse graphics goes mainstream
2020	CV/CG	DDPM diffusion [18]	Denoising (DIP!) becomes world-class generation (CG!)
2021	CV	CLIP [19]	Language ↔ image; text-conditioned everything
2022	CG	Latent/Stable Diffusion [20]	Photoreal synthesis from text, on a laptop
2023	CV+CG	3-D Gaussian Splatting [21]	Real-time radiance fields; vision and graphics share one representation
2023	CV	Segment Anything (SAM) [22]	Promptable, general-purpose segmentation

The convergence: vision as inverse graphics

This is the climax, and it’s the cleanest way to see why the three arrows are collapsing into one. The unifying idea is analysis-by-synthesis: to understand an image (vision), keep a model, render it (graphics), compare to the photo (image processing — a pixel difference), and nudge the model until the render matches. Recover the scene by running graphics in a loop and differentiating it. That’s differentiable rendering, and it’s the engine under NeRF [17] and Gaussian splatting [21].

The intuition: define a loss as the pixel difference between your render and the observed photo, then follow its gradient downhill in scene space.

$\mathcal{L}(\theta) = \sum_{\text{pixels}} \big(\mathcal{R}(\theta) - O\big)^2, \qquad \theta \leftarrow \theta - \eta\,\frac{\partial \mathcal{L}}{\partial \theta}$

Worked example — recovering the ball's position by inverse graphics

Let the one unknown be the ball’s horizontal position $\theta$ (its world $X$ ). The true scene has $\theta^{*} = 0.6$ m; we start with a guess $\theta = 1.0$ m. Near the true value the loss behaves like $\mathcal{L}(\theta)\approx k(\theta-\theta^{*})^2$ for some $k>0$ (small position errors produce proportional pixel errors). The gradient is then:

$\frac{\partial \mathcal{L}}{\partial \theta} = 2k(\theta - \theta^{*}) = 2k(1.0 - 0.6) = 0.8\,k \;>\; 0$

A positive gradient means the update $\theta \leftarrow \theta - \eta\,(0.8k)$ decreases $\theta$ , pushing our guess from $1.0$ back toward the true $0.6$ .

What it means: we recovered a fact about the scene (where the ball is) by rendering it forward and differentiating. Vision (image → model) was solved by running graphics (model → image) in reverse through gradient descent. The three arrows became one loop.

The analysis-by-synthesis loop. Graphics renders, image processing measures the gap, vision updates the scene — one cycle that contains all three fields.

Try it — recover the ball by descent

Drag $\theta$ off the target and hit Run: gradient descent rolls the guess downhill to $\theta^{*}$ — vision solved by differentiating graphics. Crank the learning rate $\eta$ too high and watch it overshoot, the same failure mode real optimisers fight.

The same collapse shows up everywhere in 2020s research:

Diffusion models [18][20] generate by denoising — the most classical image-processing operation — turned into world-class synthesis. The denoiser is conditioned on text or an image (vision). One model, three arrows.
NeRF / Gaussian splatting [17][21] take photos (vision input) and fit a scene you can render from new viewpoints (graphics output). The representation is shared.
GANs and style transfer [10][14] were the early bridge: networks that analyse and synthesise in the same breath. For how we even measure generated-image quality, see the FID metric.

Limitations & caveats

The taxonomy is a teaching tool, not a wall. Plenty of real work is deliberately in-between: edge detection is “image → edges” (DIP or CV?), image inpainting is enhancement and synthesis, super-resolution is processing and mild hallucination. Don’t argue about which bucket a paper goes in — the arrow diagram is for orienting, not adjudicating.
“Vision is just inverse graphics” is a useful slogan that overclaims. Much of human and machine vision is recognition and inference that never reconstructs a renderable 3-D scene (is this email-worthy? is that a tumour?). Analysis-by-synthesis is one powerful framework, not the definition of vision.
The forward/inverse framing assumes a generative image model exists. It’s cleanest for geometry and light. For high-level semantics (“is this person about to cross the street?”), there is no tidy renderer to invert.
The worked numbers are deliberately toy. Real cameras have lens distortion, non-square pixels, and noise; real shading has specular and indirect light; real detectors don’t threshold a channel. The arithmetic here is for intuition, not for copying into production. Treat every example as “the smallest thing that shows the shape of the idea.”
Dates are first-appearance markers, not clean origins. Most milestones had precursors, and “who was first” is genuinely contested for several (the birth of CV in particular). The timeline shows roughly when an idea entered the mainstream, not a patent office record.

Takeaways

Image processing, vision, and graphics are three directions of one arrow between a model and an image — image→image, image→model, model→image.
Forward (graphics) is well-posed; inverse (vision) is not — the scale–depth ambiguity on our ball is the whole reason vision is the hard direction.
They share a substrate (the sampled image, the camera, convolution, linear algebra) even when their goals diverge.
After ~60 years apart, analysis-by-synthesis and differentiable rendering fused them: NeRF, diffusion, and Gaussian splatting are all three arrows in one system.

References

[1] Gonzalez, R. C., & Woods, R. E. (2018). Digital Image Processing (4th ed.), ch. 1–3. Pearson.

[2] Szeliski, R. (2022). Computer Vision: Algorithms and Applications (2nd ed.), ch. 1. Springer. Free PDF: szeliski.org/Book.

[3] Foley, J., van Dam, A., Feiner, S., & Hughes, J. (2013). Computer Graphics: Principles and Practice (3rd ed.), ch. 1. Addison-Wesley.

[4] Kajiya, J. T. (1986). The Rendering Equation. SIGGRAPH ‘86, Computer Graphics 20(4), 143–150.

[5] Pharr, M., Jakob, W., & Humphreys, G. (2023). Physically Based Rendering: From Theory to Implementation (4th ed.). MIT Press. Free online: pbr-book.org.

[6] Kirsch, R. A. (1957). First digitally scanned image, U.S. National Bureau of Standards. History: nist.gov.

[7] Sutherland, I. E. (1963). Sketchpad: A Man-Machine Graphical Communication System. PhD thesis, MIT.

[8] Phong, B. T. (1975). Illumination for Computer Generated Pictures. Communications of the ACM, 18(6), 311–317.

[9] Marr, D. (1982). Vision: A Computational Investigation into the Human Representation and Processing of Visual Information. W. H. Freeman.

[10] Goodfellow, I., et al. (2014). Generative Adversarial Networks. NeurIPS. arXiv:1406.2661.

[11] He, K., Zhang, X., Ren, S., & Sun, J. (2015). Deep Residual Learning for Image Recognition. arXiv:1512.03385.

[12] Ronneberger, O., Fischer, P., & Brox, T. (2015). U-Net: Convolutional Networks for Biomedical Image Segmentation. MICCAI. arXiv:1505.04597.

[13] Long, J., Shelhamer, E., & Darrell, T. (2015). Fully Convolutional Networks for Semantic Segmentation. CVPR. arXiv:1411.4038.

[14] Gatys, L. A., Ecker, A. S., & Bethge, M. (2015). A Neural Algorithm of Artistic Style. arXiv:1508.06576.

[15] Vaswani, A., et al. (2017). Attention Is All You Need. NeurIPS. arXiv:1706.03762.

[16] Dosovitskiy, A., et al. (2020). An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. arXiv:2010.11929.

[17] Mildenhall, B., et al. (2020). NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. ECCV. arXiv:2003.08934.

[18] Ho, J., Jain, A., & Abbeel, P. (2020). Denoising Diffusion Probabilistic Models. NeurIPS. arXiv:2006.11239.

[19] Radford, A., et al. (2021). Learning Transferable Visual Models From Natural Language Supervision (CLIP). ICML. arXiv:2103.00020.

[20] Rombach, R., et al. (2022). High-Resolution Image Synthesis with Latent Diffusion Models. CVPR. arXiv:2112.10752.

[21] Kerbl, B., Kopanas, G., Leimkühler, T., & Drettakis, G. (2023). 3D Gaussian Splatting for Real-Time Radiance Field Rendering. SIGGRAPH. arXiv:2308.04079.

[22] Kirillov, A., et al. (2023). Segment Anything. ICCV. arXiv:2304.02643.

Tagged