Colour Science & Perception
what the eye settles, and what it cannot
Concept

Colour filter array — where it appears

The mosaic of coloured filters over a sensor that gives each pixel one of three channels rather than all three. What is missing at every pixel is then reconstructed, which is why a camera's colour and its resolution are not independent.

Named by 9 essays across one field — each of them below, with the objects they name alongside it.

A camera's spectral sensitivities, after the infrared-cut filter. Silicon quantum efficiency times the colour-filter dye times the infrared-cut filter, per channel, on a grid running to 1100 nm rather than to 780. With the filter removed, 68 per cent of the area under the three curves lies beyond the visible band, and all three curves are the same curve out there.

A camera is a fourth observer

The 1931 functions, the 1964 functions and a person's own cones are three sets of three curves that collapse a spectrum onto three numbers. A camera is a fourth, built from silicon and dye rather than from pigment and neural wiring, and it agrees with none of them.

imaging · Capture Everything between the photons and the picture, and what each stage decides. The 8 stages of a camera pipeline. Only the second is physics; every one after it is a decision somebody made, and the reason two cameras pointed at the same scene disagree is that they made different ones.

Raw is not a picture

A raw file is three integrals per site and a list of decisions nobody has taken yet. Every one of those decisions has a defensible answer and none of them has a correct one, which is why two converters open the same file and disagree.

imaging · Capture The blue channel, as the three things multiplied to make it. Silicon's quantum efficiency, the colour-filter dye's transmittance, and their product. The dye is the only stage carrying any colour information and it is clear above about 800 nm — transmittance 0.92 at 900 nm against 0.06 at 550.

Silicon sees past the visible

A sensor's response ends at 1107 nanometres because that is the band gap, and the colour-filter dyes have stopped absorbing three hundred nanometres before it. So all three channels measure the same thing over the last third of the range, and every published sensitivity plot is drawn after the component that hides it.

imaging · Capture How much of what an unfiltered sensor records is invisible. The share of a camera's raw signal coming from beyond 780 nm, against the colour temperature of the lamp, for one surface with a near-infrared reflectance of 0.62. Without the filter it runs from 98 per cent at 2000 K to 51 at 9000; with it, under five per cent everywhere.

The filter that makes colour possible

An infrared-cut filter is not a refinement on a colour camera. Removing it collapses the separation between the three channels by a factor of nearly nine under tungsten, and the component that keeps colour photography working is the one nobody photographs.

imaging · Capture A grey edge, reconstructed from a Bayer row, arrives coloured. Above: an achromatic step through 24 sensor sites, with green sampled on the even ones and red on the odd. Interpolating each channel separately reconstructs them from data taken on either side of the edge, so their ratio moves. Below: the resulting chroma, peaking at 144 per cent of the local mean, and 112 per cent once colour differences are interpolated instead.

A grey edge arrives coloured

A sensor site measures one channel and the other two are interpolated from neighbours that sat somewhere else. Across a black-and-white step that reconstruction gives an achromatic scene a chroma of 144 per cent of its own local mean, and nothing in the scene or the sensor was coloured.

imaging · Capture The basis a camera balances in is a different basis for every light. A camera's white balance is a per-channel gain on raw values, which is a von Kries adaptation in whatever basis the filter dyes give it. That basis is not a property of the dyes alone: it is the dyes and the light in the room, and it moves when the light does. Each bar is how far the basis has turned, in degrees, from where it sits under D65. A sensor satisfying the Luther condition would have a bar of exactly zero on every row, because for such a sensor the light cancels — which is the one property nobody buys a sensor for.

A camera balances in another basis

White balance is a per-channel gain on raw values, which makes it a von Kries adaptation in whatever axes the filter dyes happen to give. Those axes are not a property of the dyes alone — they move with the light, by up to seventeen degrees across the adaptation census — and the sensor for which they would not move is the one that adapts worst of all.

imaging · Capture Which of a camera's three dyes each direction moves. A grid with one column per direction — stiffest on the left, flattest on the right — and one row per parameter of a camera's three dyes. Each cell's bar length is that parameter's share of that direction, so a column with one long bar is a direction that moves one thing. The stiffest column is dominated by blue centre, at a weight of 0.96. The flattest column is spread across blue width, red width, green width — a combination rather than any single number, which is why a specification listing one tolerance per parameter cannot express it.

Where a camera is blind to itself

A colour filter array is six numbers — three dye centres and three bandwidths — and how well the resulting sensor adapts is far more sensitive to some combinations than to others. The stiffest direction is almost entirely where the blue dye sits. The flattest is all three bandwidths at once, and the design can move twenty-five times further along it for the same cost.

imaging · Capture The arguments a standard observer does not have. Seven choices inside a set of colour-matching functions, each with the shape it takes and what it is worth in ΔE₀₀ on a red pigment under a 6500 K radiator. Six are measurements: a field size, an age, a macular density, a cone optical density, three peak wavelengths and a rod contribution. The seventh is not — a change of basis is a change of curves and not a change of observer, and its entry is exactly zero because the space an experiment measures is what an observer is. Printing that zero beside the others is the clearest statement of what the other six are measurements of.

A sensor has no lens

A camera is a fourth observer and it is the only one with none of the six arguments this round measured. It does not age, it has no macular pigment, its response does not broaden with density and its peaks do not vary — so it is perfectly reproducible and is not any of the observers it is fitted to reproduce.

imaging · Capture One infrared-cut filter, crossed at four angles. The transmittance of the same interference filter for light crossing it at the centre of a frame and at three steeper angles, from 560 to 760 nm. Its half-transmission edge is at 665 nm straight on and moves to 658, 646 and 630 nm, following the edge wavelength times the square root of one minus the squared sine of the angle over the square of the stack's effective index, 1.8. The deepest reds are what it takes away, and a lamp decides how much light there was in them.

The corner of the frame has another filter

A camera's infrared-cut filter is an interference stack, and light crossing it at an angle sees its edge moved towards the blue — from 665 nm at the centre of a frame to 646 at twenty-five degrees. The corner's red channel loses a tenth of its response under daylight and a sixth under tungsten. A grey-card gain map makes the corner exactly right under the lamp it was made under; under a tungsten lamp the same grey is 2.6 colour differences wrong, under a white LED 4.2, and even under its own lamp the coloured patches are not corrected, because a moved edge is not a gain.

imaging · Capture

Named alongside it

The objects these essays reach for when they reach for this one.

Camera rawSpectral sensitivitySiliconColour matrixIlluminantInfraredLuther conditionWhite balanceCalibrationChromatic adaptationDemosaicPlanck's law

All concepts