The algorithms that guess the light

Colour constancy is the default: the visual system discounts the illuminant so thoroughly that most people go a lifetime without noticing that a white shirt is orange indoors and blue in shade. That essay shows the phenomenon. This one asks the engineering question that follows, because a camera has to answer it before it can write a file, and it has to answer it in software with none of the visual system’s advantages.

The question is: given only the light that arrived, what colour was the light?

The problem cannot be solved #

What reaches a sensor from a surface is the product of two spectra: what the light was, and what fraction of it the surface returned. Integrated through the sensor’s three channels, that becomes three numbers which are a product of two things nobody measured separately.

No arithmetic separates a product into its factors. A red surface under white light and a white surface under red light send the same photons, and there is nothing in the photons that distinguishes them. The problem is underdetermined in exactly the way that word is meant: there are more unknowns than equations, and no cleverness closes the gap.

Every estimator therefore imports information from outside the image. Each one assumes something about the world, uses that assumption to break the tie, and is correct exactly to the degree the assumption holds. The formula is the uninteresting part. The assumption is the whole thing, because the assumption is what a scene can be built to violate.

Grey-world, and what happens to it #

Grey-world is the oldest and simplest: average each channel across the scene, and declare that average to be the illuminant, on the assumption that the average surface is neutral.

Handed a scene that satisfies its premise — seven broad reflectances spread across the spectrum, averaging to something close to neutral — it returns the illuminant with an angular error of 2.5°. That is respectable; a few degrees is around the threshold at which a white-balance error becomes visible, which is itself a claim that needs a difference formula to make precise.

Handed five greens and nothing else, it returns 43.1°.

Forty-three degrees is not a degraded estimate. It is a different colour entirely, and the failure mode is the one every photographer recognises: a picture of a lawn, or a forest, or a green-painted room, comes back looking magenta, because the camera decided the scene’s greenness was the light rather than the grass.

The failure is not a bug and cannot be patched. Grey-world is doing exactly what it claims to do, on a scene whose average reflectance is emphatically not neutral, and returning the answer its premise implies.

Max-RGB, and the scene it needs #

Max-RGB takes the maximum in each channel instead of the mean, on the assumption that something somewhere in the scene reflects fully in every band — that there is, in effect, a white card in the picture.

When that is true the problem stops being underdetermined at all. A perfect white reflector returns the illuminant unchanged, so reading it off is not estimation but measurement. Handed a scene containing one, max-RGB returns the illuminant with an angular error of 0.0000° — exact to the precision of the arithmetic.

That exactness is the assertion this file is anchored on, because it is the one case with a knowable right answer and it catches a transposed matrix or a forgotten normalisation — errors which leave every other number in the computation looking entirely plausible.

The weakness is the obvious one. Most photographs contain no white. Many contain a specular highlight, which is nearly as good and is why highlight-based methods exist. Many contain neither, and then max-RGB is reading the brightest coloured thing in the frame and calling it the light.

There is a second and nastier failure: a clipped channel. A sensor channel that has saturated reports its maximum regardless of what the scene did, so max-RGB on an overexposed image reads the clipping level in every channel and returns neutral, confidently, always. That is a failure of the input rather than of the estimator, and it belongs with the other cases where what an instrument reports is not what arrived.

Nobody wins #

The interesting result is not that each estimator has a failure case. It is that the failure cases do not nest, so the estimators are not ranked.

Across the four scenes drawn in the hero figure, all lit by the same illuminant, the winners are grey-world and max-RGB — more than one, on scenes differing only in what surfaces are in them. Grey-world wins on the balanced scene, where its premise holds. Max-RGB wins on the green scene and on the white-card scene, where its premise holds and grey-world’s does not.

Which algorithm is best is therefore a property of the scene, not of the algorithm. A camera that commits to one has not chosen which algorithm to trust; it has chosen which photographs to be wrong about.

That is why every real camera does something more complicated: switching between estimators on scene statistics, weighting them, learning a prior over illuminants from a large corpus, or detecting faces and skin and using those as the reference surface. All of those are ways of importing more outside information, which is the only thing that can help, because the underlying problem has not become any less underdetermined.

What the scenes are made of #

Every scene here is a set of spectral reflectances under a spectral illuminant, integrated through a named observer — not a list of RGB triples.

That is not fastidiousness. The failures being demonstrated are failures about light, and a scene assembled from RGB values would already have thrown away the part that makes them interesting. In particular, a scene built from RGB cannot show what happens when the illuminant’s spectrum has structure the surfaces do not, which is precisely the situation a fluorescent tube creates and precisely the situation in which estimators trained on daylight fall over.

The surfaces themselves are smooth single-band reflectances — a base plus a Gaussian — rather than measurements of real pigments. That is a deliberate limit and it is stated here rather than left to be discovered: these are constructed surfaces, and a ranking between two constructions is a fact about the constructions, which is a lesson this site paid for once already in its expansion phase and does not intend to pay for twice. What the figures establish is the structure of the failures, not a league table.

Linear sRGB stands in for a camera’s raw channels. It is the wrong sensor for any particular camera and the right one for this argument, since the point is about the shape of the problem rather than about a device.

Two more scenes say that the failures are properties of the assumptions rather than of the four scenes the table happened to use.

The symmetry is worth stating because it is the clearest evidence that these are not defects. A warm scene under neutral light and a neutral scene under warm light are, to grey-world, the same input. They are the same input to any method that looks only at the average, and they would be the same input to a perfect one, because they are the same photons up to the ambiguity nobody can resolve. Grey-world is not making a mistake; it is returning the answer that its assumption uniquely determines, on a case where the assumption is false and no information in the image says so.

What was computed, and how #

Four assertions, each of which would have caught a different error.

Max-RGB recovers the illuminant from a scene containing a perfect white, to under a degree. This is the anchor: it is the one configuration with a knowable right answer, and it fails loudly if the sensor matrix is transposed or the normalisation dropped.

Grey-world is exact when the scene averages to neutral and wrong when it does not — both halves in one assertion, because either alone is uninformative. An estimator that is always right is not doing estimation; one that is always wrong is not an estimator.

No single estimator is best on every scene. The strong claim, checked by collecting the winner of each of four scenes and requiring more than one distinct winner.

Angular error ignores brightness, checked by scaling an estimate by thirty-seven and requiring the score not to move.

The refusals matter as much: a scene with no surfaces, a one-patch scene asked for its edges, a Minkowski norm below 1, and an estimate of zero length are all fed in on purpose and all must be rejected. The last is the interesting one — a zero-length estimate has no direction, so the angle to it is undefined, and returning zero degrees for it would report perfect agreement from an algorithm that had returned nothing.

The measure, and why it ignores brightness #

Estimators are scored by angular error: the angle between the estimated illuminant vector and the true one, in degrees, in linear sensor space.

The choice of measure is doing real work. The one thing no estimator can recover is how bright the light was — a dim white light and a bright white light produce sensor values differing by a scale factor, and nothing distinguishes that scale factor from the surfaces being uniformly darker. So a measure that penalised getting the magnitude wrong would be penalising algorithms for failing at something impossible.

Angular error is invariant to the magnitude by construction, which is asserted rather than assumed: scaling an estimate by a factor of thirty-seven changes its score by less than 10⁻⁹, and an exact estimate scores zero.

That collapse is the deepest limit and it is easy to miss among the more visible ones. Every estimator above reduces the illuminant to three numbers before it starts, because three numbers is all a three-channel sensor delivers. So none of them can distinguish two illuminants that are metameric under the sensor — and two such illuminants will render the same scene differently, in ways no white balance can undo, because the difference is in bands the sensor never separated.

This is the reason why correcting a photograph taken under a cheap fluorescent tube never quite works. The white balance can be made exactly right, in the sense that a white card comes out white, and the coloured surfaces are still wrong, because the tube’s spiky spectrum interacted with those surfaces’ reflectances in a way that three numbers cannot record and three numbers cannot repair.

What a degree is worth #

A few degrees is around the threshold at which a white-balance error becomes visible is left as a claim needing a difference formula, and the formula is one this collection already has, so the claim can be settled rather than deferred.

Take a neutral surface, correct it with an estimate that is off by θ degrees, and measure the result against the true white in ΔE₀₀. In the green–magenta direction — the one grey-world fails in on the green scene — the answers are:

One degree is already worth three ΔE₀₀, which is three times the tolerance a paint contract is written in and well past anything a careful eye would call invisible. A few degrees is not a threshold; it is several multiples of one.

That reprices grey-world’s good case. Its 2.5° on the balanced scene — the scene built to satisfy its premise — costs 7.2 ΔE₀₀ on a neutral surface. The word used for it here is respectable, and in the unit this collection uses for everything else it is seven times a tight tolerance. The estimator that gets its assumption granted still delivers an error a paint supplier would reject outright.

A scene with a balanced set of surfaces under a tungsten lamp is the case grey-world was designed for, and it is where it wins.

The scale is compressive at the top and that changes the story #

The same table says something about the failure case, and it cuts the other way.

Grey-world’s collapse on the green scene is a 17-fold rise in angular error, 2.5° to 43.1°. In ΔE₀₀ it is a rise from 7.2 to 32.5, a factor of 4.5. Angular error is a strongly compressive scale at the top end: the marginal cost of a degree falls from 3.2 ΔE₀₀ near zero to under 0.8 by forty degrees, because a chromatic adaptation transform pushed far enough runs out of room in a bounded colour space.

So forty-three degrees is not a degraded estimate, it is a different colour entirely is right about the direction and overstated about the distance. The catastrophic case is four and a half times the good case, not seventeen — which is still large, and is the same order of magnitude rather than a different one. Both cases are far outside tolerance, and that is the honest summary: grey-world is not usable on a green scene, and it is not usable on the balanced scene either, by the standard this collection applies to a painted panel.

A scene of warm surfaces under a warm lamp is where every assumption about the surfaces fails at once.

Angular error is anisotropic, and nothing says so #

The measure’s one stated property is that it ignores brightness, asserted by scaling an estimate by thirty-seven and requiring the score not to move. That invariance is real and it is the right design. The property that goes unstated is that the measure is not invariant to direction.

Sweeping the estimate’s error all the way round the plane of chromaticities at a fixed angular error:

The same angular error is worth twice as much in one direction as in another, consistently, across the whole range. So a table of angular errors is not a table of comparable quantities: an estimator that fails by 3° towards green is doing more damage than one failing by 5° towards amber, and the scoreboard reports the second as worse.

None of that invalidates the essay’s finding, and it is worth saying why. The finding is that the winner changes with the scene, and a measure that is monotone in the error along each direction cannot manufacture a change of winner out of nothing — max-RGB’s exact zero is exact under any measure, and grey-world’s 43.1° beats nothing under any measure. What the anisotropy affects is the margins, which the essay does not lean on.

It does mean one thing for anybody extending the table. Two estimators separated by less than a factor of two in angular error are not ranked, because the direction of their errors can be worth that much on its own, and the benchmark literature’s practice of reporting medians over large corpora is partly what averages that away.

A scene containing an actual white is the case where one estimator is exactly right and the others are not.

Where the model stops #

Three limits, in increasing order of importance.

The scenes are flat lists of surfaces with no geometry. Real illuminant estimation has shading, interreflection, multiple light sources with different colours in different parts of the frame, and specular highlights that carry the illuminant directly. Grey-edge in particular is implemented here on a one-dimensional strip, which is what a first-order spatial derivative reduces to when there is only one dimension; the real operator is two-dimensional and is applied to an image.

The estimators are the classical ones. Learning-based methods now outperform all of these substantially on benchmark datasets, and they do it by importing a prior over which illuminants are common — which is more outside information, exactly as the argument predicts, rather than a way around the underdetermination.

And nothing here is a model of what the visual system does. Human colour constancy is much better than grey-world and is not obviously any of these algorithms; it uses cues these do not have, including the fact that the observer has been adapting to the illuminant for some seconds and knows a great deal about what surfaces are usually like. The relationship between the engineering problem and the perceptual one is a real research question, and this essay is about the engineering one.

One specific asymmetry is worth naming. The visual system does not have to produce an answer that is right in the sense these algorithms are scored on. It has to produce an answer that is stable — the same shirt reading the same colour as the wearer walks from the window to the lamp — and stability is a much weaker requirement than accuracy. An observer whose constancy were systematically biased by ten degrees in a fixed direction would never find out. A camera with the same bias produces visibly wrong photographs, because they are compared against memory and against each other.

Who found it, and when #

Grey-world is usually credited to Buchsbaum in 1980, though the idea of normalising by the scene average is older and appears in television engineering well before that. The retinex family, from Land and McCann’s work in the 1960s and 1970s, is the ancestor of max-RGB — Land’s original formulation normalises by the maximum along paths through the image, and the “white patch” reduction of it is what survives in practice.

Shades-of-grey, which unifies the two under a Minkowski norm with $p = 1$ giving grey-world and $p \to \infty$ giving max-RGB, is Finlayson and Trezzi, 2004. Grey-edge, which applies the same family to spatial derivatives, is van de Weijer, Gevers and Gijsenij, 2007. The unification is the useful part of that literature: it turned a collection of separate heuristics into one parameterised family, and made it obvious that the parameter is choosing an assumption rather than tuning a method.

Angular error as the standard measure, and the practice of reporting median rather than mean because the distribution has a heavy tail, both settled in the mid-2000s benchmark work.

Where this goes next #

The estimators here guess an illuminant. What to do with the guess is a separate question with its own answer and its own failure modes — four ways to move a white point, of which the naive one is provably wrong and is used anyway.

The other direction is towards the thing being estimated. The illuminant is half the answer sets out why a source’s spectral structure decides which surfaces it flatters, and every estimator above is blind to structure by construction: they all reduce the illuminant to three numbers before they start, which is the same collapse the eye performs and the same information loss.