The highlight is the white balance

Two essays on this site have been circling each other for a phase. One established that no illuminant estimator wins on every scene, because each is an assumption about the surfaces. The other established that a highlight on a glossy object is a sample of the lamp, to within about one ΔE, because a dielectric interface reflects without caring what is underneath it.

Nobody put them together. This essay does.

The claim #

The interface component of a glossy surface recovers the illuminant to 0.30 degrees on a scene where grey-world is 14.8 degrees wrong, and it does so from the physics of a dielectric boundary rather than from an assumption about what the scene contains.

The comparison is the point rather than the number. An estimator that only wins on scenes chosen for it has not been tested, so the scene here is chosen to break the competition — and the highlight method is unaffected by the thing that breaks it, which is the argument.

Why it is a different kind of estimator #

The estimators a camera normally uses are assumptions about scenes.

Grey-world assumes the average surface reflectance is neutral, so the average of what arrives is the illuminant. It is right about a great many scenes and wrong about foliage, a green wall, a room lit through a coloured blind, or anything dominated by one hue.

Max-RGB assumes something in frame is white, so the per-channel maximum is the illuminant. It is right when the premise holds — measured here at 0.00 degrees on a scene containing an actual white — and wrong when nothing white is present, which is common indoors.

Grey-edge assumes the average of the derivatives is neutral, which is a weaker assumption and fails differently.

Each is a statement about the world that happens to be usable, and a camera choosing one has chosen which photographs to be wrong about.

The dichromatic estimator is not of that kind. It rests on the reflection model: light leaving a glossy dielectric is the sum of a body component — the illuminant multiplied by the reflectance, which is the product this whole site is built on — and an interface component, which is the illuminant multiplied by a Fresnel term that is very nearly flat across the visible band.

$$L(\lambda) = \rho(\lambda)\,E(\lambda) + k\,R_F(\lambda)\,E(\lambda)$$

Since $R_F$ is nearly constant, the second term is nearly $E$ itself. Subtracting a shaded reading of a surface from a highlight reading of the same surface isolates the illuminant without ever knowing $\rho$.

What it recovers #

Measured on a scene of five glossy green surfaces under illuminant A: the recovered spectrum is 0.30 degrees from the true lamp in XYZ angular error.

Grey-world on the same five surfaces is 14.8 degrees. The scene has no neutral average — every surface is a variation on green — so grey-world’s premise is simply false, and it estimates a greenish illuminant and corrects the scene towards magenta.

The ratio is about fifty, and the assertion in the library requires the highlight method to beat grey-world by at least a factor of five on this scene, which is a claim that would fail if the implementation drifted.

Neither the scene nor the lamp is held fixed by anything, and moving each in turn separates what the method depends on from what it does not.

Two more scenes under a warm lamp say that the method’s advantage is not an artefact of the daylight the first three were lit by.

Its premise, and the fact that it refuses #

Every estimator has a premise. This one’s is that something in the scene is glossy, and the interesting property is that it can tell when the premise fails.

Grey-world cannot. Handed a scene that is entirely one hue, it returns an estimate — a confident, wrong one — because the average of a set of numbers always exists. There is nothing in the arithmetic that says this scene does not satisfy my assumption.

The dichromatic estimator has a natural failure signal: if there is no interface component, the difference between the lit and shaded readings is zero, and there is nothing to normalise. The implementation here throws rather than returning something, with a message naming the premise:

the highlight estimator found no interface component — nothing in this scene is glossy, which is the premise it has and cannot check

That refusal is asserted in the library alongside the positive claims, because an assertion that has never rejected anything proves nothing.

An estimator that knows when it does not apply is worth more than a slightly better one that does not, and it is the single largest practical advantage of this approach over the averaging methods.

Why cameras do not use it #

Given all that, the obvious question is why automatic white balance is still dominated by assumptions about scenes.

Highlights are often clipped. A specular reflection is the brightest thing in a frame by a wide margin, and it is routinely at or past the sensor’s ceiling — at which point the measurement is destroyed and the recorded hue has turned. An estimator that needs unclipped highlights needs an exposure chosen for them, which is not the exposure anybody wants.

Finding them is a segmentation problem. The method needs two readings of the same surface, one specular and one not, which means identifying that two regions of an image are the same material. That is easy for a person and hard for an algorithm, and it is where most of the literature’s effort has gone.

Not every scene has a glossy surface, and a method that refuses is a method that has to be backed by another one — so the practical system is a cascade, and cascades are harder to tune than a single estimator.

And the interface component is only nearly the illuminant. The Fresnel reflectance of a dielectric is not perfectly flat — it depends on the refractive index, which disperses — and the previous essay measured the interface component as sitting 1.06 ΔE00 from the lamp while the body component sits 33.7 away. A ratio of thirty, and not zero.

What “correct” white balance would even mean #

A question the whole white-balance literature mostly steps around, and which this site is unusually well placed to ask: supposing the illuminant were recovered exactly, what should be done with it?

The naive answer is to divide it out — scale each channel so the illuminant maps to equal-energy white, and the resulting picture shows reflectances. That is a complete answer to a question nobody asked, because a person looking at the original scene was not seeing reflectances either.

Human chromatic adaptation is not complete. Under a tungsten lamp the scene does not look as it would under daylight; it looks warmer, and the degree of adaptation is a parameter that the CIE’s own transforms carry explicitly. A photograph fully corrected to a neutral white point therefore shows something nobody in the room saw — it is more accurate than the experience it is a record of.

So a camera’s white balance is doing two things at once and only one of them is estimation. It estimates the illuminant, and then it decides how much of it to remove, and the second decision is an appearance judgement rather than a measurement. Manufacturers set it by taste and by market, and it is one of the more reliable ways to tell two brands apart.

Which means an exactly correct estimator would not settle the argument. It would settle the first half and hand the second half, unchanged, to whoever chose the rendering — and that is the shape of nearly everything in this field.

Where the model stops #

A single dielectric interface is modelled, and real surfaces are layered. A car’s clear coat over metallic flake, a varnished painting, wet skin: each has more than one interface and a subsurface term that is not the simple body component. The dichromatic model is a good description of plastics, paints and most dielectrics and a poor one of metals, which have no body component at all and whose specular reflection is strongly coloured.

Metals are the clean failure. Gold and copper reflect selectively, so their “highlight” is not the illuminant, and an estimator applied to a brass instrument returns the brass. That is not a subtle failure mode; it is a whole class of surface on which the method is confidently wrong, and it cannot detect the case.

And the geometry is stated rather than solved. The figures fix an incidence angle and a fraction $k$ of the interface component caught by the viewing direction; a real scene has a distribution of both, and recovering them is the segmentation problem again.

The eye does something else again #

It is tempting to treat the highlight method as a model of what human colour constancy does, and it is not, though the ingredients overlap.

Constancy is the default for a person: a sheet of paper looks white under daylight and under tungsten without any deliberate act, and the mechanism is a combination of retinal adaptation, spatial comparison and a good deal of inference from the scene’s structure. It is fast, it is partial, and it is not a division by an estimated illuminant.

There is real evidence that specular highlights are among the cues the visual system uses — observers’ constancy improves when glossy objects are present, and it improves further when the highlights are geometrically consistent with the rest of the scene. So the physics this essay depends on is available to a person and is apparently used.

But a person does not compute an illuminant and divide. The clearest evidence is that adaptation is incomplete and asymmetric: the residual cast is real, it differs between illuminants, and it is a parameter in every appearance model rather than an error to be removed. A camera dividing exactly would be doing something a person does not do.

So the highlight estimator is a good physical method and a poor perceptual model, and the gap between those two is the boundary this whole collection runs along: matching is not appearance, and an estimator recovers a physical quantity rather than an experience.

The generalisation #

The transferable claim is about the difference between an estimator justified by a mechanism and one justified by a regularity, and it is worth more than the specific method.

A regularity-based estimator says that in the data behind it, this quantity is usually about that. It works where the regularity holds, fails where it does not, and — crucially — cannot tell the two cases apart from the inside, because the computation is the same either way.

A mechanism-based estimator says that this quantity is that, because of how the world is put together. It works where the mechanism operates, fails where it does not, and can usually detect its own domain, because the mechanism has observable preconditions.

The second is worth substantially more than its accuracy advantage suggests, and this essay’s numbers understate the case. Fifty times better on one scene is a headline; being able to refuse is the durable property, because a system that knows when it is out of its domain can be composed with others and one that does not cannot.

The pattern recurs throughout this collection. Estimating a gamut volume two independent ways and requiring them to agree is a mechanism check. Requiring a form-factor matrix to close to one is a mechanism check. In each case the value is not the accuracy but the fact that a violation is detectable — and detectability is a property of mechanisms rather than of fits.

A scene with fewer glossy surfaces is the case the method should struggle with, and it is the comparison that says how much of the advantage is the Fresnel reflection.

Four estimators, and what a scene decides #

It is worth laying the whole set out together, because the useful summary of this and the essay below it is not a ranking but a table of premises.

Grey-world needs the average reflectance to be neutral. It is excellent on a varied scene, hopeless on a monochromatic one, and cannot detect which it has. Measured here: 2.5 degrees on a balanced scene, 43.1 on an all-green one.

Max-RGB needs something white in frame. It is exact when the premise holds — 0.00 degrees on a scene containing a perfect white, which is the estimator recovering its own assumption — and drifts badly when the brightest thing is coloured.

Grey-edge needs the average gradient to be neutral, which holds on textured scenes and fails on scenes dominated by one boundary.

The dichromatic method needs something glossy and dielectric. It is unaffected by every one of the failures above, and has a failure of its own that none of them shares: metals.

No estimator wins on every scene, and that is a theorem-shaped fact rather than an engineering gap. Estimating an illuminant from an image is underdetermined: the same three numbers per pixel are consistent with a bright surface under a dim light and a dim surface under a bright one, and with a yellow surface under white light and a white surface under yellow light. Every estimator resolves the ambiguity by adding an assumption, and the assumptions are not all true at once.

A warm scene under daylight is the combination that defeats both assumption-based estimators at once.

The 0.30 degrees is the Fresnel tilt and nothing else #

The estimator’s residual and the previous essay’s measurement of the interface component are the same quantity in two units, and putting them in one unit says that the implementation contributes nothing.

An angular error of one degree costs between 1.5 and 3.2 ΔE₀₀ depending on which direction it points, so 0.30 degrees is 0.44 to 0.97 ΔE₀₀. The interface component’s own distance from the lamp is 1.51 at normal incidence, 1.06 at 55 degrees and 0.20 at 80 — which converts to 0.47 to 1.03 degrees, 0.33 to 0.73, and 0.06 to 0.14.

The measured 0.30 sits inside the band a moderately glancing highlight predicts, and there is no room left in it for anything else. So the estimator is not approximately recovering the interface component; it is recovering it exactly, and the entire residual is the dispersion of the refractive index — a property of the material, not of the algorithm.

That is worth knowing because it says where the method’s ceiling is and how to reach it. A steeper viewing angle would lower the floor: at eighty degrees the interface component sits 0.20 ΔE₀₀ from the lamp, which would put the estimator at about a tenth of a degree. An estimator that preferred grazing highlights over near-normal ones would be three times more accurate for free, and grazing highlights are also the brightest — so the two things it wants improve together.

And no amount of algorithmic work goes below that. The dichromatic model’s premise is that the Fresnel term is flat, and it is flat to about five per cent across the band. Every implementation of this method inherits that, whatever it does with segmentation, clipping or geometry.

The favourable case for the assumption-based estimators is a scene of many colours under a warm lamp, and it is the one worth putting the method against.

Two premises, two shapes of best case #

The table of four estimators has a structural feature its rows do not state: max-RGB’s best case is exactly zero and the dichromatic method’s is not, and the difference is in the kind of premise each carries.

Max-RGB’s premise — something in frame is white — is either satisfied or not. When it is satisfied the problem stops being estimation: a perfect white reflector returns the illuminant unchanged, and reading it off is a measurement. That is why the figure is 0.00 and not 0.02.

The dichromatic premise — the Fresnel term is flat — is never exactly satisfied, because a refractive index disperses. So even on a scene built entirely for it, the method has a floor, and the floor is the 0.30 degrees above.

Exactly satisfiable premises give exact answers; nearly true ones give floors, and that is the more useful way to sort estimators than by their measured errors. It also explains why the fifty-fold advantage over grey-world on this scene is not the durable number: it is a ratio between a floor and a failure, and both ends move with the scene.

A green scene under a tungsten lamp is the other corner of the same table, and it is where max-RGB nearly recovers.

The refusal covers one failure and not the other #

An estimator that knows when it does not apply is worth more than a slightly better one that does not is the essay’s strongest claim, and the method’s own two failure modes are not alike.

Absence of gloss is detectable. No interface component means nothing to normalise, and the implementation throws with the premise named. That is the case the refusal covers, and it covers it completely.

A metallic highlight is not. A brass instrument produces a large interface component that is strongly coloured, so the arithmetic runs, returns a confident answer, and the answer is the brass. Nothing in the reading distinguishes a gold highlight from a tungsten lamp on a white plastic, because both are a bright, warm, spectrally sloped interface term.

So the detectability advantage is partial in a specific way: it detects the absence of the premise and not its corruption. That is the ordinary shape of a mechanism check — a form factor’s rows sum to one whether or not the geometry was the intended one, and a two-derivation agreement catches a wrong constant and not a wrong model.

It also gives the cascade its architecture. The method must be gated on material class rather than on its own output, because its own output is well-behaved on exactly the surfaces it is wrong about. A system that ran it and fell back when it threw would still be confidently wrong on every photograph of metal — and metal is precisely the material somebody photographs when colour matters.

Who noticed, and when #

Steven Shafer set out the dichromatic reflection model in 1985, and the paper’s central observation is exactly this essay’s: that the two components separate in colour space, with the body component lying along one direction and the interface component along the illuminant’s, so that the pixels of a single glossy object form a plane whose second axis is the light.

Gershon Klinker, Shafer and Takeo Kanade turned that into an algorithm in 1988, and the geometric picture — a dog-leg or skewed-T shape in colour space, with the specular branch pointing at the illuminant — has been the basis of the specularity-based estimation literature since.

What kept it out of cameras for thirty years is the list above, and what has changed recently is that segmentation stopped being hard. Learned segmentation finds same-material regions reliably enough to make the classical method practical, and several current systems combine a specularity-based estimate with an averaging one and a learned prior. The physics did not need improving; the thing standing in front of it did.

Where the ladder goes next #

Downward, this rung sits on the highlight is the lamp, which built the physics, and on the algorithms that guess the light, which built the scoring apparatus and the competition.

Upward, the remaining stages of the pipeline each have a rung. Correcting colour costs noise is the matrix’s price, a blown highlight turns is what happens when the very measurement this essay depends on is destroyed, and a photograph is not a measurement closes the field.