Which changes of light pay for it

A basis fitted to a census of illumination changes beats one built from dichromat confusion points by 0.68 ΔE00 on average. That sentence is true, and it is a summary of fourteen numbers that do not resemble each other: two of them run the other way, and three of the remaining twelve supply half of the whole advantage.

The claim #

The fitted basis’s advantage is concentrated and its disadvantage is systematic. Against the receptor construction it wins on twelve rows of fourteen, but half of its whole advantage comes from three of them — a tungsten lamp and two coloured walls — and the two rows it loses are both cases where the spectra are not smooth. Set beside CAT16 as well, it is the lowest of the three on only eight.

• Tungsten: 2.28 for the receptors, 0.60 for the optimum — a gap of 1.68, the largest in the census.
• Two bounces off a green wall: 3.75 against 2.21.
• A three-emitter source: 0.78 against 1.38 — the receptors win, and by nearly as much as they lose elsewhere.
• Daylight to a blackbody at the same temperature: 0.19 against 0.31, both tiny.
• Half the total gap comes from three rows. Tungsten, the two-bounce green wall and the red wall contribute 4.50 of the 9.51 units summed across the census.
• So the mean is a weighted opinion about which lights matter, and the weights are equal because somebody chose equal.

What a census is, and what it is not #

The census is fourteen changes of context, each a pair of them, and it is a construction rather than a survey. Two daylights, a daylight against a blackbody at the same temperature, tungsten, three discharge sources, a three-emitter source, three coloured walls, and two filters inside the eye. Each row is scored as the mean CIEDE2000 an adapted observer is left with over a hundred and twenty-five constructed surfaces.

There is no claim anywhere that these fourteen are how often anybody meets each. A person living in a northern city meets daylight and a warm-white LED almost exclusively; a printer’s viewing booth is a fixed D50; a film set is tungsten by choice. The census is a list of the changes this collection can model exactly, and averaging over it with equal weights is a decision that has never been examined here.

Examining it is what this essay does, and the finding is that the decision is load-bearing.

Where the fitted basis wins #

Twelve rows of fourteen, but the win is not spread over them: three rows supply half the total and five supply two thirds. All of the large ones have a common structure — the change is large and it is smooth.

Tungsten is the biggest. Going from D65 to a tungsten lamp is a change of about 4,000 kelvin in correlated colour temperature and a smooth reweighting of the whole spectrum towards the long wavelengths. The receptor basis leaves 2.28 and the optimum leaves 0.60, which is a factor of 3.8.

The three wall rows are the same shape at smaller amplitude, and a wall applied twice is the wall applied once, squared — a broad absorption band whose depth doubles in the exponent. The receptors leave 3.75 on the two-bounce green wall against the optimum’s 2.21.

What these four share is that the illuminant ratio — the spectrum of the second light divided by the first — is a smooth, monotone or single-humped function of wavelength. That is precisely the case a set of narrow, well-separated channels handles well: each channel sees one region of the spectrum, the ratio is nearly constant across that region, and a per-channel gain reproduces it.

Where the receptors win #

Two rows, and both of them for the same reason.

A three-emitter source — three narrow bands standing in for a laser projector or a quantum-dot backlight — costs the receptors 0.78 and the optimum 1.38, a deficit of 0.60 and the second largest single entry in the census by absolute size. A blackbody against a daylight at the same temperature costs 0.19 and 0.31; both are small, and the ratio is still more than one and a half.

A third row is worth adding even though the optimum technically wins it. On the triphosphor tube the receptors leave 2.57 and the optimum 2.42, so the entire advantage of nine numbers fitted to the census is 0.15 units — and CAT16, which was not fitted to this census at all, does better than both at 2.32.

The mechanism is the sharpening argument run backwards. A narrow emission line falls inside all three cone channels at once, because they overlap heavily, and multiplies all three by related amounts — which is close to a scaling, and a scaling is exactly what a diagonal handles. Sharpened channels each see one line or none, so three lines move three channels independently and unequally, and there is no diagonal that reproduces it.

So the choice between bases is, in part, a bet on the spectral smoothness of the lights people will live under. Every fitted transform in use makes that bet, and it was placed when the alternatives were daylight, tungsten and fluorescent tubes.

What the bet is worth now #

The lighting stock has changed since Bradford’s entries were published in 1993, and it has changed in the direction that makes the bet worse.

A phosphor-converted white LED is smoother than a triphosphor tube and less smooth than daylight, and it costs the receptors 2.51 and the optimum 1.76 — a win for the optimum, but a much smaller one than tungsten’s. The three-emitter row, which stands in for the narrowband sources now appearing in displays and in high-efficacy lighting, is a loss.

None of this makes a fitted transform a bad choice. It makes the size of its advantage a function of a lighting stock, and the honest form of the comparison is not one number but a picture of which rows it is winning.

The two rows nobody would put in a lighting census #

Two of the fourteen are not lamps at all: the fovea against ten degrees out, and a lens at twenty against a lens at seventy. Both are filters inside the observer, and both belong here for a reason that is easy to miss.

An adaptation model does not know where a change of spectrum came from. If the light arriving at the receptors has been reweighted, the machinery is the same whether a lamp changed, a wall reflected, or the macular pigment thickened — and a lens that has yellowed with age is a spectral filter permanently in front of one observer.

The two rows behave differently from each other and from every lamp. The macular comparison costs the receptors 1.35 and CAT16 0.37; the ageing lens costs the receptors 0.77 and the optimum 0.12. Both are among the easiest rows for the fitted bases and neither is easy for the receptor construction.

That is worth noticing because it is the one place where the ordering has a physiological reading available. An observer’s own filters change slowly and are in front of the receptors continuously, so whatever a lifetime of adaptation does about them, it is not the same problem as a lamp changing in a second. The census treats the two identically, and nothing in the arithmetic could tell them apart.

What an unequal weighting would do #

The question the equal weighting invites is what a realistic one would give, and the answer can be bounded without any survey.

If the census were weighted entirely towards the four rows the fitted basis wins by most, its advantage would grow from 0.68 to about 1.4. If it were weighted entirely towards the two it loses, the advantage would become a deficit of 0.36. So the sign of the comparison is decided by the weighting, and the magnitude varies by a factor of several.

That is a strong statement and it should be read carefully. It does not say the fitted transforms are wrong; on any weighting resembling the lights people actually live under — mostly daylight, some tungsten, some phosphor-converted LED, and rooms with coloured walls in them — the fitted basis wins comfortably. What it says is that the comparison is not a fact about vision, it is a fact about a lighting stock, and it would have to be redone if the stock changed enough.

The stock is changing. Narrowband sources are arriving in displays, in horticultural lighting and in high-efficacy white sources, and the census’s three-emitter row is where the receptors already win.

What was computed, and how #

Every row’s change is exact before any adaptation is applied. A surface’s tristimulus values under light a and light b are related by R(b) R(a)⁻¹, where R is the response of the 1931 observer to a three-dimensional family of constructed reflectances, and the build asserts that this relation is a matrix at a tolerance of 10⁻⁹ on every row.

The adapted observer’s answer is a diagonal, read as the ratio of the two whites in the candidate basis. It is not fitted to anything, per row: the same fixed basis is used for all fourteen, which is the point — an observer has one mechanism, not one per lamp.

The residual is the mean CIEDE2000 between where each surface actually sits under the second light and where the gain puts it, computed in CIELAB against the second light’s own white. Two hundred and forty surfaces per row.

The three bases in the figure are the construction from the confusion points, CAT16, and the basis found by minimising the mean over the census. The last of those has seen every row, which is why the comparison is reported per row: a fit’s in-sample advantage is exactly the kind of number an average hides.

The row that is nearly free for everybody #

One row is easy for every basis in the table and is worth naming because it is the control the census needs.

Going from D65 to a blackbody at 6504 K is a change of spectrum with almost no change of chromaticity: the two lights are at the same correlated colour temperature by construction and differ in the fine structure of the spectrum rather than in its slope. Every basis leaves under a third of a unit, and the receptor construction leaves the least at 0.19.

That row exists to show that the census is not simply ranking changes by how large they are. It is a real change of spectrum — a daylight and a blackbody at the same temperature are not the same light and their difference is measurable — and adaptation handles it easily in every basis because the white barely moves, so the diagonal is barely doing anything and there is little for it to get wrong.

The lesson for reading the rest of the census is that a residual is a statement about what the gain failed to do, not about how different two lights are. Which changes of light are free is the question that essay asked directly, and this row is its clearest positive case.

The held-out version #

The obvious objection is that the optimum was fitted to these fourteen rows, so of course it wins on them. This collection has the machinery to answer it, and the answer is reassuring rather than dramatic.

Fitting on half the rows and testing on the other half still beats the published transforms out of sample. So the advantage is not an artefact of fitting, and the essay’s point is not that the optimum is illusory. It is that the advantage is unevenly distributed, and that the distribution is legible: smooth changes yes, narrowband changes no.

What the shape of the distribution says #

The gaps, sorted, are 1.68, 1.54, 1.28, 0.93, 0.93, 0.83, 0.75, 0.73, 0.65, 0.40, 0.36, 0.15, −0.12, −0.60. That is not a distribution with a centre; it is a long right tail, two negative entries and a cluster of small positives in between.

A mean of a distribution shaped like that is a summary of the tail, and the tail here has a name: large, smooth reweightings of the spectrum. Tungsten and coloured walls are the census’s biggest smooth changes, and they are what a sharpened basis is for.

The consequence for reading a published improvement is direct. When an adaptation transform is reported as reducing mean error by some percentage over a corresponding-colour data set, the sentence is almost certainly about the handful of largest changes in that set, and the transform’s behaviour on the small and awkward ones is not in the number. The two negative entries here are not visible in 0.68 at all.

Two rankings, and only one of them is about smoothness #

Those gaps are ordered by size, and size is not the ordering the mechanism produces.

Ranking the fourteen rows by their gap and, separately, by how far the white point moves between the two lights gives a Spearman coefficient of 0.908. Ranking them by the gap and by how far the illuminant ratio departs from a smooth curve — the residual of a quadratic fitted to the logarithm of the ratio — gives −0.055, which is nothing whatever. So how much a row contributes to that 0.68 is decided by amplitude, and the four largest contributors are simply the four largest movements of the white: the two-bounce green wall at 0.197 in chromaticity, tungsten at 0.156, the red wall at 0.107 and the single green wall at 0.105.

Rank the same rows by the ratio instead — how many times better the fitted basis is, rather than by how many units — and the two correlations change places. Against non-smoothness the coefficient is −0.657, against the unexplained fraction of the log ratio −0.512, and against the movement of the white only 0.402.

Both halves of the account above are therefore true, and they are about different statistics. Smoothness decides how much better a sharpened basis is. Amplitude decides how much that is worth in units. A mean of differences answers the second question while the mechanism lives in the first, which is how a row can be the best row in the census and contribute almost nothing to the headline.

One row is exactly that. The ageing lens is the fitted basis’s largest ratio anywhere in the census — 0.768 against 0.125, a factor of 6.1 — and it contributes 0.64 units to a total of 9.47, which is ninth of fourteen. It is a smooth filter of modest amplitude, and that is the fitted basis’s best case in every sense except the one being summed.

How concentrated is “concentrated” #

Half the advantage arriving from three rows of fourteen sounds like a distribution with almost nothing else in it, and the arithmetic is more moderate than that.

Across the twelve rows the fitted basis wins, the effective number of contributors — the square of the sum of the gaps, divided by the sum of their squares — is 9.43. Twelve equal contributors would give twelve. So the distribution behaves like nine and a half rows pulling the same way rather than like three.

Both readings are right and they answer the two questions a reader would actually ask. Where does the advantage come from? Disproportionately from the tail: three rows of fourteen supply 47 per cent of it where an even split would supply 21. Would the advantage survive losing the tail? Comfortably. Strike the three largest rows and the remaining eleven still average 0.45 in the fitted basis’s favour, because nine of them are still positive.

That is a different conclusion from the one the tail alone suggests, and it is the one that matters for whether the basis is worth having. The concentration is real, it is not extreme, and what the tail decides is the size of the number rather than its sign.

Where the model stops #

Equal weights. Nothing here derives a weighting from how often anybody meets each light. A census weighted by occupancy would be a different object, would need survey data this collection does not have, and would give a different optimum.

Fourteen rows. The set contains no coloured LED, no sodium lamp, no candlelight and no underwater illumination, all of which are real and all of which would be hard. The tables that describe illuminants stop in several different places, and the census inherits those stopping points.

And a surface family is a model of a world. The reflectances are smooth constructions spanning three dimensions, chosen so that a change of light is exactly a matrix. Real surfaces are not three-dimensional, and when they are not, the change of light stops being a matrix at all.

The generalisation #

An average over a constructed set is a statement about the set, and the way to find out how much of one it is, is to look at the terms. Here six of fourteen terms have the opposite sign to the average, which is a great deal of structure to have summarised into a single improvement figure.

The check costs nothing and is almost never done in print. A transform is reported with a mean error over a data set; the terms are in the same computation and are usually not shown. When they are shown, as here, the reported improvement acquires a scope: it applies to smooth changes and reverses on narrow ones.

Where the ladder goes next #

One row is the worst row for nearly every basis in the comparison, and it is not a lamp. What the hardest change in the census actually is says something about where the difficulty in adaptation really lives.