A gain needs a basis

There are five chromatic adaptation transforms in ordinary use and they disagree with each other. An earlier essay measured the disagreement and left the obvious question standing: if they all implement the same idea — scale three signals by the ratio of the two whites — what exactly are they disagreeing about?

They are disagreeing about which three signals. A diagonal matrix is only diagonal in some particular set of axes, and every one of these transforms is a choice of axes with a von Kries scaling bolted on behind it. XYZ scales X, Y and Z. The von Kries transform uses the Hunt–Pointer–Estevez cone fundamentals. Bradford and CAT02 were fitted to corresponding-colour experiments and have entries no cone response could have. CAT16 is the current recommendation.

None of them was derived from the spectra of the lights they are meant to handle, and it turns out that they could have been.

The claim #

The basis a change of light is diagonal in can be computed rather than fitted, and doing so makes the trade the standards are sitting in visible as a number.

• A change of light is a gain in basis B exactly when B T B⁻¹ is diagonal, and the axes that diagonalise a given T are its left eigenvectors, which are available in closed form.
• The basis daylight picks out leaves nothing at all on the change that defined it, and almost nothing on the others — ΔE00 0.117 and 0.165 across the other two daylight rows, against CAT16’s 0.980 and 0.508. That is a factor of 5.3.
• And it is a trade, not an improvement. On the four discharge lamps in the census the same basis averages ΔE00 2.00 against CAT16’s 1.65. It is worse than every published transform on the lights it was not built for.
• A numerical search finds the same three axes — nine free numbers, minimising a mean colour difference, agreeing with the eigenvectors to within 0.41 degrees on the worst axis, and reaching the same residual to three figures.
• And no published transform is anywhere near it. The closest is Bradford, whose furthest axis is 38 degrees away.

What a basis is, drawn #

Every one of these transforms is three directions in tristimulus space. A colour is projected onto those three directions, each projection is multiplied by the ratio of the whites along that direction, and the result is projected back. The whole content of a transform is the three directions.

Two things about that picture are worth pausing on. The first is how different the published transforms are from one another — the first axis in particular ranges over most of the available directions, which is a reminder that the choice was never settled by anything the transforms have in common — the same freedom that lets six numbers make a space. The second is that the daylight basis has a negative middle coefficient on its first axis, in company with the fitted transforms and not with the physiological one.

Where the closed form comes from #

The change of light is a matrix T, exactly, on a three-dimensional set of surfaces. A basis makes it diagonal exactly when the basis rows are T’s left eigenvectors, so finding the basis is an eigen-decomposition of a 3×3 and nothing more.

The characteristic cubic is solved through its trigonometric form rather than iterated, because the claim being made is that a particular basis falls out of the spectra exactly, and an iterative method stopped early would give a basis that was nearly right and prove nothing. A change of light whose matrix had a complex pair would be one no real basis diagonalises at all; the machinery says so and refuses rather than returning one root.

Applied to D65 against D50, the three axes come out as

(0.9227, −0.3604, −0.1370) · (−0.4442, 0.8953, 0.0350) · (0.1201, −0.1966, 0.9731)

which is a recognisably cone-like set of directions, sharpened: each axis takes rather more of its own primary and rather more negative of its neighbours than the fundamentals do.

The number that makes it a finding #

A basis computed from one change of light will of course be perfect on that change. That is arithmetic, not evidence. The finding is what it does on the changes it was not computed from.

On D65 to daylight at 4000 K it leaves ΔE00 0.117. On D65 to daylight at 10 000 K, 0.165. CAT16 leaves 0.980 and 0.508 on the same two. So the axes that make one daylight change diagonal make almost all of every other daylight change diagonal too — which is a statement about daylight rather than about the fit.

The reason is the commutation table from the census: daylight changes commute with each other to a couple of parts in a thousand. Matrices that commute share eigenvectors. A family of changes that nearly commutes nearly shares a basis, and daylight is such a family.

And the trade #

Held against the discharge lamps, the same basis is worse than everything. Its mean over those four rows is ΔE00 2.00; CAT16 manages 1.65, Bradford 1.71, CAT02 1.64. The daylight basis is not a better transform. It is a transform that has been told which family to be good at.

That is what the published transforms are, seen from here. They were fitted to corresponding-colour data collected under a mixture of light sources — daylight simulators, tungsten, fluorescent booths — so they are compromises between families whose matrices do not commute, and they are optimal for none of the families separately. Over the census as a whole Bradford wins at ΔE00 1.14; the daylight basis comes second at 1.21; CAT16 is 1.31 and plain XYZ scaling is 2.37.

The worst basis is the best one somewhere #

There is one row where the ordering inverts completely, and it is worth its own section because it is the clearest evidence that these numbers are about spectra rather than about quality.

Scaling X, Y and Z directly — the transform this site’s own notes describe as the oldest mistake still shipping, worst overall by a wide margin at ΔE00 2.37 — is the best of all six bases on a triphosphor tube. It leaves 1.61 where CAT16 leaves 2.32, Bradford 2.33, the daylight basis 2.50 and the physiological von Kries transform 2.63. It is also the best of the six on a three-emitter source, at 1.26.

Nothing about that is a virtue of XYZ. The matching functions overlap heavily and are broad, and a change of light that is three narrow spikes moves a broad, overlapping set of channels more evenly than it moves a sharpened, decorrelated one. A basis that is bad at separating anything is, for exactly that reason, hard to knock out of alignment.

The practical reading is uncomfortable and probably correct: which adaptation transform is best depends on the lamp, and the ordering reverses inside the range of lamps a person meets in an ordinary week. A colour management system picks one transform and applies it to a photograph taken under a fluorescent tube and to a proof viewed in a daylight booth, and the transform that is right for one is the transform that is wrong for the other. There is no field in an ICC profile for which lamp is in the room.

Two routes, and the one that lied #

The closed form has a second opinion available. A search over the nine entries of a basis, minimising the mean colour difference left after adaptation over three changes of daylight, knows nothing about eigenvectors; the algebra never evaluates a colour difference. If the two agree, the closed form is very probably the right closed form.

They do agree, to 0.19, 0.17 and 0.41 degrees on the three axes, and to three figures on the residual.

The two devices this collection models that do not adapt in a published basis are the sharpest test of the claim, and both are drawn from the same machinery.

Getting there needed a restart, and the failure is worth keeping. From a single simplex, Nelder–Mead returned the same answer to four significant figures at nine hundred, three thousand and nine thousand iterations. That reads exactly like convergence. It was not: the simplex had collapsed onto a subspace and was reporting the best point in it, and that point was twice the closed form’s residual and thirteen degrees away on one axis.

A search that gives the same wrong answer however long it runs cannot be diagnosed by running it longer, and the obvious diagnostic — check whether it has converged — returns yes. The only reason this one was caught is that there was a second route to compare it against. Rebuilding the simplex around the best point and going again is what broke it out.

The honest version of a fitted basis #

A basis fitted to a set of changes will always look better on that set than a published transform does, and that says nothing at all. The test is to fit on half the census and measure on the other half.

Fitted on half, it reaches ΔE00 0.939 in sample and 1.034 out of it. CAT16 manages 1.272 and 1.351 on the same halves; Bradford 1.093 and 1.187. So a fitted basis is genuinely about twelve per cent better than Bradford on light it has not seen — a real but modest advantage, and much smaller than the in-sample number would have suggested.

The ten per cent is not all fitting, and separating the two makes the result stronger rather than weaker.

Both published transforms are also quoted on the two halves, and neither of them was fitted to anything — so whatever they do between the halves is the halves differing in difficulty rather than any transform learning anything. Bradford goes from 1.093 to 1.187, a rise of 8.6 per cent; CAT16 from 1.272 to 1.351, a rise of 6.2. The second half of this census is simply about seven per cent harder than the first, for everybody.

The fitted basis rises by 10.1 per cent. Divide out the seven that every transform pays and what is left attributable to fitting is 2.5 per cent, not ten. Nine free parameters fitted on seven changes of light, and the honest overfitting penalty is a fortieth of the answer.

That matters for how the held-out number should be read. The fitted basis’s error is 14.1 per cent below Bradford’s in sample and 12.9 per cent below it out of sample — a shrinkage of 1.2 points, which is the 2.5 per cent overfitting arriving in the place it should. So the advantage is not a twelve per cent that used to look much larger; it is a twelve per cent that used to look like fourteen. The essay’s own caution was aimed at the right question and priced against the wrong baseline: a held-out number has to be compared with a held-out number for something that did not learn, and both were sitting in the same table.

The general form is worth keeping, because the mistake is easy to make anywhere. An in-sample-to-out-of-sample drop is not a measure of overfitting unless the two samples are equally hard, and the cheapest way to find out whether they are is to put an unfitted competitor through the same split. Here there were two, and they agreed with each other to two percentage points about what the split alone was worth.

The inversion is bigger than a reordering #

The section on XYZ says the ordering reverses on a triphosphor tube, and reversal understates it. Reading each basis’s triphosphor row against its own census mean:

XYZ manages 1.61 against an overall 2.37, so the tube is 32 per cent easier than its average row. CAT16 manages 2.32 against 1.31, so the tube is 77 per cent harder. Bradford is 2.33 against 1.14, more than twice its average, and the daylight basis 2.50 against 1.21, slightly worse still.

So the tube is not merely a row where the ordering happens to flip. It is the one row in the census that is easier than average for exactly one basis and much harder than average for every other, and the one basis is the one nobody defends.

That reframes what a sharpened basis buys. Sharpening decorrelates the axes, which is what makes a smooth change of light nearly diagonal — and decorrelated axes are precisely the ones a narrow spectral feature can knock apart, because a spike landing on one sharpened channel and not its neighbour is a large differential change. Broad overlapping channels see the same spike as a smaller differential because they all catch some of it. The property that makes a basis good at daylight is the property that makes it bad at lines, and the two are the same number with the sign of the argument changed.

Which is a sharper account of the trade than the essay gives elsewhere. It is not that the published transforms are compromises between two families that happen to want different things. It is that the mechanism of being good at one family is the mechanism of being bad at the other, so no basis can be good at both and the compromise is forced rather than chosen.

The split is by alternating position rather than by kind, so both halves contain daylight and discharge. Split by kind, the held-out half would be a different question rather than a held-out sample of the same one.

Who found it, and when #

Von Kries’s 1902 hypothesis is the diagonal. It says nothing about axes because nothing was known about the axes; the cone fundamentals were not measured for another sixty years.

The recognition that the axes are a free choice, and that sharpening them improves the fit, is Finlayson, Drew and Funt’s from the early 1990s, and it was arrived at as an optimisation over data. What the eigen-decomposition adds is that for a single change of light the answer is not an optimisation at all — it is three eigenvectors, available immediately, exact.

The compromise the standards are in has been visible in their own numbers all along. Bradford’s negative entries were remarked on when it was published; the usual account is that a fitted transform need not be physiological. The account here is narrower and sharper: it need not be physiological because it is not describing a receptor, it is describing a family of illuminants.

What was computed, and how #

Each basis is applied to every census row by taking the ratio of the two whites along its three axes, applying that diagonal, and measuring the mean CIEDE2000 left over a hundred and twenty-five surfaces. Nothing is fitted per row.

The eigen-decomposition is exact. The search is Nelder–Mead over nine numbers with each row renormalised before use — the renormalisation is not cosmetic, since B⁻¹ diag(d) B is invariant to scaling any row of B, so three of the nine parameters do nothing at all and a search that does not know that wanders off along them and reports a basis with an entry of 10¹².

The held-out fit uses the same search on seven rows and reports the mean on the other seven.

Where it stops #

The daylight basis is the eigenbasis of one change of daylight. A different daylight pair gives a slightly different basis, and the essay’s claim is that the differences are small — which is the commutation number rather than a separate result. It is not a claim that a canonical daylight basis exists, any more than there is a D65 lamp.

Nothing here says the eye’s adaptation is diagonal in any basis. The census measures what the best diagonal could do, and the appearance models add a degree of adaptation and a nonlinear response precisely because a diagonal is not enough. Reading these numbers as a description of a mechanism would be reading an upper bound as a measurement.

And the comparison across bases is a comparison in CIEDE2000, which is a formula with opinions of its own. A different difference metric would reorder the close rows, though not the wide ones — the factor of five between the daylight basis and CAT16 on daylight survives any metric this site carries.

Where the ladder goes next #

If the axes are a choice, then every device that applies a gain has made one. A camera’s axes are its filter dyes, and those turn out not to be fixed at all. A press has no axes because it has no gain. And colour management’s rule for a change of substrate is a von Kries adaptation in the worst basis in this table, which is a choice made by a standards committee and never described as one.

The other direction is the lights themselves. If daylight commutes with itself and discharge lamps do not, that is a property of spectra worth measuring on its own terms, and which lamp changes are free is that measurement.