A partial correction is worth its fraction

A diagonal gain leaves 1.31 units of a change of light behind, and the matrix that would remove the rest exists and is computable. The question a bounded mechanism would ask is not whether it can have the whole correction, but whether a little of it is worth having.

The claim #

The residual falls almost exactly linearly in how much of the correction is applied, so there is no cheap first tenth.

• The correction exists per row. For each change of light there is a matrix C such that applying the diagonal gain and then C gives the exact answer, with zero residual.
• Interpolating is well-defined. I + α(C − I) runs from the published model at α = 0 to the exact answer at α = 1.
• The curve is a straight line. Over all fourteen rows and nine settings, the largest departure from removed = α is 2.2 percentage points.
• A tenth of the correction removes a tenth of the residual — between 9.5 and 10.9 per cent, against 10 exactly.
• And it had no reason to. A colour difference is a strongly non-linear function of a tristimulus value and the correction is a matrix rather than a scalar.

What the correction is #

Take one census row: a light a, a light b, and the family of surfaces. A change of light acting on this family is exactly a 3×3, call it T, with a residual of 7 × 10⁻¹⁴ ΔE2000 — a theorem rather than a fit.

An adapted observer applies a different matrix: G, the diagonal gain in a fixed basis, whose three numbers are the ratio of the two whites. What is left over is the census’s published residual.

Define C = T⁻¹ G⁻¹. Then C · G = T⁻¹ exactly, so applying the gain and then C gives the exact answer and leaves nothing. C is the correction, it is different for every row, and it is not a model — an observer would have to know which row it was in.

The interpolation is C_α = I + α(C − I), which is I at zero and C at one, and applying C_α · G gives a residual that runs from the published one to zero.

Why linear rather than geometric #

There are two natural ways to apply a fraction of a matrix and they give different curves, so the choice needs stating.

Geometric — C^α, by eigendecomposition — is natural if the correction is thought of as a transformation being applied in fractional amounts, the way a rotation of ninety degrees applied half is a rotation of forty-five.

Linear — I + α(C − I) — is natural if it is thought of as a correction with a gain on it: the full correction computed, then turned down.

This collection uses the linear form because it already interpolates one thing that way and consistency is worth more than either argument. CIECAM16’s degree of adaptation D pulls each channel’s gain towards one by 1 − D, which is linear in exactly this sense, and it is what a partially adapted mechanism is modelled as doing. A collection that interpolated a gain linearly and a correction geometrically would be making an unstated claim about the difference between them.

The two forms agree to first order at α = 0 and diverge slowly, so nothing in the finding below depends on the choice — which was checked rather than assumed.

The curve #

The largest departure from the diagonal anywhere in the table is 2.2 percentage points, on the row for a green wall bounced twice at α = 0.3.

A tenth of the correction removes a tenth of the residual. There is no cheap first tenth, no knee, no diminishing return and no accelerating one. The curve is a straight line to within the width of a pen.

Why that is a surprise #

Nothing in the construction makes it likely, and it is worth listing what would have had to be true for the expected curve to appear.

A colour difference is not linear in a tristimulus value. CIEDE2000 involves a cube root, three weighting functions, a hue angle and a rotation term. A perturbation applied to one argument does not produce a proportional change in the output except in a limit nobody is in here — the residuals run from 0.26 to 3.37 units, which is well outside any linear regime the formula has.

A matrix correction is not a scalar. C − I has nine entries, and applying a fraction of it moves the prediction along a nine-dimensional direction. The residual is an average over 125 surfaces of a non-linear function of a point moving along that direction, and averages of non-linear functions of linearly moving points are not linear.

And the residual is a mean over a set with structure. A third of the surfaces contribute nothing at all — they are flat greys on which the gain is exactly right — and the rest contribute very unequally. A correction acting differently on the contributing and non-contributing members would bend the curve.

So the prediction written down before the arithmetic ran was that the curve would be concave: most of the benefit early, because a large error responds to a first-order correction and the remainder is second-order structure. That is the fourth prediction this round got wrong, and it was wrong in the way that closes a possibility rather than opening one.

Why it comes out straight anyway #

The mechanism is worth working out, because it says which of the three objections above is doing the least work.

At α, the predicted tristimulus for surface i is (I + α(C − I)) G x_i, which is p_i + α d_i where p_i is the von Kries prediction and d_i = (C − I) G x_i is the direction to the exact answer. So each surface’s prediction moves in a straight line, at constant speed, from where the gain put it to where it should be.

The residual for surface i is a colour difference between the true colour and that moving point. At α = 1 it is zero, so the moving point ends at the true colour, which means the whole trajectory is a straight line towards the target. A colour difference along a straight line towards its own zero is very nearly proportional to the distance remaining, because a difference formula is a norm and every norm is homogeneous of degree one along a ray.

The non-linearities in CIEDE2000 are non-linearities in where in the space the pair sits, not in how far apart they are along a fixed direction. Moving along a ray towards the target changes the separation and barely changes the location. The formula’s non-linearity is in the wrong coordinate to bend this curve.

The residual 2.2 points is what is left: the location does move a little as the point approaches, so the weighting functions change slightly along the way, and the rows where it shows most are the ones whose corrections have the largest chroma component.

The explanation predicts its own exception #

An explanation is worth what its prediction is worth, and this one makes a sharp one.

If the linearity comes from the residual being a norm evaluated along a ray — homogeneous of degree one, so halving the remaining distance halves the reported difference — then a unit that is not a norm should break it, and should break it in a computable way.

CAM16-UCS is not a norm. Its distance is the Euclidean distance in its uniform coordinates raised to the power 0.63, which is where its whole near-end behaviour comes from. Along the same ray the remaining difference should therefore fall as (1 − α)⁰·⁶³, so the share removed should be 1 − (1 − α)⁰·⁶³, which is well below the fraction applied: seventy per cent of the correction should remove only 53 per cent of the reported difference.

The departure from the diagonal under the appearance unit is up to 17.4 percentage points, against 2.2 under the published one. And the departure from 1 − (1 − α)⁰·⁶³ is 1.3 points, over every row and every setting.

So the straight line is a property of the ruler and not of the correction. Five of the six units are norms in the relevant sense and give a straight line to within a couple of points; the one that is not gives exactly the curve its exponent asks for. A reader who wants a partial correction to be worth more than its fraction can have that, and the way to get it is to change the unit rather than the mechanism — which is not a way to get it at all.

It also sharpens the negative result. Under the appearance unit a partial correction is worth less than its fraction, not more. There is no unit on this menu under which a cheap approximate correction is a good deal.

The argument can be made one step sharper, and the sharper version predicts the size of what is left over.

The ray is straight in tristimulus space, and every difference formula on the menu is evaluated after a non-linear map into some other space — a cube root into CIELAB, a whole appearance model into CAM16-UCS. So the path is not straight where the difference is taken, and the exact-linearity argument does not apply on the nose.

What rescues it is that the displacement is short. The residuals run from 0.26 to 3.37 units, which are small movements in a space a hundred units across, and any smooth map is affine to first order over a short enough interval. An affine map takes a ray to a ray and scales lengths by a constant, so a norm evaluated along the image of the ray is still proportional to what remains of it. The linearity is a first-order result rather than an identity, and the 2.2 percentage points is the second-order term.

Which means the departure should grow with the residual, roughly as its square, since the curvature term is the one being neglected. The essay’s own rows say so without having been asked: the worst departure is on a green wall bounced twice, which is the harshest row in the census at 3.37 units, and the mildest row’s curve is straight to the width of the line. Two ends of a fourteen-fold range in residual, and the departure appears at the end where the ray is longest.

That is a check rather than a new measurement, and it is the kind worth having — an explanation that names which of its own terms was dropped, and then finds the dropped term where it said it would be.

The generalisation #

The finding turns out not to be about chromatic adaptation at all, and saying so is the most useful thing in this essay.

Strip the colour out. There is a wrong answer, a right answer, a straight path between them, and a ruler. A partial move along that path is worth exactly its fraction — always, for reasons that have nothing to do with what is being corrected, provided three things hold: the target is exact, the path is straight, and the ruler is a norm over a short enough interval to be affine.

So the familiar hope — the first half of the fix gets most of the benefit — is not a fact about difficult problems. It is a signal that one of those three has failed, and which one it is carries its own information:

• The target is not exact. If the endpoint still has a residual, the ray does not end at zero, and the curve bends. This is the ordinary case in fitting, where “the correction” is a best fit rather than a solution.
• The path is not straight. A correction applied in stages, or one whose parameters interact, moves along a curve, and a curve towards a target is longer than the ray — so the early part, which is roughly along the ray, does buy more than its share.
• The ruler is not a norm. CAM16-UCS is the worked example and it bends the curve the other way, so a cheap correction is worth less than its fraction rather than more.

None of the three obtains here, which is why the line is straight and why the negative result is as complete as it is. A “diminishing returns” curve is evidence about the shape of a problem, and its absence is evidence too — it says the remaining gap is not structured, has no easily-removed part, and will be paid for at full price or not at all.

What it closes #

Three possibilities are removed by a straight line, and together they are most of what a bounded mechanism might have hoped for.

No cheap approximation. A mechanism able to apply a rough, small correction gets exactly the share it pays for. There is no version of the argument in which a crude second stage buys most of the remaining accuracy. That matters because the residual is not evenly spread over the surfaces — a third of them cost nothing at all — and an uneven residual is exactly the shape a cheap correction usually exploits.

No natural stopping point. A concave curve has a place where the marginal return falls below the marginal cost, and a designer can point at it. A straight line does not; every increment costs the same and buys the same, so the decision is entirely about the cost side — and the cost side is nine numbers the observer cannot obtain.

And no evidence for a two-stage mechanism. If the residual had fallen sharply for small α, that would have been weak evidence that a small post-receptoral correction is worth having and might exist. The straight line is weak evidence the other way: a mechanism that implemented a tenth of the correction would gain a tenth, which is unlikely to be worth a mechanism.

Taken with the finding that no fixed correction survives being held out, and with the theorem that says the exact answer is a matrix at all, the three together are close to a complete negative result about the gap: the remaining 1.31 units are neither available for free nor cheap to approximate. They are information about the particular room, and an observer that had it would not need the model.

Where the model stops #

The correction is computed per row from the exact matrix, so the whole exercise assumes the exact matrix exists — which is a theorem about this family of surfaces and not about surfaces in general. On the clamped realistic family the theorem fails, because clipping makes the family non-linear in its parameters, and a correction that made one row exact on the idealised set would leave something on the realistic one.

The linearity is measured at nine settings between 0 and 1. It says nothing about α above 1 — overcorrecting — which is a different question and would be a strange thing for a mechanism to do.

And the straight line belongs to the ruler rather than to the correction, which the next section establishes by finding the one ruler that breaks it.

Who found it, and when #

The homogeneity argument is elementary and is the reason it is worth writing down: the result is not deep once seen, and it was not seen before it was computed. That is the ordinary condition of a numerical experiment in a field where the formula is complicated enough that nobody reasons about it directly.

The interpolation itself has a near relative in the literature. Partial adaptation — an observer part-way between two states — is modelled in CIECAM by a scalar degree of adaptation applied to the gain, and there is a long history of measuring how far short of complete real adaptation falls. That is a fraction applied to the diagonal; this is a fraction applied to what is left over after the diagonal, which is a quantity no mechanism has been proposed for and which therefore nobody had reason to interpolate.

Where the ladder goes next #

Three of the six models on the ladder require the observer to know something, and the amount each requires has been counted. What has not been said is what kind of claim that count is, and it is a stronger one than a parameter count usually makes: a model whose parameters are quantities the observer cannot obtain is not a worse model of the same thing, it is a model of something else.