The normaliser carries the error too

Five nanometres has been good enough for a century and the usual explanation is that it is fine enough. It is partly that. It is also that the arithmetic everybody uses cancels its own error without being asked to, and the cancellation has a condition on it.

The claim #

A tristimulus value is a ratio and the grid appears in both halves of it, so most of a quadrature error divides out — and the amount that divides out is a measurement rather than a constant.

• On smooth lights the division removes between a third and three quarters of what the sum got wrong.
• The condition is that the two sums be wrong in the same way, which requires the error to be a smooth function of wavelength that both integrals share.
• On a flat sample the cancellation is total and exact, because the two sums are not merely correlated but proportional.
• On a line spectrum it collapses, because the sample’s sum and the white’s sum miss different lines, and that is a large part of why aliasing is so much more expensive than its share of the spectrum suggests.

What is actually divided by what #

A tristimulus value is X = k Σ R S x̄ Δλ and the constant k is 100 / Σ S ȳ Δλ. Two sums, over the same grid, over the same light, with the same observer functions.

Then CIELAB divides again. L*, a* and b* are functions of X/X_n, Y/Y_n and Z/Z_n, and the white point X_n is the same integral with the reflectance set to one — a third sum, again over the same grid.

So a colour is a ratio of quantities that share every choice the tabulation makes. If the grid is coarse, all of them are coarse in the same direction. Nobody would compute a white point on a different grid from the sample, and it is that unexamined convention which does most of the work.

The convention is not universal, and where it is broken the cancellation is lost. Reading a sample at ten nanometres against a white point taken from a published table at one is such a case, and it is common in software that treats an illuminant’s white as a fixed constant rather than as something to be integrated alongside.

The measurement #

For each light: the relative error in Y made by the five-nanometre sum, the colour error with the white computed at a tenth of a nanometre, and the colour error with the white computed on the same coarse grid.

Two things in that table are worth separating before anything is concluded.

The first is the size of the raw error against the size of the colour error. A tungsten lamp’s five-nanometre sum is wrong by less than two parts in a hundred thousand and the colour it produces is wrong by 0.013 ΔE₀₀. Those are the same fact in two currencies: a colour difference is a highly amplified reading of a tristimulus error, because CIELAB takes cube roots of small ratios and then measures distances between the results.

The second is the factor column, which is what the essay is about, and it is smaller than the word “cancellation” suggests.

That figure and the one above it make the essay’s mechanism visible without any algebra. A pale, smooth sample cancels well; a saturated or structured one cancels badly; and the ordering is the same as the ordering of how much of each sample’s spectral variation the light does not share. It is the pairing structure of this whole round appearing inside a division rather than inside a departure.

The cancellation is real and modest #

Between 1.3 and 4.1 is a useful factor and it is not the order of magnitude the word implies. It is worth being exact about why, because the naive expectation is much larger.

The two sums share a grid but they do not share an integrand. One is R S x̄ and the other is S ȳ, and a quadrature error depends on the integrand’s curvature. Where the sample varies rapidly, the numerator’s error and the denominator’s are different sizes, and only the part they have in common divides out.

The part they have in common is the light. A quadrature error that comes from the light’s own structure appears in both sums in proportion and cancels almost completely; one that comes from the sample’s structure, or from the observer’s, appears in only one. So the factor is large when the light is the difficult term and small when the sample is.

That is confirmed by the row that behaves best. The white LED’s factor of 4.1 is the highest in the table, and the white LED is the light whose structure is closest to the step without crossing it — the light for which the shared term dominates. The laser projector’s 1.3 is the lowest, and there the sums are not sharing an error at all; they are both wrong about different things.

There is a second reason the factor is modest and it belongs to the difference formula rather than to the arithmetic. A cancellation that removes ninety per cent of a tristimulus error does not remove ninety per cent of the resulting colour difference, because the map from one to the other is not linear near neutral: ΔE₀₀ has cube roots in it and a chroma weighting underneath, and small residuals in the chromatic coordinates are amplified relative to their size in X, Y and Z.

So the factor column reports the cancellation as a reader of colour differences experiences it, which is the useful thing to publish and is not the cleanest statement of the mechanism. In tristimulus terms the division removes considerably more than the table suggests; in the unit anybody writes a tolerance in, it removes what the table says.

The exact case, and what it implies #

There is one sample for which the cancellation is total, and it makes the mechanism unmistakable.

If R(λ) = ρ for all wavelengths then every term of the numerator is ρ times the corresponding term of the denominator, so the ratio is ρ exactly, whatever the grid does. That is the identity a flat reflectance has and it holds even when the quadrature is catastrophically wrong — the laser projector reduced to one line still gives a perfectly neutral grey card at the right lightness.

The implication for anybody debugging a pipeline is direct. A colour error that survives on a neutral is not a quadrature error. It is a white-point error, an adaptation error or an arithmetic error, and the grid can be eliminated from the diagnosis in one reading. That is a great deal to get from a single sample.

The implication in the other direction is less comfortable. A calibration performed on neutrals cannot detect a tabulation fault, and neutrals are what instruments are calibrated with.

Why this is the same argument as dividing by the paper #

The structure here has appeared in this collection before, in an applied setting, and recognising it as the same thing is useful.

Dividing by the paper is what makes a printed measurement mean something: a density or a reflectance factor quoted against the substrate removes everything the substrate and the light have in common, so a sheet’s yellowness stops contaminating an ink’s colour. The mechanism is identical — a ratio whose two halves share a nuisance term — and so is the condition, which is that the nuisance must genuinely be shared.

Where the applied version fails is instructive for the numerical one. A brightened paper divided by itself does not remove the brightener, because the brightener’s emission depends on the ultraviolet and the ink on top of it changes how much ultraviolet gets through. The nuisance is not shared once the sample modifies it, and the ratio stops helping exactly there.

The spectral version has the same failure mode with the same shape. A grid error is shared while the sample does not interact with the thing the grid is getting wrong. A sample with structure at the same wavelengths as the light’s lines breaks the sharing, and the cancellation goes with it.

The range’s figures are worth reading in this essay’s terms because the range is the tabulation decision where the cancellation does most work. Twenty-one per cent of a thermal source’s power lies outside the band and about three per cent of its visual product, and the colour moves by half a unit. Each of those three steps is a division removing a shared term, and the total reduction from the first number to the last is a factor of about forty.

What happens when the two grids differ #

The convention that saves everybody is that one grid is used throughout, and it is worth measuring what breaking it costs, because software breaks it routinely.

Computing a sample at five nanometres against a white point taken finely is the third column of the table above, and it costs between 1.3 and 4.1 times the correct arithmetic. That is the honest size of a mistake that looks like an improvement: taking the white point from a more accurate source is more accurate about the white point and less accurate about the colour.

The general statement is that a ratio should be computed with both halves at the same fidelity, and that improving one half alone can make the answer worse. It is counter-intuitive enough that it deserves a name in any pipeline that mixes sources, and it is exactly what happens when an illuminant’s tabulated white point is used alongside a measured spectrum. The same trap has a printed cousin: a tolerance quoted against a declared substrate rather than the measured one is the same mismatch of fidelities, with a sheet of paper in place of a grid.

A third division exists and it does not behave like the other two.

CIELAB divides X, Y and Z by the white point’s own three values, which cancels a grid error again — but this division is per channel, and a quadrature error is not the same in the three channels. x̄ has two peaks and ȳ has one; z̄ is confined to the short wavelengths where the light’s structure is often strongest. So the three ratios inherit three different residuals, and what survives the division is a chromatic error rather than a lightness one.

The measurements bear that out. Across the surface family under a fluorescent tube, the five-nanometre grid moves L* by a median of 0.11 and moves the chromatic coordinates by enough to produce a median 0.83 ΔE₀₀. The lightness is nearly protected and the hue is not, which is the opposite of what a reader would guess from the fact that ȳ is the function the normaliser is built from.

That has a practical consequence for tolerance work. A tabulation fault presents as a hue shift, not as a lightness shift, and a tolerance written mostly around lightness — which many are, because lightness is what a press drifts in — will not catch it.

What was computed, and how #

The raw error in Y is the unnormalised sum over the site’s range at five nanometres against the same sum at a tenth, as a relative difference. Both are over the same range, so no truncation is in the number.

The uncancelled colour is computed by taking the coarse sample sum and the fine white together, which is a pipeline nobody would build deliberately and everybody builds by accident. The cancelled colour is the ordinary calculation. The factor is their ratio.

The assertion this family carries is that the ordinary arithmetic is never worse than the mixed arithmetic on any light whose error is above the floating-point floor. Two of the six lights are excluded from that test because both of their numbers are zero, and a ratio of two zeros is not a measurement — which is a rule this collection has had to state before — most sharply when three of eight conditions turned out to be identities and five were limits.

There is one more place the same division appears and it is the one with the largest stakes. An adaptation transform maps a colour from one white to another, and every published transform is a ratio: cone responses under the source white divided by cone responses under the destination white. If both whites are integrated on the same grid, a tabulation error largely divides out of the transform as well, and the basis the ratio is taken in becomes the only thing left that a grid could disturb.

That is a comforting result and it comes with the usual condition. It holds when both whites are computed and fails when one of them is a constant from a standard, which is how nearly every implementation is written — the destination white is D65’s three published numbers and the source white is integrated from a measurement. A pipeline in that shape has no cancellation in its adaptation step at all, and the error it carries is the full quadrature error rather than the reduced one.

Where the model stops #

The cancellation measured here is between two sums over the same light. It says nothing about a comparison between two different lights, which is what an adaptation calculation is, and there the two white points are different objects and no cancellation is available at all.

The factors are measured on six constructed lights and forty-two constructed surfaces. They are not universal constants and should not be quoted as one; what is general is the mechanism and the condition, and the numbers are what those produce on this set.

And the whole argument assumes the white point is computed rather than declared. A pipeline that hard-codes D65’s tristimulus values as three constants has no cancellation whatever, and a great deal of software does exactly that.

The generalisation #

The habit is about where to look for an error that has already gone away.

A quantity computed as a ratio of two similar integrals is far more accurate than either integral, and the reason is not luck. It is that the two share a nuisance parameter, and the ratio is a projection that removes it. Recognising the pattern tells a reader where accuracy can be spent and where it cannot: refining the numerator alone is worse than useless, and refining both together buys much less than the individual error suggests.

The failure mode is the reverse and it is common. Somebody measures the error in one half of a ratio, finds it alarming, and refines that half. The result is a pipeline that is more accurate in its parts and less accurate in its answer, with the improvement visible in every intermediate diagnostic. Accuracy is a property of the quantity that leaves the calculation, not of the ones inside it.

Who found it, and when #

The cancellation is implicit in every colorimetric standard, which specify that the illuminant, the observer and the sample be tabulated at the same interval and summed together. The ASTM’s practice for computing tristimulus values goes further and publishes weighting factors that fold the illuminant and the observer into a single table per interval, which makes the shared grid structural rather than conventional.

What the standards do not do is quantify the cancellation, and there is a reason: doing so requires a reference finer than the tables, and until the functions were available analytically there was nothing to compare against. That is the same obstacle this round had to build its way around, and the way around it is the next essay.

Where the ladder goes next #

Every number in this section rests on a reference, and a reference for a tabulation cannot be another tabulation. What was built instead, what it cost, and what it means for a collection to check its own arithmetic against a model of itself.