The colour is right first

An instrument with n filters determines an n-dimensional projection of the spectrum and is exactly blind to the rest. That is a limitation, and by itself it would be a manageable one — an instrument that could not see a lamp would presumably report something obviously wrong about it.

It does not. It reports the right colour.

The claim #

The colour of a reconstructed spectrum converges far faster than the spectrum itself, because the observer’s three functions are broad and smooth and therefore lie almost entirely inside the row space of even a modest filter bank. So the check a user is most likely to run — does the reconstruction have the right colour — is the check that will pass while the reconstruction is wrong.

• At twelve readings the fluorescent tube’s colour is right to 0.48 ΔE00 while 66 per cent of its spectrum is in the instrument’s null space.
• Daylight is right to 0.52 at twelve readings with 4.6 per cent unseen, so the two lamps look equally successful on the colorimetric check and are not remotely equally measured.
• The mechanism is that the residual is nearly a metameric black. What the filters miss is narrow structure, and narrow structure integrated against three broad functions contributes very little.
• And the cost lands on any second illuminant. A reconstruction whose error is invisible to the observer under one light is not invisible under another, because a different light weights the missing part differently.
• So the colorimetric check is not a weak test of the spectrum. It is a test of something else.

Why the colour converges first #

The observer’s three colour-matching functions are broad, smooth, non-negative bumps a hundred nanometres wide. A filter bank of a dozen elements spans a subspace that contains almost all of each of them — an overlap of well over ninety-nine per cent — so the projection that the instrument determines contains nearly the whole of what the observer weights.

Put the other way round: the part of the lamp the instrument misses is nearly orthogonal to the observer as well. It is not exactly a metameric black, but it is close enough that its tristimulus contribution is small, and it gets closer as the bank grows.

That is not a defect in the observer, and it is not an accident. Both objects are smooth for the same reason — a receptor’s absorption band and a practical optical filter are both broad — and any two smooth kernels on the same interval overlap heavily.

How close the two operators are #

The claim that the observer lies almost inside a filter bank’s span is easy to check and is worth checking, because the whole argument rests on it.

A dozen Gaussian filters spread over the visible range span a twelve-dimensional subspace of the eighty-one-dimensional band space. Each colour-matching function projected onto that subspace loses well under one per cent of its own magnitude — they are smooth, they have no structure narrower than a filter, and they are exactly the kind of object a coarse bank represents well.

So the composition reconstruct, then integrate against the observer is very nearly the same as integrate against the observer directly. The reconstruction step is close to a no-op for colorimetric purposes, which is why it neither helps nor hurts, and why a colorimetric check on it is close to vacuous.

The same argument says what a useful check would have to look like: an operator with structure narrower than the filters. A narrow reflectance is one. A second filter bank offset by half a spacing is another. A different observer is not — the ten-degree functions are just as smooth as the two-degree ones and lie in the same subspace to the same accuracy.

What it costs, and where #

A spectral measurement of a lamp is not usually made in order to know the lamp’s colour. Its colour can be measured with three filters and no reconstruction at all. It is made in order to predict what the lamp will do to surfaces that have not been chosen yet — which is a colour rendering index, a metamerism index, a proofing condition, or a rendering of a scene.

Every one of those multiplies the lamp by a reflectance before integrating, and a reflectance is exactly the thing that can undo the near-orthogonality.

A narrow reflectance sitting on a mercury line is the worst case. The lamp’s true spectrum has a spike there and the reconstruction has a smooth bump; multiply both by a reflectance that is large only in that band, and the two integrals differ by a large factor rather than by half a unit.

A broad reflectance is the best case, and it is the case every check tends to use, because the standard test colours are broad. The test samples a colour rendering index is computed on are moderately saturated Munsell chips with smooth reflectances, chosen in the 1960s to represent ordinary objects — an entirely reasonable criterion which happens to make the index insensitive to exactly the error this essay is about. The index is one observer’s opinion in more ways than one, and this is a second.

The size of the effect is set by the overlap between the missing part of the lamp and the structure of the surface. For a broad surface it is a fraction of a unit; for a surface whose reflectance rises sharply across a mercury line it can be several. The distribution has a long tail and the mean is not the number to plan around, which is the general property of any error whose size depends on a coincidence of positions.

The lamp that is measured worst scores best #

The two lamps are described as looking equally successful on the colorimetric check, and the two pairs of numbers say something sharper than that.

The tube has fourteen times as much unmeasured spectrum and the better colorimetric score. Not an equal one — an eight per cent better one, on a check that is supposed to be at least loosely related to how well the spectrum was recovered.

That inverts the essay’s framing in the useful direction. Equally successful invites the reading that the check is insensitive; the actual ordering says it is anti-correlated on this pair, and an anti-correlated detector is worse than an insensitive one, because a user comparing two instruments or two lamps would be led to prefer the worse measurement.

The size of the failure can be stated as a ratio. If the colour error tracked the unseen share at all — if a lamp with fourteen times the missing spectrum showed fourteen times the colour error — the tube would read 7.5 ΔE00. It reads 0.48. The colorimetric check registers one fifteenth of what a proportional detector would, and that factor of fifteen is the essay’s central claim in one number.

How much of the space twelve filters reach #

Two further numbers fall out of the same pair and they say something about the lamps rather than about the check.

Twelve filters in an eighty-one-band space determine 14.8 per cent of the dimensions. Daylight nonetheless lies 95.4 per cent inside that span, and the tube 34 per cent.

Daylight’s figure is the more surprising of the two and it is a fact this collection establishes elsewhere: natural daylight is very nearly three-dimensional, so almost any dozen smooth filters will contain it. The 4.6 per cent that escapes is not daylight’s own complexity but the mismatch between a Gaussian bank and the daylight basis — twelve filters chosen to span the daylight eigenvectors would leave essentially nothing.

So the two lamps are not two hard cases and one easy one. They are a source that is nearly low-dimensional and one that is not, and the reconstruction problem is trivial for the first whatever the bank looks like. That is worth saying because it bounds how far the essay’s warning reaches: for smooth, low-dimensional sources — which is most of what a spectroradiometer is pointed at outdoors — the colorimetric check and the spectral one really do agree, and the trap is empty.

Why the standard test colours cannot spring the trap #

The essay observes that a useful check needs an operator with structure narrower than the filters, and that the colour-rendering test samples are too broad. Both halves have a scale.

Twelve filters across four hundred nanometres sit about 33 nanometres apart, so anything smoother than about that lies in their span and cannot probe what they missed. A mercury line is a few nanometres wide — about seven times narrower than the spacing, which is why the tube’s spectrum is two thirds invisible. And the Munsell chips a colour-rendering index is computed on vary on a scale of roughly a hundred nanometres, which is three times broader than the filters.

So the test samples are on the wrong side of the filters by a factor of three, and the lamp’s own structure is on the right side by a factor of seven. A rendering index computed from a coarsely measured narrowband lamp is therefore doubly insulated from the error: the reconstruction loses the lines, and the samples could not have seen them anyway.

That gives the essay’s practical advice a threshold rather than a direction. A probe reflectance must be narrower than the instrument’s filter spacing to test anything the instrument did not measure, and the spacing is a number every instrument’s own specification carries. A twenty-nanometre band on a mercury line is a check; a Munsell chip is not; and the difference between them is computable before any measurement is made.

The general shape of the trap #

The pattern is a validity check computed with the same operator that lost the information.

A colorimetric check on a spectral reconstruction integrates against three broad functions. The reconstruction was produced by integrating against a dozen broad functions. The two operators are nearly the same operator, so the check is nearly guaranteed to pass — it is asking whether the projection is a good projection.

This has a name in other fields and deserves one here: the check is inside the row space. A residual measured only in directions the measurement already determined is not a residual at all.

This collection has met the same shape twice before and it is worth naming the family. A camera matrix evaluated on the chart it was fitted to reports a residual insensitive to the directions the chart failed to pin. A 3×3 fitted on six near-white sheets reported an excellent residual and was measuring the dimensionality of its test set. In all three the check and the fit share an operator, and in all three the failure is silent.

The honest checks are the ones that leave it. Predict what the reconstructed lamp does to a surface with narrow structure. Predict what it does under a second observer. Compare against a measurement made with a different bandpass. Each of those probes a direction the reconstruction was not fitted in, and each can fail while the colorimetric check passes.

What was computed, and how #

The reconstruction is the minimum-norm solution — the projection onto the row space of the filters — so no basis and no prior enters. The colour comparison converts both the true spectrum and the projection to tristimulus values, normalises each to unit luminance, and takes ΔE00 in CIELAB against a D65 white.

Normalising to unit luminance is a decision and it is the right one here. A reconstruction that is systematically dim would otherwise be penalised for a brightness error, and brightness is the one thing three filters get right for free. What is being measured is chromatic agreement.

The unseen share is the same quantity the previous essay reports: the norm of the residual over the norm of the spectrum, unweighted across the eighty-one bands. Its being unweighted is exactly what makes the comparison in this essay possible — one number weights every band equally and the other weights them by the observer, and the whole finding is the gap between them.

The sweep is not perfectly monotone in colour error, and the reason is worth stating rather than smoothing: the residual’s tristimulus contribution can change sign as filters are added, so a bank that happens to leave a residual the observer weights positively can be beaten by a smaller bank whose residual cancels. That is a real property of the projection and not noise, since there is no noise anywhere in the computation.

Two more reconstructions say that the ordering — colour first, spectrum later — is a property of the instrument rather than of the lamp it is pointed at.

Where the model stops #

Nothing here says a reconstruction is useless. For a smooth source it is excellent in both senses, and for any source it is exactly right about the directions it determines. What is being warned against is a specific inference: from the colour is right to the spectrum is right.

And nothing here is about a fitted reconstruction. Real spectral estimation uses a basis — principal components of measured daylights, or of Munsell reflectances — and such a fit can recover structure the projection cannot, when the source belongs to the family the basis was built from. When it does not, the fit produces a plausible spectrum with confident detail in it, which is worse than a projection’s honest smoothness, and the colorimetric check will pass for that too.

The lamps are constructed. The tube’s lines are at mercury’s wavelengths with stated widths, and the ratio of line power to continuum is a parameter. A tube with weaker lines would have a lower floor and the same shape.

Who found it, and when #

That colorimetric agreement does not imply spectral agreement is the oldest fact in this subject — it is metamerism, stated in 1852 and the reason colour science exists as a discipline separate from spectroscopy. What is less often stated is its consequence for validation: that a colorimetric check on spectral data is testing a projection.

The standards reflect the knowledge without quite stating the reason. Colour rendering indices are required to be computed from spectral data rather than from tristimulus values, and the required bandpass and interval are specified — because the people who wrote them knew that a narrow-band source measured coarsely gives an index that is wrong in a way its own colour will not reveal. The specification is the fix and the argument for it is usually left out, which makes it look like conservatism rather than a consequence.

Four filters is fewer than anybody builds and daylight is the easiest lamp, which makes the pair the most favourable case available.

The one case where it does not bite #

There is a class of measurement for which the trap is harmless, and identifying it is the practical half of the argument.

If the only thing the reconstruction will ever be used for is a colorimetric prediction under the same light it was measured under, then the projection is exactly as good as the truth, because the observer’s contribution from the null space is what the colour agreement already measured. Nothing is being extrapolated and the reconstruction’s spectral wrongness is invisible for the same reason it is harmless.

The moment a second light, a second observer or a surface with structure enters, the case changes, and it changes discontinuously rather than gradually — the missing part is weighted by whatever is doing the weighting, and the new weighting has no reason to resemble the old one. That is why a metamerism index computed from a coarsely measured lamp is untrustworthy in a way the lamp’s own colour coordinate is not.

The rule that follows is short: a reconstruction may be used at the operating point it was measured at and nowhere else without a statement of what it did not measure.

The generalisation #

The rule is short and applies well beyond instruments: do not validate a reconstruction with the operator that produced it.

A measurement determines a projection; a check computed in the row space of that projection cannot fail; and the checks people reach for are almost always in the row space, because they are made of the same broad, smooth, convenient kernels the measurement was made of.

The corollary is a test worth building into any pipeline that reconstructs one thing from measurements of another. Predict something the measurement did not constrain, and check that. It is harder to arrange than a residual, it is often the only honest check available, and it is the difference between knowing that a model reproduces its own data and knowing anything else at all.

Twenty-four filters on a fluorescent tube is a serious instrument against the hardest lamp, and it is where the residual is smallest and still there.

What a reconstruction ought to be shipped with #

Three items, and the first two are already computable by anybody who has the instrument’s filter shapes.

The dimension of what was determined, which is the rank of the filter bank rather than the number of values printed. An instrument printing eighty-one numbers from a bank of rank forty is printing a forty-dimensional object in eighty-one coordinates, and a user has no way to know that from the file.

A statement of the reconstruction rule. A projection and a basis fit are different objects and the second invents detail. Which one produced the file decides what the file may be used for, and it is not usually recorded.

And a sensitivity, rather than a residual. The useful number is not how well the reconstruction reproduces its own readings — which is exactly, by construction — but how much a prediction made from it would move if the unmeasured part were different. That is one extra computation and it is the only one that answers the question a user has.

None of this is exotic. It is the same three-part demand a camera profile needs and the same one a whiteness figure needs, which is a sign that the demand is about measurement rather than about any of the three subjects.

Where the ladder goes next #

The same structure decides what a fitted model can be trusted for, and the general version of it is a fit that is exact on its own data and empty everywhere else — which is where this collection’s whole account of identifiability arrives.