Three numbers cannot see a line

An instrument that measures light does one thing: it multiplies the arriving spectrum by a filter, integrates, and reports a number. A colorimeter has three such filters. A spectrophotometer has thirty-one or a hundred, arranged as a grating and a detector array, but the principle is identical and so is the arithmetic.

What such an instrument determines is a projection, and the projection has a null space.

The claim #

A bank of n filters reading an eighty-one band spectrum determines the projection of that spectrum onto an n-dimensional subspace, exactly, and says nothing at all about the remaining 81 − n dimensions. How much of a lamp lives in the part that is determined is a property of the lamp, and nothing about the instrument tells its user which case they are in.

• Three readings leave 12.5 per cent of daylight unseen and 74.6 per cent of a fluorescent tube, measured as the share of the spectrum’s own magnitude lying in the filters’ null space.
• Twelve readings leave 4.6 per cent of daylight and 66.0 per cent of the tube. Four times the instrument buys the tube almost nothing.
• Forty-one readings — a ten nanometre bank — still leave 19.9 per cent of it. The mercury lines are narrower than any filter in the bank.
• The reconstruction used is the minimum-norm one, which is the projection itself, so the number is a statement about the instrument rather than about anybody’s prior.
• And the shape of the spectrum decides everything. The same instrument is nearly exact on one lamp and hopeless on another, with nothing in its output to distinguish the two cases.

That last point is the one worth carrying away, because it inverts the usual way an instrument is judged. An instrument’s specification is a statement about the instrument: its accuracy, its repeatability, its traceable calibration. The quantity that decides whether a measurement means anything is a statement about the pair — this instrument and this source — and no amount of care with the first half settles it.

What the readings determine #

Write the spectrum as a vector ρ of eighty-one band values and the filters as rows of a matrix F. The readings are y = F ρ.

The set of spectra consistent with y is an affine subspace: any particular solution plus anything in the null space of F, which has dimension 81 − n. Nothing distinguishes the members of that set — not a better algorithm, not a longer integration time, not a more careful calibration.

The one point of the set that no reading argues for is the minimum-norm solution ρ̂ = Fᵀ (F Fᵀ)⁻¹ y, which is the projection of the truth onto the row space of the filters. That is what is drawn throughout this essay, and choosing it is a decision worth stating: a reconstruction fitted to a basis is a prior, and a prior recovers detail the instrument did not measure. Legitimate practice, and a different claim.

Why the lamp decides #

The unseen share is the norm of the component orthogonal to every filter, divided by the norm of the spectrum. Both terms are properties of the pair — instrument and lamp — and the instrument is held fixed here.

Daylight is nearly in the span from the start. It is a smooth, broad function with no structure narrower than a filter, so a handful of broad integrals capture nearly all of it and each further filter takes another bite out of what is left. Twelve readings reach 4.6 per cent.

There is a useful rule of thumb hiding in the comparison. The share a bank of n broad filters misses is governed by how much of the source’s power sits in features narrower than a filter — so a source with a smooth envelope and a few narrow lines has an unseen share that is roughly the fraction of its power in the lines, and adding filters removes the envelope’s contribution and leaves the lines’ untouched. That is exactly the shape of the tube’s curve: a steep initial fall as the continuum is captured, and then a long flat approach to a floor set by the lines.

A fluorescent tube is not. It is a broad phosphor continuum with mercury lines a few nanometres wide standing on top of it, and a line is very nearly orthogonal to any filter wide enough to be useful. Its inner product with a filter is small no matter where the filter is; the line contributes almost nothing to the reading and almost everything to the norm.

What “unseen” is a share of #

The quantity being reported deserves a paragraph, because a percentage of a spectrum is not an obvious thing and there are several one could mean.

It is the Euclidean norm of the residual divided by the Euclidean norm of the spectrum — the length of the part orthogonal to every filter, as a fraction of the whole. That treats each of the eighty-one bands as equally important, which is exactly right for the question what does the instrument determine and exactly wrong for the question what does this cost.

The alternatives answer different questions and both are computed elsewhere in this collection. Weighting by the observer asks what the loss costs a colour, which is the next essay’s subject and gives a much smaller answer. Weighting by a surface’s reflectance asks what it costs a particular measurement, and the answer depends on the surface — which is the whole reason a spectral measurement is made rather than a colorimetric one.

The unweighted version is the conservative and instrument-centred one, and it is the one an instrument’s user has no way to compute, because computing it requires knowing the spectrum the instrument was supposed to measure.

This is not the same as metamerism #

The resemblance to two spectra with one colour is exact and the two are worth keeping apart, because they are the same theorem applied to different instruments.

Metamerism is this argument with n = 3 and the filters being the colour-matching functions. The eye is a three-channel instrument, its null space is seventy-eight-dimensional, and a metameric black is a member of it. Everything in this essay is the general form.

What differs is what the null space is for. For an eye, being blind to metameric black is not a defect: two spectra with the same projection genuinely look the same, and that is what colour is. For an instrument, being blind to a mercury line is a defect, because the reason for measuring the lamp was to predict what it will do to some surface that has not been chosen yet — and that prediction needs the parts of the spectrum this observer happened not to weight.

So the same fact is a definition in one case and a limitation in the other, and the difference is entirely in what the measurement is for.

What was computed, and how #

The bank is n Gaussian filters with centres spread evenly across 380 to 780 nanometres and a width tracking the spacing, so that a bank of any size covers the range rather than leaving gaps between its filters. A fixed width would have made the sweep non-monotone for a reason that is about the instrument’s gaps rather than about how many numbers it returns, which is the question; the first draft did exactly that and reported a tube worse at twenty readings than at twelve.

The projection is computed by solving the n × n Gram system rather than by fitting to a basis, which is what makes the answer independent of any assumption about spectra. The unseen share is the Euclidean norm of the residual over the norm of the spectrum, both on the site’s own eighty-one band grid.

The grid itself is a prior and is the honest limit of the whole exercise. A mercury line has a physical width of well under a nanometre; on a five nanometre grid it is already a single band with the line’s total power in it, which is the choice this collection made in its first commit and re-examined once. The numbers here are therefore about a line that has already been broadened to five nanometres, and a finer grid would make the tube’s floor higher rather than lower.

Three quarters of the norm is about a fifth of the power #

The measure is stated carefully — a Euclidean norm over eighty-one equally weighted bands — and the consequence of that choice is larger than the paragraph stating it suggests.

A norm ratio does not weight bands equally in any sense a reader would expect. It weights values equally, and a narrow feature has a large value in one band while a continuum of the same total power has a small value in eighty. Put three mercury lines carrying a fraction of the power into single bands and spread the rest smoothly, and their share of the Euclidean norm is:

The amplification at small shares is √(78/3) = 5.1, and it is a property of the arithmetic rather than of any lamp: concentrating power into a twenty-sixth of the bands multiplies its norm by the square root of twenty-six.

Read backwards, that prices the essay’s headline figures. A 74.6 per cent unseen norm corresponds to roughly 18 per cent of the tube’s power; 66.0 per cent to about 15 per cent; and the ten-nanometre bank’s 19.9 per cent to under 4. Those are the numbers a lighting engineer would recognise as the mercury lines’ share of a triphosphor tube, and they are what the essay’s own rule of thumb — an unseen share that is roughly the fraction of its power in the lines — actually predicts.

So the rule of thumb is right about the mechanism and wrong by a factor of five about the size, because the mechanism is stated in power and the measurement is taken in norm. The correction does not weaken the finding; the instrument really does determine nothing about those lines. It changes what three quarters is three quarters of.

Six filters is twice three and a quarter of twelve, which makes it the reading that says whether the unseen share falls with the instrument’s size or with something else.

Why this is the right measure anyway #

Having said that, the unweighted norm is still the number to publish, and the reason is worth having because it is not obvious once the amplification is visible.

The question the measure answers is what the instrument determines, and for that question every band is a coordinate and the row space is a subspace — there is no privileged weighting, and any weighting imported at this stage is a claim about what the measurement will later be used for. A power weighting would say a line is unimportant because it is faint. A line’s importance depends entirely on whether some future surface has a feature at that wavelength, and choosing the weighting before choosing the surface is the error the essay is warning about.

The honest form is therefore to publish both. The norm share is the instrument’s number and the power share is the reader’s intuition, and the gap between them is the factor of five above. Publishing only the first invites the reading that three quarters of a tube’s light goes unmeasured; publishing only the second invites the reading that a fifth is a small problem, when the fifth in question is precisely the part that decides a metameric match under that lamp.

The floor moves faster than the threshold argument suggests #

The account of why more filters help slowly predicts a hard threshold at the point where a filter’s width crosses a line’s, and the sweep’s own numbers put that crossing later than the fall.

The bank’s filter width tracks its spacing, so it is about 133 nanometres at three readings, 33 at twelve and 10 at forty-one. The line, already broadened to the grid, is five nanometres wide. At forty-one readings the filters are still twice the line’s width, and the unseen share has already fallen from 66.0 to 19.9 per cent — most of the way to zero, on the wrong side of the stated threshold.

Two things are happening at once and the threshold account describes only one. A filter does not have to be as narrow as a line to see it; it has to be narrow enough that the line’s contribution is not swamped by everything else the filter integrates. A neighbouring pair of filters straddling a line also localises it, because the ratio of their two readings moves when the line does. So the recovery is gradual over a wide range and turns sharp only at the very end.

The practical version of the rule survives with a different constant. Compare the bandpass to the narrowest feature and expect useful recovery from about five times it, rather than at parity — which moves the answer for a mercury line from an instrument nobody buys to a ten-nanometre spectrophotometer that many laboratories already have.

Where the model stops #

Real instruments are not Gaussian filter banks. A grating spectrophotometer has a bandpass set by its slit, with a roughly triangular profile and stray light in the wings; a filter colorimeter has whatever its glass does. The shapes change the numbers and not the structure — any linear instrument determines a projection and is blind to its complement.

Noise is not modelled at all. Every reading here is exact, so the unseen share is a pure statement about the null space. Real readings have noise, which turns the sharp boundary between seen and unseen into a graded one and makes the effective null space larger than the nominal one — so the numbers here are optimistic.

The filters are also assumed to be known exactly. A real bank’s transmittances are themselves measured, on an instrument with its own bandpass, and an error in the filter shapes rotates the row space rather than enlarging it — a different failure that the projection framing does not capture. What an instrument brings with it is a longer list than its filters.

And nothing here is about accuracy. An instrument can be perfectly calibrated, perfectly repeatable and perfectly traceable, and still return three numbers. What it reports and what it measures are different questions, and this is the second one.

Who found it, and when #

The linear algebra is older than the instruments and is not in dispute anywhere. What is worth noting is where the practical knowledge sits: lighting metrology has known for decades that a colorimeter cannot be trusted on a discharge lamp, and says so in its standards, which is why colour rendering indices are required to be computed from spectral rather than tristimulus measurements.

The number this essay adds is the size of the effect at various instrument resolutions, and in particular the floor. It is easy to assume that a spectrophotometer at ten nanometres is spectral enough for anything; on a mercury line spectrum, a fifth of the lamp is still outside what it determines.

A smooth lamp is the case where a modest instrument does well, and sixteen readings against a white LED is about as favourable as the arithmetic gets.

Why more filters help so slowly #

The tube’s curve falls steeply at first and then very slowly, and the shape has a cause worth stating because it decides whether buying a better instrument is worth anything.

A broad filter’s inner product with a narrow line is roughly the line’s power times the filter’s value at the line’s wavelength, divided by nothing — it is small because the line is narrow, not because the filter is in the wrong place. So adding filters at new positions captures more of the continuum and captures the line only when a filter’s width becomes comparable to the line’s.

That is a threshold rather than a gradient. Until the bandpass crosses the line’s width, each new filter is buying continuum; after it crosses, the line appears almost all at once. A user doubling their instrument’s channel count on the wrong side of that threshold sees the unseen share fall by a few points and concludes, reasonably and wrongly, that the remainder is irreducible.

The number to compare is therefore not the channel count but the bandpass against the narrowest feature the source has, and the second half of that comparison is exactly what the instrument cannot supply.

The generalisation #

The pattern is an instrument whose output is complete, well-formed, repeatable and silent about the thing it cannot reach.

A projection reports no residual. That is the whole difficulty: a fit reports how far it missed, and a measurement does not, because from inside the row space there is nothing to compare against. The unseen share can only be computed by someone who already has the answer, which in practice means by simulation.

The same silence is what makes a camera’s three channels so hard to argue about, and what makes an image unable to determine its own illuminant. In each case the output is a projection, the projection is complete on its own terms, and the missing directions are missing without trace.

The practical consequence is a habit rather than a calculation. Ask what structure the thing being measured has, and compare it to the resolution of the instrument — not to the instrument’s stated accuracy, which is about the readings it does take. A lamp with lines needs an instrument with a bandpass narrower than the lines; nothing about the instrument’s specification sheet says whether the lamp has lines.

What would have to be quoted #

An instrument cannot report its own unseen share, and it can report the two things that bound it.

Its bandpass, which every specification sheet already gives and which is the number to compare against the structure of the source. A ten nanometre bandpass on a mercury line spectrum is the mismatch this essay measures.

And its channel count and spacing, which decide the dimension of what is determined. The two are not the same: an instrument with a hundred channels and a twenty nanometre bandpass has a hundred readings and far fewer than a hundred independent directions, because neighbouring channels overlap.

That second point generalises the essay’s whole result in one sentence. The dimension of what an instrument determines is the rank of its filter bank, not the number of numbers it prints, and the two are routinely confused — a spectrophotometer printing eighty-one values at one nanometre with a ten nanometre bandpass is printing eighty-one numbers that lie in a subspace of dimension nearer forty.

Where the ladder goes next #

The trap is worse than the numbers so far suggest, because the colour of the reconstruction converges long before the spectrum does. At twelve readings the tube’s colour is right to half a unit while two thirds of its spectrum is unmeasured, and a reconstruction that good will predict a match under a second illuminant that does not happen.