The grid outside every figure

Sixty-one essays on this site were computed on the same wavelength grid: 380 to 780 nanometres, at 5 nanometre steps, eighty-one bands. Every illuminant, every reflectance, every matching function, every integral.

Nobody had to think about it, because for an eye the choice is not a choice. The matching functions are zero outside that band, so extending it adds zeros.

Then this field arrived.

The device as it actually ships is the third of the three plates, and it is the one every other essay on this site is quietly assuming when it says “a camera”.

The claim #

A collection built on a shared assumption meets, sooner or later, a subject for which the assumption is false — and the right response is usually not to change the assumption. This field’s grid was not widened. A second grid was built alongside it, the boundary between them was made explicit, and the difference between integrating on one and on the other became a measurement rather than a caveat.

What widening would have cost #

The obvious move is to extend the global grid to 1100 nanometres and be done. It fails in three ways, and the third is fatal.

It touches everything. Eighty-one bands become a hundred and forty-five. Every illuminant array, every reflectance constructor, every observer table, the metamer projection’s matrix dimensions, the system matrix, every figure that plots a spectrum, and every cached integral. A hundred and seven generators would need checking, and the regression test is that nothing changes except where intended — on a change that touches every array in the library.

It buys nothing for sixty-one essays. The matching functions are zero out there. Every one of those integrals would gain sixty-four terms of exactly zero, at a cost in build time and cache size, to produce identical answers.

And it cannot be done honestly. This is the one that decides it. The CIE D-series daylight illuminants are not measurements of daylight; they are reconstructions from three basis functions, and those basis functions are tabulated to 780 nanometres and stop. There is no defensible extension of D65 past the visible band. Extending it by assumption — a smooth continuation, a blackbody fit, a flat tail — would mean every infrared figure on this site depended on a number nobody measured, presented with the same authority as the rest.

What was built instead #

A second grid, local to one library, running 380 to 1100 nanometres at the same 5 nanometre step. The two share an origin and a step by construction, so moving between them is a slice rather than a resampling — which is the property that makes them comparable at all, and is why the extended grid starts at 380 rather than at 350.

Three rules govern what may live on it.

Everything on the extended grid is analytic or constructed here. Planck’s law extends exactly, because it is a formula. The sensor, the filters and the test surfaces are constructed in this library and can be defined anywhere. Nothing tabulated is extrapolated.

Anything needing a tabulated illuminant is computed on the visible grid and says so. The Luther residual, the cross-metamers, the colour matrix and the noise sweep are all visible-band computations under D65, and they are correct there.

And the projection between the grids is a named function that refuses the wrong argument. visiblePart takes a 145-band spectrum and returns 81; handed an 81-band one it throws rather than silently truncating something that was already truncated. So does extend in the other direction. Both refusals are in the rejection suite.

The measurement at the boundary #

The decision would be a caveat if it were only stated. It is a measurement because the difference between the two grids is computed.

Integrate an uncut sensor looking at a saturated surface under a tungsten lamp over 380–780 and over 380–1100 and compare: 93.3 per cent of the recorded signal lies beyond 780 nanometres. Do the same for the eye and the two grids agree to a chromaticity difference of $10^{-15}$ — not approximately, exactly, because the matching functions are zero.

That pair of numbers is the whole justification for the arrangement. The visible grid is not an approximation for the eye; it is exact. It is not an approximation for the sensor either; it is wrong, by a factor of fifteen. Two different situations that a single “this site works in the visible band” caveat would have flattened into one.

The device as it actually ships is the third of the three plates, and it is the one every other essay on this site is quietly assuming.

Which computation the 93.3 is #

That sentence names one computation and its number comes from a slightly different one, and both are worth having.

With no surface in the light path — a bare uncut sensor looking straight at a 2856 K lamp — the share of the recorded signal beyond 780 nanometres is 82.7 per cent, a factor of 5.8 rather than fifteen.

With a surface in it the share rises, because a saturated surface is dark across most of the visible band and pale in the infrared. The sweep below uses one surface at a chroma of 0.8 with a tail of 0.62, and that surface gives 93.35 per cent at 2856 K, which is the figure quoted. Across eight such surfaces it runs from 84.5 to 93.3, averaging 87.8.

Neither number is wrong and the second is the more relevant one, since a camera is always looking at something. The gap between them is the mechanism this whole field is about: the infrared share is not a property of the sensor, it is a property of the sensor, the lamp and the surface together, and taking the surface out of the calculation lowers it by ten points.

The sweep is worth reading as a curve rather than as a point. Beyond 780 nanometres sits 97.15 per cent of the recorded signal at 2200 K, 93.35 at 2856, 83.52 at 4000, 62.77 at 6500 and 50.65 at 9000 — and the cut filter leaves 0.000 per cent at every one of those temperatures, to three decimals. The filter is not a partial repair.

The caveat that was available after all #

The reason given for building a second grid is that no caveat could carry the omission, because the infrared has no known sign. It has one.

Under tungsten, the uncut sensor’s blue-to-green channel ratio is higher than the cut sensor’s on all eight test surfaces, by between 0.30 and 0.81, and its red-to-green ratio is lower on seven of the eight and unmoved on the last. Every ratio travels towards one. As a single statistic, the mean spread of the three channels about their own mean is 1.045 with the filter and 0.119 without, and surface by surface the uncut sensor keeps between 6.7 and 15.1 per cent of the colour separation. Under daylight at 6500 K it keeps between 28.8 and 52.2.

So the sentence does exist: an uncut sensor records less saturation than a cut one, always, by a factor between about two and fifteen depending on the lamp and the surface. That is a bound with a known sign, which is the thing the argument above said was unavailable.

It does not change the decision. It sharpens the reason for it, because a bound on saturation is not a bound on colour. Knowing that every reading has moved towards neutral says nothing about where in hue it has landed: the red-to-green ratio fell by 0.006 on one surface and by 2.12 on another, so one true one-way statement covers both a shift nobody could see and a shift that turns a saturated red into a near-grey. A caveat needs a sign and a bounded magnitude. This one has the first and not the second, and the second is what a reader would have to have.

The chromaticity mistake, which happened here #

The comparison above was got wrong on the first attempt, in a way worth recording because it is the characteristic hazard of having two grids.

The first version of the assertion compared the eye’s XYZ computed on the visible grid against the eye’s XYZ computed from the visible part of the extended grid, and expected them to agree to machine precision. They came out 17 per cent apart.

Nothing was wrong with either integral. The site’s blackbody normalises its output to its peak within 380–780; the extended-grid version normalises to its peak within 380–1100. For a 2856 K source those are different wavelengths — Wien puts the peak at 1015 nanometres — so the two spectra differ by a constant factor, and a comparison of absolute XYZ reports that factor as a disagreement.

The claim being made was about whether the eye sees anything past 780, which is a question about shape. Comparing chromaticity instead of XYZ gives $10^{-15}$ and answers the question that was asked.

A normalisation is a choice, two grids invite two different ones, and a quantity compared across them has to be one the normalisation does not touch. That is the specific lesson and it generalises immediately to any pair of pipelines meeting at a boundary.

What the grid already hid, twice #

This is not the first time the 380–780 range has been load-bearing on this site, and the two earlier cases are worth putting beside it because they were handled differently and both were right.

Fluorescence. An optical brightener absorbs in the ultraviolet and re-emits in the blue, and its excitation band extends below 380 nanometres — so the site’s grid truncates the excitation and every fluorescence number computed here is a declared floor rather than a value. That was recorded when it was found and the numbers are stated as lower bounds.

Line width. Any emission narrower than the 5 nanometre grid is an instrumental width rather than a real one. A mercury line is about 1 nanometre wide and appears here at 5, so a figure of a fluorescent lamp’s spectrum shows a line whose width is a property of the sampling.

Both were handled by stating the consequence rather than by changing the grid, and in both cases that was the right call: the affected quantity is a bound or a width, which is something a caveat can express precisely.

This field is the first case where a caveat could not do the job, because the affected quantity is not a bound in a known direction. An infrared contribution has a direction — it collapses every channel ratio towards one — and no bound on how far, so it makes a reading a different colour by an amount the lamp and the surface decide between them. A later section measures both halves of that. There is no sentence of the form “the numbers here are floors” that captures it.

That is the test worth carrying: a caveat works when the omission has a known sign and a bounded magnitude, and fails when it does not. The first two cases pass it. This one does not, which is why it got a second grid rather than a third sentence.

Where the model stops #

The extended grid stops at 1100 and silicon stops at 1107. The last band is a rounding rather than a physical edge, and the response between 1100 and 1107 is small and non-zero. Nothing here depends on it.

The surfaces on the extended grid are parameterised, not measured. extendSurface ramps a reflectance from its 780 value to a stated tail and holds it flat, and real near-infrared reflectance has water bands around 970 nanometres and structure that near-infrared spectroscopy exists to exploit. The model has the level and not the structure, and every figure states the tail it used.

And the arrangement is one library deep. If a second field ever needs the extended grid, the right move is to promote it — and the fact that it currently lives in lib/imaging.js rather than in lib/spectrum.js is a deliberate statement that one field needed it and the rest of the site did not.

What a reader should take from a figure on either grid #

The arrangement is only worth anything if it is visible from outside the code, so the figures carry it.

Every figure on the extended grid marks 780 nanometres with a rule and says what it is. A reader arriving here after sixty essays of 380–780 will read a wavelength axis as the familiar one unless something stops them, and a plot running to 1100 with no marker would be read as running to 700 with an unusually wide right margin.

Every figure on the extended grid names a device in its caption strip rather than an observer. That is the new band the observer index gained for this field, and it is the same argument: a plot of silicon’s quantum efficiency does not change when the colour-matching functions do, so naming an observer on it would name the wrong thing.

And every infrared figure states its surface’s tail, because that parameter is what the whole finding is sensitive to and a figure quoting a number without it is quoting a number about an unstated material.

None of that is enforced by a gate. The rule and the tail are conventions, and the only one with machinery behind it is the caption strip, which observercheck requires to be classifiable. That asymmetry is worth admitting: the parts of this arrangement that are checked are checked, and the parts that are habits are habits.

The generalisation #

The transferable problem is not about wavelengths. It is about what to do when a shared assumption underlying a body of work stops holding for one part of it, and there are three responses with quite different costs.

Widen the assumption for everybody. Correct, expensive, and it makes every existing result depend on machinery it does not use. Worse, it usually requires inventing data outside the original range, which converts a clean boundary into a diffuse one — every result now rests on an extrapolation, and nothing marks which.

Keep the assumption and add a caveat. Cheap, and it fails silently, because a caveat is not enforced. Nothing stops the next author computing an infrared quantity on the visible grid and getting a plausible answer.

Or build a second, local, explicitly-bounded system and measure the difference between them. Costs one library and a projection function, keeps every existing result exactly as it was, and — the part that matters — turns the boundary into a number.

The third is what this field did and it is the one to prefer whenever the boundary is real rather than accidental. The test for which case is which: if the assumption is false for the new work and the old work would be unchanged by widening it, the boundary is real and a second system is right. If widening would change old results, the assumption was wrong all along and the first response is the honest one.

And the enforcement is the whole point. A caveat is a sentence and a type error is a refusal, so visiblePart throws on an 81-band array and extend throws on a 145-band one. Both are in the rejection suite, because an assertion that has never rejected anything proves nothing.

Why the step stayed at five nanometres #

Widening the range was the decision this essay is about; changing the step was the one that was never seriously in question, and the reason is worth a paragraph because it points the opposite way.

Five nanometres over 380–780 gives eighty-one bands, which is enough for every smooth spectrum this site computes with and is not enough for narrow ones. A mercury line, a laser, a monochromator’s passband: all are narrower than the sampling, and the site handles each by representing a monochromatic stimulus as a triangle one grid step wide rather than as a single sample, so that the result does not silently depend on where the line falls relative to the grid.

Refining the step to one nanometre would improve those cases and would multiply every array by five, for a subject whose reflectances are smooth. The trade is the reverse of the one this essay makes: the step is a resolution choice where more is uniformly better and expensive, while the range is a domain choice where more is worthless for most of the site and impossible for part of it.

Distinguishing the two kinds of parameter is most of the decision. A resolution can be traded against cost by anybody, at any time, and the answers converge as it improves. A domain cannot: extending it requires data that either exists or does not, and where it does not, extending is inventing.

Who noticed, and when #

The visible band’s boundaries have never been sharp and the standards have moved them. The CIE’s 1931 tables ran 380 to 780; various later recommendations use 360 to 830 with the extreme values essentially zero; and instrument makers build to 360–740 or 380–780 depending on the market. Nobody disputes the physics, because the disagreement is about where a curve becomes negligible rather than about where it ends.

What is more interesting is the near-total separation of the two literatures this essay sits between. Colour science works in the visible band and treats everything else as out of scope. Remote sensing, near-infrared spectroscopy and machine vision work well past it and treat the colour-matching functions as one instrument among many. The same absorption edges, the same silicon, the same materials appear in both, described in different vocabularies, and a camera is the object that forces them together — because it is a colorimetric instrument built on a detector that neither field would have chosen for the other’s purposes.

Where the ladder goes next #

Downward, this rung sits on most things are pale in the infrared, which is the physical fact the grid cannot represent, and on a spectrum is not a colour, which is where the grid was first taken for granted.

Upward, one essay remains in this field. A photograph is not a measurement collects every limit this field has established — the kernel, the null spaces, the fitted matrix, the guessed illuminant, the invented pixels, the clipped highlights and this grid — into one answer about what a photograph can be used to establish.