A finer reading of a coarser table

A table of eighty-one numbers is often not the table a calculation wants, and the standard response is to fill in the gaps. Whether that helps or harms turns out to depend on something the table does not record, and the two answers are five orders of magnitude apart.

The claim #

Interpolating a spectral table does not recover resolution. It exchanges one error for another, and whether the exchange is a good one is decided entirely by how much structure the light has.

• On a smooth light it helps, by about a factor of five, and what it removes is not a sampling error at all — it is the end cells of the truncated range.
• On a light near the grid’s limit it harms, catastrophically: a three-emitter LED whose direct sum is exact to six decimal places comes back with an error of 0.061 ΔE₀₀ after linear interpolation.
• On a light past the limit it does nothing. All three rules and the direct sum agree to three decimals on a fluorescent tube, because they are all reproducing the same missing mercury lines equally badly.
• And a smoother rule is not a better one. Straight lines are worse than a staircase, by a factor of two, in every case where either is worse than doing nothing.

Where interpolation comes from #

Nobody interpolates a spectrum for pleasure. It happens because two tables have different intervals and something has to multiply them together — a reflectance measured every ten nanometres against an observer tabulated every five, a lamp reported at one against a filter reported at two. One has to move onto the other’s index, and moving to the finer of the two is the choice that looks lossless.

The CIE has a technical report on exactly this and it recommends a particular rule for particular reasons. What this essay asks is a narrower question than the report’s: not which rule reproduces the curve best, but which rule produces the best colour. Those are not the same question, because a colour is an integral and an integral forgives a great deal of local error while punishing certain systematic ones.

The answer depends on the thing the report is careful about and general practice is not — whether the coarse table came from sampling a spectrum or from integrating one through a slit. A slit is what makes a coarse table honest, and it is also what makes one interpolable, because a band-limited function is the only kind an interpolator can be right about.

The measurement, across five lights #

Each light is tabulated at five nanometres, read at one by each rule, and integrated against the analytic observer. The sample is a red pigment throughout. The reference is the same colour at a tenth of a nanometre over the same range, so nothing here is the truncation in disguise.

Three regimes, and they are not degrees of the same thing.

Regime one is the smooth light, where interpolating helps. The top two rows improve by a factor of five whichever rule is used, and the three rules agree with one another to the fourth decimal place. That agreement is the tell: if the improvement were about following the curve, the cubic would beat the staircase, and it does not.

What the interpolation removes is the end cells. A rectangle sum over 380 to 780 carries a whole cell at each end where the integral wants half of one, that term is first order in the spacing, and it is most of what the five-nanometre error on a smooth light actually is. Resampling to one nanometre divides that term by five, and it does so no matter what happens between the points, because the interior is smooth enough that every rule agrees there.

So the improvement is real and it is not the improvement anybody thinks they are buying. Halving the two end weights — the trapezoid rule, one line of arithmetic on the original eighty-one numbers — takes the same row from 0.0640 to 0.0045, which is nearly three times better than any interpolation and costs nothing.

Regime two is near the limit, and there interpolating is a disaster. The three-emitter lamp is the row worth staring at. Its narrowest feature is eighteen nanometres, which four tabulation points describe. Summed directly at five nanometres it is exact to five parts in ten million — better than any other entry in the table, better than the smooth lights, better than the interpolations of itself.

That exactness is not luck. It is the cancellation: the grid appears twice in a tristimulus value, once in the sample’s sum and once in the white’s, and on a light whose structure the grid nearly resolves the two errors are the same error and divide out almost completely.

An interpolator destroys that. It replaces the tabulated lamp with a different function — one that agrees at the eighty-one points and differs between them — and the sample’s sum and the white’s sum are now both taken over the new function at four hundred and one points. The errors they make are still correlated, but they are errors against a curve that is no longer the lamp. The cancellation is intact and it is cancelling the wrong thing.

The size of it is the factor of a hundred and twenty thousand between the first column and the third. Nothing about the interpolation is badly implemented; every rule reproduces the tabulated values exactly and interpolates plausibly between them. The operation is simply not information-preserving, and the direct sum had been living off information the interpolation throws away.

Regime three is past the limit, where nothing matters. The fluorescent tube’s four entries agree to two decimal places at 0.98 ΔE₀₀. Its mercury lines are a nanometre wide and the table has no information about them at all, so every rule invents the same nothing and the direct sum invents it too.

This is the clean case in the sense that no decision is being made badly. It is also the case where a practitioner is most likely to reach for interpolation, because the coarse answer is visibly wrong and a finer grid looks like the repair. It is not the repair. The repair is a wider slit at the instrument, applied before the table exists, and once the table exists there is nothing to do.

The three regimes are visible in that spread as well as in the table, and reading them together settles which of a light’s properties is doing the work. It is not the number of emitters, the correlated colour temperature or the total power; it is one length, the width of the narrowest feature, measured against one other length, the tabulation step.

Why a straight line is worse than a staircase #

Across the two rows where interpolation harms, the linear rule is worse than the held-value rule by almost exactly a factor of two. That inverts every intuition about smoothness and the reason is worth having.

The two rules make different kinds of error and only one of them cancels. A held value is wrong by roughly half the local slope times the cell width, and it is wrong in opposite directions on a rising limb and a falling one. Summed over a peak, the overshoot on one side largely cancels the undershoot on the other.

A straight line does not overshoot at all. A chord lies below a convex curve everywhere, and a spectral emitter is convex across its whole peak, so a linear interpolation under-fills every peak and over-fills every trough with no change of sign. The errors accumulate rather than cancelling, and multiplying by a one-signed observer function cannot rescue them.

A rule can be smoother and less accurate at the same time, and the property that decides which is not smoothness but whether the sign of the error alternates. That is a general fact about quadrature, and it is invisible when the only rules being compared are the two that everybody uses.

What a fill-in rule can be right about #

There is a clean statement of what interpolation can recover, and it settles the whole question.

A tabulation at spacing h determines the underlying function completely if the function has no structure finer than 2h. That is Nyquist’s condition, and the reconstruction it licenses is not a polynomial through neighbouring points but a sum over the entire table. If the condition holds, some interpolator recovers the function exactly. If it does not, no interpolator recovers anything, because the information is not in the table to be recovered.

The three regimes above are that condition, read off. Comfortably inside it, the rules agree with each other and the residual is the end term. Near the boundary, the rules diverge from one another and all of them are worse than the arithmetic that never left the table. Outside it, they converge again onto a shared wrong answer.

The middle regime is the one worth naming because it is where the practical mistakes are made, and it is narrow: lights whose finest feature is between about two and about ten times the tabulation step. That range covers most LED lighting and almost every display primary, which is to say most of what has been built in the last twenty years.

The other direction: coarsening, and doing it twice #

Everything above reads a table finely. The reverse operation is at least as common and it compounds in a way that is easy to miss.

Coarsening a one-nanometre table to ten by taking every tenth value is a point sampling, and this collection has already measured what that costs: a fluorescent tube pays about six times more for that than for averaging each cell. Averaging is a rectangular slit, applied in software, and it works for the same reason a physical slit works.

What is new is what happens when the two operations are composed. A ten-nanometre table read back at five has been point-sampled and then interpolated, and its error is not the sum of the two errors — it is the interpolation’s error measured against a curve that was already wrong. The direct sum at ten nanometres costs a red pigment 0.123 ΔE₀₀ under daylight, and no reading of that table at five can go below it, because the information ceiling was set by the coarsening.

That is the practical warning and it applies to every data-handling chain: the finest grid in a pipeline does not set its resolution; the coarsest one does, and the finest one only sets how confident the output looks.

What this collection does about it #

The collection’s own answer is to avoid the question, and that is worth stating plainly rather than presenting as a virtue.

Every spectrum here is a formula. Planck’s law is evaluated rather than looked up, the daylight reconstructions are built from three basis functions and a temperature, and every reflectance in the figure families is a stated expression in the wavelength. Nothing is ever interpolated, because nothing was ever tabulated — which is a legitimate design and is also exactly why the collection point-samples where an instrument integrates.

Two things had to be quoted rather than computed. The colour-matching functions are a measurement of people, and the daylight basis functions are a measurement of skies; both are carried as five-nanometre tables and both are used at five nanometres and never read between their points. An interpolation that is never performed has no error, and a grid that is never left cannot be caught leaving it.

It is also why the audit of this round had to build an analytic observer before it could measure anything at all. Measuring a grid needs something finer than a grid, and the tables the collection quotes cannot supply it.

Where the same shape appears elsewhere #

A colour-management profile is a lookup table with an interpolation rule attached, and every argument above transfers with the wavelength axis swapped for three axes of colour.

A profile is a fit between its nodes and its error is the same object — a shape introduced by the rule, in a place where the table has no information. Two things differ. The grid is three-dimensional, so nodes cost the cube of the resolution and the pressure to interpolate is far greater. And the underlying function is not band-limited in any useful sense, because a gamut boundary has a genuine crease in it, so the Nyquist argument does not even offer the consolation it offers here. A profile is permanently in regime three and has no regime one to retreat to.

What was computed, and how #

The three rules are implemented directly rather than called from a library, because the point of the comparison is what each does between the points rather than what a library is named.

The held rule rounds to the nearest tabulated index. The linear rule is the obvious one. The cubic is Catmull–Rom through four consecutive points, which is the cheap cubic in general use and is deliberately not the CIE’s recommended five-point Sprague form — the gap between those two is far smaller than the gap between either and the linear rule, and naming the cheap one is the honest description of what most code does.

Each interpolated lamp is integrated at one nanometre with its own white point computed the same way, and compared against a tenth-nanometre reference over the same range. The first version of this measurement used a reference running from 300 to 830 and reported 0.49 ΔE₀₀ for every rule on daylight — the range truncation, identical in all four columns, swamping the thing being measured. A comparison has to hold everything but its subject fixed, and a range is very easy to forget is a variable.

Where the model stops #

The reference is analytic and the tables in the world are not, so this measures what the rules cost on a known function. Given a real tabulated observer the experiment cannot be run at all, which is the essay’s point and also its boundary.

Sprague’s five-point rule is not measured, and it is very likely to beat Catmull–Rom on the rows where the cubic already wins. Its absence is deliberate rather than an oversight: the comparison needs the rule that is in code, not the rule that is in the recommendation.

And the lights are constructions. A real three-emitter lamp has emitter shapes that are not Gaussian and a phosphor tail no formula here carries, so its eighteen nanometres is a nominal width. The three regimes are structural; which regime a given real lamp falls in is a measurement nobody here has made.

Who found it, and when #

Sprague published his five-point interpolation in 1880 for actuarial tables. Catmull and Rom published their cubic in 1974 for computer graphics, and it reached spectral work by the route most numerical methods reach it — because it was already in the toolbox.

The CIE’s technical report 167:2005, on tabulating spectral data for colour computations, is the document that names a rule, and its more useful contribution is the insistence this essay turns on: a table produced by sampling and a table produced by averaging over a band require different treatment, and one that does not record which it is cannot be treated correctly at all. Almost no data format has a field for it.

Where the ladder goes next #

Three decisions inside a tabulation have now been measured and every one has come out as a pairing — a cost that is a product of something the index has and something the light has. The next essay takes that structure to its limit and finds the case where the product is empty: the samples for which every grid, every slit and every rule give exactly the same colour, which turns out to be a larger and duller class than it sounds.