A grid is not a resolution

A word that names three different things is worse than no word, because it lets a decision be made without being noticed. This section began by taking one apart and ends by saying what should be said instead of it.

The claim #

“Resolution” names three separable decisions with three mechanisms, three repairs and two different rankings, and every ambiguity in this section came from having one word for them.

• A step is a quadrature parameter. Its repair is arithmetic and costs computation.
• A range is a truncation. Its repair is data and costs a modelling commitment on every sample.
• An origin is an alignment. It has no repair downstream at all, and its only remedy is a filter applied before the table exists.
• And the largest of the three is not the same one twice. Under daylight the range wins by nine to one; under a fluorescent tube the step wins by six thousand to one.

What ten measurements produced #

The section’s results, gathered, on a red pigment or over forty-two surfaces as noted, all against an analytic reference:

Reading down either column is instructive and reading across is the point. Every entry in the left column is small; the largest is the range. Every entry in the right column but two is either enormous or zero; the largest is the origin, which is not a thing anybody has ever varied.

No single number summarises that table, and every published guidance figure about spectral sampling is an attempt to.

The three mechanisms, named apart #

A step is about approximation. A sum over a finite grid differs from an integral by an amount that depends on the integrand’s curvature and on the spacing, and it converges as the spacing shrinks. Every intuition anybody has about numerical accuracy applies to it, and — as it turns out — most of what is usually charged to it belongs somewhere else.

A range is about omission. The truncated integral is not an approximation to the full one; it is a different integral, computed exactly. Refining does not touch it and no amount of care inside the range can recover what is outside. Its size is a product of what the observer has at the edges and what the light puts there, which is why one end costs three thousand times the other.

An origin is about alignment. It produces no error in the ordinary sense at all, because there is no single answer to be distant from; five legitimate five-nanometre tabulations of a mercury lamp give five different colours, and none of them is the true one. Refining does not converge. The only remedy is upstream.

Three mechanisms, three verbs: approximated, omitted, aliased. A calculation can be suffering from any one of them while the other two are perfect, and the diagnosis takes three separate tests.

There is a fourth mechanism hiding among the three and it belongs to none of them. A slit is not a property of the tabulation at all; it is a property of the instrument that produced the numbers, and it decides whether the other three questions even make sense. A table of band averages and a table of point samples are different objects that no format distinguishes, and the same summation is correct for one and wrong for the other.

That is why the section keeps returning to a single sentence about provenance. Everything downstream — which rule to use, whether to interpolate, whether the origin matters — is decided by how the numbers were made, and the numbers do not carry the answer. A collection computing from formulae is in a different regime from a laboratory reporting measurements, using the same file format and the same arithmetic, and neither can tell from the other’s data which it is looking at.

The procedure, in four lines #

What replaces the word is short enough to carry.

Slide the origin. If the answer moves, the calculation is aliased and nothing downstream will help. Stop and look at where the table came from.

Change the rule. If halving the two end weights moves the answer, most of what was being called a sampling error is the truncation, and the step is not the problem.

Widen the range at a fixed step. What moves is the omission, and it is usually the largest term under a smooth light.

Then refine the step. Whatever is left is the approximation, and it is usually the smallest of the three and the only one anybody had a name for.

The order matters. Running them the other way round produces a number for each that is contaminated by the ones not yet controlled, which is how this section’s first attempt reported a step error that rose as the step got finer.

The four tests have an order and it is worth defending, because the obvious order is the reverse. An instinct that begins by refining the step is beginning with the smallest term and the one most contaminated by the others: a step measured before the range is controlled reports the truncation, and a step measured before the origin is checked reports an aliasing that refining cannot remove. Each test in the list above removes a mechanism from the ones after it.

The order also happens to run from cheapest to most expensive, which is convenient rather than deep. Sliding an origin and changing two weights are minutes of work; widening a range is a modelling commitment across every sample in a system; refining a step is the only one that costs computation and it is the last one to reach for.

Why one word survived so long #

It is worth asking why a field as careful as colorimetry has one word here, and the answer is not carelessness.

For measured data the three decisions are genuinely linked. An instrument’s bandpass, its reporting interval and its range are set together by the optics: a monochromator’s slit width fixes the bandpass, the bandpass fixes the useful interval, and the grating’s range fixes the ends. Choosing one chooses the others, so a single number describes the instrument and the word “resolution” is a fair summary of it.

The link breaks when the spectrum is not measured. A calculation from formulae has a step chosen by convenience, a range inherited from a table, an origin nobody selected and no bandpass at all. The four are then independent, and a word that assumed they were linked describes none of them.

That is the same shape as several other things this collection has found. A convention becomes invisible when it stops being a choice, and a word that summarises four linked quantities keeps being used after the link is gone.

What the section changes about this collection #

Three specific things, all of them recorded rather than repaired.

The collection point-samples where an instrument integrates. Every spectrum here is a formula evaluated at eighty-one wavelengths, which is a zero-width slit, and for its narrowband lamps that is the worst case rather than the best. Any absolute colour quoted here for a discharge lamp or a laser primary would be different on a grid beginning one nanometre later.

Its summation is the standard’s and the standard’s is for measured data. Equal weights are correct for band-integrated values and wrong for sampled ones, and this collection produces sampled ones. The trapezoid rule is the right rule here and is not in use.

And its range is the largest of its three defects under the lights it draws most. That has been on the ledger for two rounds; what is new is that it has now been measured against the alternative rather than asserted, and it wins by about nine to one.

None of the three is fixed in this round. Each is a change that would move every number the collection has published, which is a reason for care rather than for delay, and the delay is the honest state.

Those two figures, taken together with the origin sweep, are the whole diagnostic procedure applied to one sample. They took about a second of computation each and between them they say which of three decisions a calculation is suffering from — which is the outcome this section exists to make routine, and which no amount of reading about resolution would have supplied.

What a data format would have to record #

A short and slightly damning list falls out of the section, of things a spectral file would need to carry for any of this to be checkable by a reader.

The interval and the range are recorded by every format in use. The bandpass is recorded by some. The origin is recorded by none, and for measured data it does not need to be, because a properly filtered measurement is nearly origin-independent.

What no format records at all is whether the values are samples or band averages, and that single bit decides which summation is correct, whether the trapezoid rule helps or hurts, and whether interpolation is meaningful. A file that does not carry it can be processed correctly only by somebody who knows how it was made.

None of this is hypothetical for a reader of this collection. Its own tables stop in different places, its illuminants and its observers were tabulated by different committees for different purposes, and the one property that would settle how to combine them is the one nobody records.

That is not an argument for a new format. It is an argument for the sentence that should accompany any spectral data anybody publishes, and it is one sentence: these are band averages over a triangular slit of five nanometres, or these are point evaluations of a formula.

One further thing the section produced deserves to be carried out of it, because it is the only piece of good news. A colour is a ratio, the grid appears in both halves of it, and most of a quadrature error divides out — by a factor between 1.3 and 4.1 on the lights measured here, and completely on a flat sample. Five nanometres has survived a century partly because it is fine enough and partly because the arithmetic everybody uses cancels its own error without being asked to.

That cancellation has a condition, and the condition is the same one everything else in the section turned on. It works when the error is a smooth function of wavelength that both integrals share. A light with lines narrower than the step gives the two sums errors about different lines, there is nothing shared to divide out, and the factor collapses to the point of vanishing. So the mechanism that protects the ordinary case is absent in exactly the case that needs it — which is a fair one-sentence summary of the whole section.

What the section could not settle #

Two questions were opened and not closed, and both are about samples rather than lights.

The whole section’s governing ratio compares the light’s narrowest feature against the step, and the general form is about the product of light and sample. Every sample here is smooth by construction, so no measurement in the section could distinguish the two statements. A structured sample — an interference filter, a structural colour, a laser-line notch — would put a calculation in the aliased regime with a perfectly smooth light, and nothing here says how common that is.

And the extrapolation problem under the range repair is untouched. Widening a range needs sixteen new rows for every reflectance, and a reflectance below 380 nanometres is a guess whose consequences exceed the truncation it replaces. That is the reason the largest defect stays unrepaired, and it is a physics problem wearing a tabulation problem’s clothes.

There is a last observation about the summary table that only becomes visible once all ten measurements are in one place. Six of the sixteen entries are either zero or below a thousandth of a unit, and they are not the same six in the two columns. A calculation under daylight has three live decisions and three dead ones; a calculation under a fluorescent tube has a different three of each. Nothing about the tabulation changed between the two columns — the same eighty-one wavelengths, the same range, the same origin — and yet the set of questions worth asking is disjoint.

That is the strongest argument the section can make for measuring rather than reasoning. A practitioner asked which of these decisions matters would answer from experience, and their experience would be under one kind of light. The four tests take a few seconds and answer for the light in the room.

What was computed, and how #

Every number in the summary table is measured against an analytic reference at a tenth of a nanometre, whose own convergence is checked at half that and moves by 3 × 10⁻¹³ ΔE₀₀ on the sharpest case in the file. The reference had to be built because a tabulation cannot be audited against a tabulation, and the price is a model residual of 1.42 ΔE₀₀ that the comparative results do not carry and the absolute ones do.

Each of the three decisions is measured with the other two held fixed, which sounds obvious and was the section’s first mistake: measuring a step against a reference over a different range reported the truncation in the step’s column and inverted the scaling.

The figure family carries seven assertions, of which the two that matter are that a smooth light’s answer is nearly origin-independent and a spiky one’s is not, and that a finite slit improves a line spectrum’s tabulated colour by more than a factor of ten. Both are written as ratios between mechanisms rather than as tolerances, because a tolerance would be a claim about the constructed lights.

The generalisation #

The habit is about words that summarise several parameters.

A single word for a bundle of linked quantities is efficient while they stay linked and becomes a hiding place when they come apart. The tell is that people disagree about the word’s value without disagreeing about anything measurable: one person says five nanometres is plenty and another says it is not, and both are right about different members of the bundle.

The repair is always the same and is always more work than the word. Name the members, measure them apart, and find out whether their ranking is stable. If it is, the word was earning its keep. If it reverses — as this one does, by a factor of six thousand, between two lamps in the same building — then the word was carrying a decision that nobody was making on purpose.

Who found it, and when #

Every individual mechanism here is standard. Nyquist and Shannon settled the aliasing, Euler and Maclaurin the endpoints, and the truncation needs no theory at all.

What is specific to this subject is the bundling, and it comes from the instrument. Colorimetry inherited its tabulations from spectrophotometers, whose four parameters are set together by their optics, and it kept the vocabulary after computation stopped being done on instrument output. The CIE’s recommended practice is careful about the pieces; general use is not, and the gap is a century of habit rather than an error anybody made.

Where the ladder goes next #

The index is finished. What remains of the colour integral is the third factor, and it is the one this collection has named on every figure it has ever drawn without ever asking what it is made of.

The third factor is a construction, with seven arguments a standard observer does not admit to having, and the round’s second half is the same audit run against it.