A lattice is a quadrature rule

The obvious repair for a mean over 125 objects is a mean over more of them, and it does not work in the way it looks as though it should.

The claim #

The site’s test set is not a sample of a population of surfaces. It is a quadrature rule for an integral over a region, and a quadrature rule can be biased at any number of points.

• Refining the lattice does not converge. Going from 39 points to 125 to 981 moves every census row by four to five per cent, and not monotonically — the value at 981 is on the far side of the value at 125 from the value at 39.
• The reason is the constraint. The region is bounded in L¹, so a finer grid inside it admits a different proportion of highly-modulated points, and the measure being integrated against changes with the density.
• Measured against the region’s own uniform measure, the lattice is out by −0.5 to +3.1 per cent, in a consistent direction on eleven of fourteen rows. That is a bias, not a variance.
• The ordering survives all of it but one place. One adjacent pair of rows changes places, and it is the pair the standard error had already flagged.
• And the clamped realistic family is a different question, not a finer answer to this one: it sits 1 to 52 per cent below the idealised family on every row.

The difference the word “sample” is hiding #

There are two entirely different things a set of test objects can be, and this collection’s own machinery has been treating one as though it were the other.

A sample is drawn from a population. Its mean estimates the population mean, the estimate is unbiased however few members it has, and the error falls as one over the square root of the count. More members is straightforwardly better, and the improvement is quantifiable without knowing anything about the population’s shape.

A quadrature rule approximates an integral over a region. Its accuracy depends on how well the rule’s points and weights match the function being integrated, it is biased in general, and — the part that matters here — more points does not guarantee less bias. A midpoint rule with a thousand points can be worse than Simpson’s rule with ten if the geometry is against it.

The census’s 125 surfaces are the second, and the file that builds them says so in the one place it matters and then not again. Its docstring explains that the set is a lattice rather than a random sample so that a figure does not quote a number carrying a seed — which is exactly right, and is the decision that makes it a quadrature rule. The consequence of that decision is that a standard error over it means something different, and nothing downstream was told.

What refining it actually does #

The lattice is a product: five brightness levels, seven modulation depths in one direction, seven in the other, minus everything whose two depths sum to more than 0.7.

Refine it uniformly — nine levels, thirteen depths each way, over the region whose three declared numbers decide almost everything — and something happens that is not more of the same. The L¹ constraint cuts a diamond out of the square of depths, and a diamond’s corners are where the most saturated surfaces are. A coarse grid puts a point exactly on each corner; a fine grid puts points near the corner but also many more points in the diamond’s interior, where the surfaces are milder. So the fraction of the set that is highly modulated falls as the grid is refined, and since the residual grows steeply with modulation, the mean falls with it — until the grid is fine enough that the corner region is itself resolved, at which point it rises again.

Read the first three rows of any column and the sequence goes down, then up. That is not a converging sequence and it was never going to be one, because the three rows are three different rules, not three sample sizes.

The table has the reference in it as well, which means the three rules can be scored rather than merely observed to disagree — and the scores are worse than “does not converge”.

The finest rule is the least accurate. Eight times the points and 2.2 times the error against the site’s own lattice — and worse, on average, than the 39-point rule it has twenty-five times as many points as.

That is not a sequence failing to converge. It is a sequence going the wrong way, and it is what a quadrature rule looks like when the geometry is against it rather than a rule that merely needs more work. A sample would have to be actively perverse to behave like this; a rule integrating a steeply-varying function over a constrained region does it as a matter of course, because refining the grid changes the effective weighting faster than it reduces the discretisation.

And the site’s own lattice comes out best of the three, which is luck. Nothing in the choice of five levels and seven depths was made with the region’s measure in mind — those are round numbers picked to give a set of convenient size — and they happen to sit where the diamond’s corners and the grid’s points fall into a favourable arrangement. A different round number would land somewhere else. So the right reading is not the collection chose well but the collection chose arbitrarily and the arbitrary choices span a two-per-cent band, and which one is best is not a property anybody controlled.

The signs are worth one more line, because they say the disagreement is structural. The coarse and fine rules err in the same direction on two of the three rows and the site’s lattice errs the other way on all three — which is the alternation a grid produces when the number of points along an axis changes parity relative to a boundary. It is a strong hint that these three numbers are three samples of an oscillation rather than three points on a trend, and that a fourth grid would land somewhere inside the same band rather than closer to the truth.

The reference the others are read against #

If the lattice is a quadrature rule, the thing it approximates has to be named before anybody can say how good it is, and naming it is the useful move.

The integral is over the region: every admissible triple of a brightness and two modulation depths, weighted uniformly. That is a declaration too — a uniform measure over those three parameters is not the only measure available, and nothing says real surfaces are uniformly distributed in them. But it is the region the lattice was already walking, so it is the right reference for asking whether the lattice walks it well, and it is estimable without bias by drawing points at random.

Two thousand uniform draws give the fourth row above. The generator is a linear congruential one with a fixed seed rather than the platform’s random source, for the same reason the lattice exists at all: a number quoted in an essay has to be the same number tomorrow.

Against that reference the lattice is out by:

Eleven of the fourteen rows are positive, so it is a bias rather than scatter: the lattice systematically reports a residual a per cent or two above the integral it stands for. Which is small, and is not the point. The point is that it was never measured, that no figure carries it, and that two per cent is the same size as several of the gaps the census table is asked to establish.

What survives, and the one thing that does not #

Set the levels aside and ask the question the census is actually for: which changes of light adaptation handles worst.

Under the lattice, the coarse lattice, the fine lattice and the uniform measure, the fourteen rows come out in the same order except in one place. Daylight-to-D40 and daylight-to-a-three-primary-display change places between the two coarser constructions and the two finer ones.

They are separated, on the lattice, by 0.0004 ΔE*₀₀ — four parts in ten thousand, on numbers printed to four figures as 0.9796 and 0.9800. The standard error on that gap is 0.034, so the ratio of the gap to its own error is 0.01, and the table was printing a difference that was never there.

That is the reassuring reading and it is worth stating as the finding: a ranking is far more robust to the choice of test set than the numbers in it are. A common-mode factor of two per cent shifts every row together and reorders nothing. The census’s conclusions — that a green wall bounced twice is the worst thing in it, that a blackbody at the same temperature as daylight is the mildest — are not statements about the lattice.

The fifth column, which is not the same question #

The clamped realistic family is in the table and is the odd member, and putting it there without saying so would be the mistake the whole phase is about.

It is 120 smooth curves of varying slope and curvature, clamped into the physical range, and it is the set the census’s own honesty check has always used. The clamp is the difference: a family that has to be clipped is not linear in its parameters, so a change of light is no longer exactly a matrix on it, and every row is answering a slightly different question.

The rows come out systematically lower — by 1.1 per cent on the blackbody row and by 51.7 per cent on the macular one. That direction was already asserted in the census’s own machinery, which requires that every row be smaller on the real surfaces than on the idealised ones and reports the census as an upper bound. What was not there is how large the gap gets, and the answer is that on the change of light with the sharpest spectral structure it is more than half.

And the clamped family reorders two further pairs, one of them separated by nearly nine standard errors. That is the boundary of what a standard error over a set can tell anybody, and it has an essay of its own.

Why nothing caught it #

Because the failure mode of a quadrature rule is a plausible number.

A biased sample announces itself eventually: draw more and the estimate moves toward the truth, and a stable estimate is evidence. A biased quadrature is stable at its own answer. Running the census a hundred times gives the same 0.4585 every time, which looks exactly like precision and is in fact the rule reproducing its own bias with perfect fidelity.

The one instrument that would have caught it — evaluating the same integral by a second rule — is the one nobody runs on a number they are confident about. The site has had two families in the same file since the round that built the census: the idealised one and the clamped realistic one, with an assertion that they agree in ordering. The assertion compares orderings. It has never compared levels, because the levels were expected to differ and the difference was the point of having two families. So the two-rule check existed, was checked whenever the collection was rebuilt, and was pointed at a different question.

What to do with it #

Three things, and the third is the one worth carrying to another subject.

Publish the ratio, not the level, wherever the argument allows it, which is the same conclusion the worst-case ratios reach from the other end. A residual quoted as this change of light is 3.6 times worse than that one carries almost none of this uncertainty, because a common-mode factor cancels. A residual quoted as 1.635 ΔE*₀₀ carries all of it. Most of what this collection actually argues is comparative and can be written that way.

Where a level is genuinely wanted, quote it against the uniform measure, because that is the estimate with no rule inside it. The lattice remains the right thing to draw figures from, since it is deterministic and a figure must not carry a seed.

And say which set. A residual with no set beside it is a number whose meaning depends on a file the reader has not read. That is the convention every figure in this family now follows, and it costs one line of caption.

There is a fourth thing not to do, and the scored table above is the argument for it. Do not refine the lattice. The instinct when a number is suspected of being coarse is to compute it on more points, and here that instinct costs eight times the work and returns an answer twice as far from the integral. If a level is wanted more accurately the route is a different rule — the uniform draw, which is unbiased and needs two thousand cheap evaluations — and not a denser version of the rule already in place.

That is the transportable half of this essay. A refinement is only an improvement for a rule whose error is dominated by discretisation, and a rule whose points are placed by a constraint has a second error term that refinement moves rather than shrinks. Which term dominates is not visible from inside a single rule, and the only way to find out is to evaluate the same integral a second way — which is the check the collection had and had pointed elsewhere.

Where the model stops #

None of this says the uniform measure is correct. It says it is unbiased for the region, and the region is itself a declaration — three parameters with two ranges and a constraint, none of them measured. A uniform density over those three parameters is a modelling choice exactly like the lattice was, and a reader who thinks real surfaces are concentrated near low saturation would want a different weight and would get different numbers.

What the exercise does establish, and what does not depend on the choice — as the elasticity table shows from a third direction — is the size of the sensitivity: several per cent on the level, essentially nothing on the order, and one adjacency that was never real. That much is a property of the census rather than of anybody’s preferred measure.

Who found it, and when #

The distinction is Gauss’s, and it is the whole reason quadrature has a literature: the accuracy of a rule depends on matching it to the function, and adding points to a badly matched rule is the least efficient repair available. The specific trap here — a lattice inside a constrained region, whose corners are where its answer lives, refined uniformly, whose effective weighting shifts as it refines — is familiar to anybody who has integrated over a simplex, and it is the reason such integrals are done by transformation or by Monte Carlo rather than on a grid.

Its arrival here follows an essay about extrema two rounds ago, which established that a maximum over a sample is a lower bound and a mean is safe. That is true, and it is a statement about sampling. The set here is not sampled, and the property a mean has under sampling is not the property it has under quadrature — which is a distinction that essay had no reason to draw and this one exists to.

Where the ladder goes next #

The levels move by a few per cent and the order barely moves at all, which raises the question the next rung answers: how much of the ordering is actually established. Some of the census’s steps are twenty standard errors wide and some are a hundredth of one, and telling them apart needs the error on a gap rather than the error on a row.