A linear repair for a bilinear loss

Two slits are not one slit found that a slit on a lamp’s table and a slit on a sample’s each make their own table honest on a coarse grid, and that the colour computed from the two is still wrong wherever a narrow feature in one meets a narrow feature in the other. The exact statement was that the average of a product is the product of the averages plus a covariance, and that separately blurred tables discard exactly that covariance. It ended on the obvious repair: the Stearns correction sharpens a table blurred by a triangular slit of one interval, it is applied to spectrophotometer data as a matter of routine, and whether applying it to both tables recovers the product is the same calculation with the correction inserted.

Four fifths of it, and a floor that is not a blur #

The correction recovers most of what two separate slits cost and cannot touch the rest, because what is left is not a blur. A three-term filter is linear; the covariance it would have to restore is bilinear in the two factors; and a linear operator applied to each factor separately produces no bilinear term.

• Two separate slits leave the notch’s colour 0.727 colour differences wrong. Sharpening both tables takes it to 0.188 — a factor of 3.9 better.
• One slit on the light the sample actually reflects leaves 0.019. The corrected pair is ten times further from the truth than that.
• It falls short at 30 of 33 positions of the notch across the line, by up to a factor of 11.
• Correcting both is not the average of correcting each. Both together give 0.188; the lamp alone 0.478 and the sample alone 0.464, whose mean is 0.471.
• On a smooth lamp the correction does close the gap and pass it, reaching 0.0019 against 0.0051 for one slit on the product — because there is no covariance there to lose.

Where the correction does what it says #

The correction was published for a case that is not this one, and it is worth watching it work there first.

Point sampling costs 2.14 colour differences and a pair of slits 0.0049, which the correction takes to 0.0019. The correction is doing exactly what it is for: a slit blurred a table, the blur is a known linear operation, and its inverse is a three-term filter that undoes most of it.

That case is the majority of colorimetry, and the slit is what makes it legal is the essay that established it. A neutral has no grid established that a factor with no structure has nothing for any operation on the other to disagree with, and a smooth lamp is flat across five nanometres everywhere — so the two blurs and the one blur of the product are the same thing, and any repair to one table is a repair to the calculation.

And the corrected pair beats one slit on the product there, 0.0019 against 0.0051, which is the honest form of the claim. A table sharpened back towards its unblurred shape is a better table than a blurred one when there is nothing else going on.

The floor, and what it is made of #

On a notch sitting on a mercury line there is something else going on, and the correction runs into it.

The positions are the ones where the grid starts showed a point-sampled answer swinging wildly across, and the correction inherits none of that swing. The sharpened pair lies below the blurred pair at every position — the correction never makes anything worse here — and above one slit on the product at 30 of the 33 positions, by a factor of up to 11. The gap is largest where the notch’s edges cross the line, which is exactly where the covariance the earlier essay identified is largest.

The reason is a statement about operators rather than about spectra. The quantity that was lost is ⟨L · S⟩ − ⟨L⟩ · ⟨S⟩, the covariance of the lamp and the sample inside the slit’s window, and it is bilinear: it is linear in the lamp with the sample held and linear in the sample with the lamp held, and it is not linear in the pair. The correction is a convolution — a weighted sum of three neighbouring table entries — and a convolution applied to ⟨L⟩ produces a better estimate of L and nothing that depends on S. There is no arrangement of linear filters applied to the two tables separately whose output contains the product of the two structures, because the output of a linear filter on a table is a linear functional of that table alone.

So the residual is not a matter of the correction being approximate. A three-term filter is not an approximation to the missing term; it is the wrong kind of object.

Why the improvement is as large as it is #

Four fifths is a lot to recover from a repair that cannot reach the residual, and the reason is worth stating because it is what makes the result useful rather than merely negative.

The damage two separate slits do has two parts. The first is that each table is blurred, and a blurred table summed on a coarse grid is wrong in its own right — that is the part the correction inverts, and it is the larger part almost everywhere, because a slit five nanometres wide does a great deal to a notch twelve nanometres wide and rather little to their overlap. The second is the pairing error, which is the covariance, and it is the part that survives.

The proportions are not fixed. On a smooth lamp the second part is zero and the correction reaches the truth. On a twelve-nanometre notch on a line it is about a fifth of the total after correction — 0.188 remaining out of 0.727 — and on a two-nanometre notch it is almost all of it, 3.84 out of 4.46. So the correction’s apparent effectiveness falls as the case gets harder, which is the opposite of reassuring: the arrangement where it recovers most of the damage is the arrangement where there was least to recover.

That is the number to carry from this. A laboratory reporting that the correction improved its results by a factor of four is reporting something true about its easy cases. The statistic that would tell it about the hard ones is the ratio of what the correction leaves to what one slit on the product would leave — which needs the fine data the correction was adopted to avoid needing, and is the reason the improvement is the number that gets quoted.

The cross term, drawn #

There is a check on that argument that does not require any algebra, and it is worth making because the algebra is the whole of the claim.

If the two blurs damaged the colour independently, then correcting both would land at the average of correcting each alone — two independent errors, each halved when its own repair is applied.

Correcting the lamp alone gives 0.478 and the sample alone 0.464, whose mean is 0.471. Correcting both gives 0.188. The two repairs are not independent: together they reach much further than either suggests, because each of them is removing part of a damage that has both factors in it.

That is the cross term appearing as a measurement, and it says the same thing the algebra does from the other direction. A damage with a cross term in it responds to repairs with a cross term in them — and the part that does not respond at all is the part that is bilinear in the two structures rather than in the two repairs.

Every width of notch #

The floor is not a property of one notch, and the width sweep is where that is easiest to see.

The correction sits below the uncorrected pair at every width and above the one-slit product at every width. At forty-eight nanometres — a notch so broad it is smooth across five — everything agrees and there is nothing to correct. At one nanometre, which is what a notch filter actually is, the sharpened pair leaves 4.15 colour differences and one slit on the product leaves 0.033.

That is the practical statement. A laboratory that applies the correction to its spectrophotometer’s output, as many do, has improved its sample tables and has not made a notch filter’s computed colour trustworthy under a line lamp. The line can be in the sample is where the problem was first stated and its repair is still the only one that works: measure the light the sample actually returns, once, through one slit.

A narrow notch, and where the correction runs out #

The width sweep has one end worth taking on its own, because it is where a notch filter actually lives and where the residual stops being a decimal place.

At a notch two nanometres wide the numbers are no longer small. Point sampling is 4.96 colour differences, two separate slits 4.46, and the correction brings that to 3.84. One slit on the product is 0.012. The correction has done its usual job — it has removed the blur — and the blur was never most of the problem here: the two tables’ pairing error is, and it is three hundred times what a single slit leaves.

That is the width at which the arithmetic stops being a caution and becomes a wrong answer. A notch filter blocking a laser line is one to two nanometres wide, its whole purpose is to sit on a line, and a colour computed for it from two corrected tables is out by nearly four colour differences — which is not a subtle error in a filter’s colour but a different filter.

What a colour engine could do about it #

Three things, and the first is the one nobody does.

Record whether a table has been corrected. The Stearns correction is applied by some instruments and not others, sometimes as a setting and sometimes silently. A table that has been sharpened and a table that has not are different objects — the same kind of unrecorded difference what the instrument reports is about — and combining a corrected lamp table with an uncorrected sample table is a third arrangement that this essay has not measured and that nothing prevents.

Do not correct twice. A correction applied to an already-corrected table sharpens a spectrum that was not blurred, which introduces structure rather than removing it. The overshoot is small on a smooth table and is not small on a steep edge, and nothing in a table’s file says whether it has been through the filter once. A finer reading of a coarser table is the neighbouring case of a table read as though it held more than it does.

And keep the fine tables where both factors have structure. That is the recommendation the earlier essay made and this one narrows: a correction is not a substitute for it. A filter maker’s table at a nanometre and a lamp maker’s line list, multiplied and blurred once, reproduce the one-slit answer, which is the arrangement a grid is not a resolution argues for; a corrected pair of coarse tables does not, and is ten times further away.

How the correction was applied #

The correction is the published three-term deconvolution of a triangular slit of one interval — −0.083, 1.166, −0.083 — applied to the blurred table on the five-nanometre grid, with the two end entries left alone because the filter has no neighbours there. The corrected table is read back by linear interpolation, which is what a colour engine handed a corrected table would do.

The slit is triangular with a full width at half maximum of five nanometres, integrated at a tenth of a nanometre. The sample is a constant reflectance of 0.72 with a Gaussian notch 0.66 deep; the lamp is a fluorescent tube with mercury lines 1.2 nm wide on a phosphor bed. Every colour is summed on the five-nanometre grid with its own white and compared in ΔE₀₀ against the unblurred product summed at a tenth of a nanometre.

The six cases are the five the earlier essay measured with two more inserted: both tables sharpened, and each sharpened alone. The one-sided cases exist for the cross-term comparison and for nothing else, since no instrument corrects one table and not the other on purpose.

What this does not settle #

The correction modelled is the three-term one for a triangular slit whose width equals the sampling interval, which is the case it was published for and is the common instrument design. An instrument whose slit is wider than its interval needs a different filter, and a longer filter is still linear, so the argument transfers and the numbers do not.

The correction is applied on the coarse grid. An instrument that sharpened a finer internal table before writing out a coarse one would be a different arrangement, and a better one, because the sharpening would happen where the structure still is.

And nothing here measures what a correction does to a table it has already been applied to. That is the second recommendation above and it is stated rather than computed; the direction is clear and the size is not.

Still open: whether the residual is computable from the two tables alone #

The residual is a covariance, and a covariance is a quantity about two things. What is interesting is that a colour engine handed two blurred tables has more information about it than it uses.

The covariance inside a window is bounded by the two factors’ own variances inside that window, by the Cauchy–Schwarz inequality, and both of those variances are estimable from the blurred tables themselves: a table that has been blurred still carries how much structure it had, in how much the correction changes it. So an engine could compute an upper bound on the error it is making, for every pair of tables it is given, without any reference and without any fine data.

Whether that bound is tight enough to be useful is the question. A bound that says the colour might be wrong by up to four colour differences when it is in fact wrong by 0.02 is a bound nobody will act on; one that tracks the actual error within a factor of two or three would turn a silent failure into a flag. The calculation is the same one run here with the inequality in place of the truth, and it needs no new measurement of anything.

A repair has a shape, and so does the damage #

The habit is about matching the two.

A published correction comes with a statement of what it undoes — here, a convolution by a known kernel — and applying it to something else is a hopeful act. The useful question is not whether the correction is accurate but whether the damage has the same form as the thing the correction inverts: a linear operator inverts a linear operation, and applied to a loss of a different form it removes whatever part of the loss happens to be linear and leaves the rest.

The move is to decompose the damage before choosing the repair. Here the decomposition was already written down — the average of a product is the product of the averages plus a covariance — and the two terms have different forms, so a repair for one cannot be a repair for the other. That takes one line and it predicts the result.

The failure mode is to judge a repair by how much it improves things. Four fifths is a large improvement and it is the wrong statistic: what matters is whether the remaining fifth is the same kind of thing as the four, because if it is not, no amount of more of the same repair will reach it.