A sensor designed for its inverse

A display’s adaptation basis is three chromaticities and it holds still. A camera’s is its three dyes and the light in the room, because the illuminant appears twice in the expression and does not cancel. So the same design question — choose the axes for how well a gain works in them — has a different shape for a sensor, and the difference is the whole of what distinguishes the two devices.

The claim #

Three Gaussian dyes chosen to make their own inverse a good adaptation basis reach 0.974 ΔE00 — the best figure any basis achieves at all — against 1.62 for the silicon-and-filter-array sensor this collection models. The Luther constraint that keeps the result a camera costs almost nothing. And the basis so obtained swings by six degrees across the illumination census, because it is a property of the room as well as of the dyes.

• The designed dyes sit at 610, 542 and 449 nm with widths of 35, 26 and 30 nm.
• They are narrower than a real colour-filter array’s, which is what sharpening looks like when a search does it.
• Held to a Luther residual of 0.16 rather than 0.28, the adaptation figure moves from 0.974 to 0.977. The constraint is nearly free.
• The basis drifts 6.1° at worst, against a display’s zero and a real sensor’s 16.6°.
• So the design works and does not hold, which is the sentence that separates the two devices.

What a camera’s basis is #

White balance is a per-channel gain on raw values, so it is a von Kries adaptation, and the basis it acts in is whatever carries tristimulus values to raw. That map is S B(E)⁻¹, where S is the sensor’s response to a reflectance basis under the illuminant E and B(E) is the standard observer’s response to the same reflectances under the same light.

The illuminant is in that expression twice and does not cancel. The eye’s basis is a property of three pigments; a camera’s is a property of three dyes and the light. That is an established result on this site and it is the reason the design question here is not simply the display question with different hardware.

The exception is a sensor satisfying the Luther condition, for which the light does cancel exactly — and what that condition does and does not fix turns out to be the neighbouring essay’s subject rather than this one’s.

The design space and the constraint #

The dyes are modelled as three Gaussians in wavelength with free centres and widths, cut to zero beyond 700 nm, which is the infrared filter every camera has stated rather than modelled. Six parameters, matching the display problem’s six exactly so that the two answers are comparable.

The constraint is the camera’s version of a gamut floor. Without one the search walks straight to three narrow lines, which adapt superbly and cannot measure a colour: a sensor whose channels are three spikes reports almost nothing about a spectrum, so every metamer of the eye becomes a different colour to it and every non-metamer may become the same one. The condition that keeps the answer a camera is Luther’s — that the sensitivities be a linear combination of the matching functions — and the constraint here is a ceiling on how far the fitted combination misses.

What the design finds #

At a Luther ceiling of 0.28 — a little tighter than the silicon sensor’s own 0.32 — the search returns dyes at 610, 542 and 449 nanometres with widths of 35, 26 and 30, leaving a Luther residual of 0.228 and an adaptation figure of 0.974.

That figure is not merely good. It is, to four decimal places, the same as the unconstrained optimum over all nine numbers of a basis. Three physical dyes reach the best adaptation basis there is, which is the same finding the display problem produced — the good bases are all realisable — arrived at through a completely different parametrisation.

Tightening the ceiling barely moves it. At 0.20 the design leaves 0.975; at 0.16 it leaves 0.977. So over the range where a sensor is recognisably colorimetric, adaptation quality is nearly independent of how colorimetric it is — which is a stronger statement than the display case’s, because the two properties here were expected to fight.

What the designed dyes look like, and what a real one looks like #

The two sets differ in a way that is easy to describe and instructive.

A real colour-filter array’s channels are broad, heavily overlapping and asymmetric. The green channel spans most of the visible band; red and green overlap through the whole of the yellow region; and blue has a long tail into the green. That shape is not an oversight — it is what maximises photon capture, which is what a sensor’s noise performance is made of, and it is also roughly what the eye does.

The designed set is narrower everywhere: 35, 26 and 30 nanometres of standard deviation against channels that are two to three times that. It puts its long-wave dye at 610 nm rather than out near 650, which pulls it away from the region where the eye has almost no sensitivity, and it separates the middle and long channels by 68 nm where the cones separate theirs by about 30.

The pattern is a caricature of a sharpened basis, and that is what it should be: the objective rewards spectral independence and nothing in it rewards catching photons.

That is the trade the Luther constraint is standing in for, imperfectly. The constraint keeps the sensor able to measure colour; what it does not do is keep the sensor able to measure it in reasonable light, because nothing about the Luther condition mentions photons.

What the constraint does and does not bind #

The most surprising number in the essay is that tightening the Luther ceiling from 0.28 to 0.16 costs three thousandths of a ΔE00.

The natural expectation is a fight. Being colorimetric means the channels are a linear image of the matching functions, which are broad and overlapping; adapting well means the channels are narrow and independent; those sound like opposite requirements, and this collection has said so in print.

They are not opposite, and the reason is the linear image part. A sensor’s basis is S B(E)⁻¹, and a linear recombination of the channels changes S and changes the basis with it. So the family of Luther-satisfying sensors is not one sensor with one basis; it is a family whose basis ranges over everything, which is the neighbouring essay’s whole subject.

Once that is seen, the near-freeness of the constraint here stops being surprising and becomes an instance of it. The search is walking in a six-parameter family that contains, close by, sensors whose spectral shapes are nearly a linear image of the matching functions and whose basis is sharpened — because the two properties are about different aspects of the same three curves.

What the design costs, in the two currencies it does not count #

The objective rewards spectral independence and nothing else. Two things it does not count are computable from the same curves.

Light. Integrating each sensor’s three channels against D65 and summing, the designed dyes capture 0.607 of what the silicon sensor captures — thirty-nine per cent of the photons gone, or 0.72 of a stop. The colorimetric sensor captures 1.229 of it, twenty-three per cent more than silicon. So the narrowness that buys the adaptation figure costs about three quarters of a stop of exposure, which is a factor of 1.28 on the shot noise in every reading.

That is the trade the sentence about photon capture names, priced. It is a real cost and not a fatal one: three quarters of a stop is a smaller penalty than most people would guess for channels two to three times narrower, because a narrow channel gives up breadth and keeps its height.

The other objective. The designed sensor’s basis is, to four decimal places, the basis that minimises the adaptation residual over all nine free numbers — which is the headline above. It therefore inherits that basis’s other property: an ellipse anisotropy of 7.697, against the silicon sensor’s 2.457 and the colorimetric sensor’s 3.710.

So the three sensors rank one way on adaptation — designed 0.974, silicon 1.622, colorimetric 2.457 — and very nearly the other way on discrimination: silicon 2.457, colorimetric 3.710, designed 7.697. The design that reaches the best adaptation basis available also reaches the worst discrimination geometry of the three, by a factor of three over an ordinary camera.

That is not a defect in the search. It is the search doing exactly what it was told. Two objectives over the same nine numbers point apart, and a design that optimises one lands where the other collapses — the same result the eight-matrix scatter reports, arriving here through three dye centres and three widths instead of nine free coefficients.

The practical reading is that this answer is right about its own question and is not a recommendation. A sensor built to these dyes would white-balance better than anything on the market, lose three quarters of a stop, and be the worst instrument in this collection for anything that turns on differences between colours being commensurable.

And then the room moves it #

A display’s primaries are its axes whatever the room is lit by. A camera’s are not, and the designed sensor is no exception.

The designed sensor’s basis swings up to 6.1 degrees across the census; the silicon sensor’s swings up to 16.6. So the design more than halves the drift as well as the residual, which is a genuine improvement — and it does not remove it, because nothing short of the Luther condition can.

What that means in practice is that the 0.974 is a figure at the light the design was performed under. Under a triphosphor tube the basis is a different basis, and its residual is whatever that different basis delivers. A display’s axes are a decision; a dye-filter camera’s are a decision and a hostage.

The two devices side by side #

The display and the camera versions of this design problem are worth setting out together, because they have the same dimension, the same objective and the same near-free constraint, and three different answers.

The display cannot quite reach the optimum because its rows must be duals of three realisable chromaticities; the camera can, because its curves have more freedom than three points do. And the third column — a sensor whose sensitivities are exactly a linear image of the matching functions — reaches it and holds still, which is the arrangement that ought to be impossible and is not.

What was computed, and how #

The sensor’s response to the reflectance basis is integrated on this collection’s five-nanometre grid, extended into the infrared for the dye model and truncated by the cut filter. The camera basis is S B(E)⁻¹ with E the reference context, which is the construction the essay on a camera’s own basis already used.

The Luther residual is a least-squares fit of the three sensitivities to the three matching functions, reported as a relative error. It is the measurement Luther’s condition is stated as here and is likewise not new.

The search is Nelder–Mead over three centres and three widths with a soft penalty on the Luther excess, started from a plausible dye set. The budget is checked: the answer is identical to four decimals from two hundred steps upwards.

The drift is measured as the worst principal angle between the basis at the reference light and the basis at each census light, matched row to row — the same measurement applied to the modelled sensor, so the two numbers are directly comparable.

What it would take to build one #

The design is a set of three curves and the question of whether anything could have them is worth a paragraph, because the answer is nearly.

Interference filters routinely achieve pass bands of twenty to forty nanometres, which is the width the design asks for, and multispectral cameras use exactly that — filter wheels or patterned dichroic arrays with narrow bands. What they give up is light, and they give up a great deal of it: a filter passing a 26-nanometre band out of a 300-nanometre visible range transmits under a tenth of what a broad dye does, so the sensor is two to three stops slower before anything else is considered.

For a photographic camera that is disqualifying. For an instrument it is not, and the design here is closer to a description of a colorimeter’s front end than of a camera’s. A tristimulus colorimeter’s filters are broad because they are trying to match the observer; an instrument designed for this objective instead would be a different device with a different purpose.

Which points at where the result is actually useful. It is not a proposal for a sensor. It is a measurement of what a camera’s white balance gives up by having the dyes it has — a factor of 1.7 in residual and a factor of 2.7 in basis drift — and that number belongs in the same list as the noise cost of a colour matrix and the error in guessing the illuminant, which are the other two terms in the same budget.

Where the model stops #

Gaussian dyes are not dyes. A real colour-filter array is a dye transmittance times a silicon quantum efficiency, with tails, side lobes and an infrared response that has to be filtered. The Gaussian family exists here to give the search the same number of parameters as the display problem, not to be manufacturable.

Narrow dyes cost light. A filter that passes a 26-nanometre band throws away most of the photons that reach it, and the noise consequence is real and is not in this arithmetic. Correcting colour costs noise is the essay about the matrix half of that trade; the filter half is larger and is not modelled here at all.

And a camera does more than balance. Demosaicing, tone mapping, the profile fitted to a chart and the estimation of the white in the first place are all downstream of these three curves, and estimating the illuminant is a bigger error than anything measured here.

Why the drift is halved and not removed #

Six degrees against sixteen is a large improvement and it has a mechanism worth stating, because it is the same one that produced the improvement in the residual.

The basis is S B(E)⁻¹, and how much it moves when E changes depends on how much the ratio of the sensor’s response to the observer’s response varies across the spectrum. If the two sets of curves had the same shape the ratio would be constant, the light would cancel, and the basis would not move at all — which is the Luther condition restated.

A real sensor’s curves differ from the matching functions in a complicated way, with the largest disagreements in the short wavelengths where x̄ has a second lobe no filter has. So a change of light that redistributes energy between the two regions where the disagreement is worst moves the basis a long way. The census’s worst rows for drift are the narrowband ones for exactly that reason: a line lands in one region and not the other.

The designed sensor’s curves are closer to being a linear image of the matching functions — that is what the Luther ceiling is enforcing — so the disagreement is smaller and the swing is smaller. Halving it is what a residual of 0.23 rather than 0.32 buys.

Removing it entirely needs the residual to be zero, and a Gaussian is not a linear combination of the matching functions for any centre or width. That is not a limitation of the search; it is what Luther’s condition says, and no filter with a non-negative transmittance can satisfy it, because the matching functions go negative in the middle of the construction.

The generalisation #

When two devices implement the same operation, the interesting difference is which of its parameters are fixed by the hardware and which by the situation.

A display and a camera both apply a per-channel gain, and both have a basis. The display’s is three chromaticities in a specification; the camera’s is three dyes convolved with whatever is illuminating the scene. Every conclusion that transfers between the two devices transfers because of the shared operation, and every conclusion that does not fails because of that one difference.

It is worth having as a habit because the shared operation is what makes the analogy tempting. The design problem posed here is the display’s problem with the same objective, the same dimension and the same near-freeness of the constraint — and a different answer about whether the answer stays put.

Where the ladder goes next #

The one sensor whose basis does hold still is the one that satisfies Luther’s condition exactly, and this collection has been saying for some time that such a sensor adapts badly. It does not, and the correction is the next rung.