A corner is corrected by one row

The corner of the frame has another filter found that the infrared-cut filter’s edge slides nineteen nanometres towards the blue at the corner of a short lens, that a grey-card gain map corrects the grey exactly under its own lamp, and that it leaves coloured patches wrong even there, because a moved edge changes the shape of a channel and a gain can only change its size. It ended on the obvious next correction — a colour matrix per position, fitted to a chart rather than to a grey card — and on three questions about it: how much it removes, how many lamps it needs, and how many positions.

Three numbers do what nine do #

A correction with shape halves the corner’s colour error under its own lamp, the whole of that work is done by one row of the matrix, and how well the row travels to another lamp depends on which lamp it was fitted under.

• At twenty-five degrees under daylight a gain map leaves coloured patches a mean of 1.86 colour differences off; the red row leaves 0.93 and the full matrix 0.90. The matrix’s green and blue rows are the identity to within 2.5 per cent.
• Under each of four lamps the row leaves between 44 and 65 per cent of what the gain map leaves.
• Fitted under one lamp and used under another, the row beats the gain map on the grey in eleven of twelve cases, by up to eighteen times — and in the twelfth, fitted under tungsten and used in daylight, it leaves the grey 9.17 off against the gain map’s 5.46 and one patch 17.7 off.
• Off the chart it was fitted on, under tungsten, the row loses to the gain map: 3.25 against 2.32, with a worst surface of 11.6.

Why a matrix should help #

A gain map multiplies each of the corner’s three raw channels by one number, which makes one colour exact — the grey card it was measured on — and treats every other colour as though the corner’s channels were the centre’s, scaled. They are not. The moved edge of the filter that makes colour possible has cut away the deepest reds from the red channel’s sensitivity, and a red patch puts more of its light in that band than a grey does, so a red patch has lost a larger share of its red reading than the grey card lost.

A matrix can partly repair that, because the other two channels carry information about the missing band. A surface that reflects strongly in the deep red usually reflects less in the green, and a surface whose red reading has dropped at the corner can be recognised as such by its green and blue readings. A three-by-three matrix taking the corner’s raw values to the centre’s, fitted on a chart photographed at the corner, is the best linear use of that information, and it can be made to keep the grey exact by scaling its rows.

At twenty-five degrees under daylight it does what the reasoning suggests. The mean error on coloured patches falls from 1.86 to 0.90 and the worst from 3.65 to 2.31. At thirty-five degrees, the steepest corner of a phone lens, the mean falls from 3.81 to 1.81. Under tungsten at twenty-five degrees the fall is from 1.52 to 0.91; under the white LED and the fluorescent tube, from about 0.8 to under 0.4.

Six of the nine numbers are the identity #

The matrix the fit returns is not a general three-by-three. Under daylight its green row takes green to green with a coefficient of 0.98 and borrows under two per cent from red, and its blue row is blue to four decimal places. Under tungsten the green row borrows 2.4 per cent from red and 2.5 from blue; under the LED and the tube, around one per cent.

That is the physics arriving in the fitted numbers. The moved edge sits in the red: at twenty-five degrees it takes a tenth of the red channel’s response to daylight and a sixth of its response to tungsten, the green channel loses about one per cent and the blue nothing at all. A correction for a change confined to one channel is confined to that channel’s row, and a fit given all nine numbers spends six of them reproducing the identity.

So the correction worth holding is three numbers: the red row, with the green and blue channels left to their gains from the grey card. Fitted that way it leaves 0.93 under daylight against the full matrix’s 0.90, and its worst patch is better, 2.12 against 2.31, because six free coefficients were fitting the chart’s own quirks rather than the edge. At every angle from ten degrees out, the row and the matrix are within a fifth of each other.

What the row rebuilds #

The row predicts the centre’s red reading from the corner’s red, green and blue readings, and its three numbers say how it does it.

Under daylight the row is 1.33 times the corner’s red, minus 0.33 of its green, plus 0.16 of its blue. The red coefficient above one puts back the lost share. The negative green coefficient takes some of it away again for surfaces bright in green, which lost less of their red than their red reading suggests, and the small blue term adjusts for how much of a surface’s light sits in the short end. Under tungsten the same row is 1.49, minus 0.67 and plus 0.63, and under the LED and the tube about 1.16, minus 0.17 and plus 0.13.

Two things set the size of those coefficients, and both belong to the lamp. The first is how much the corner lost: tungsten, with most of its power towards the red, loses 15.3 per cent of its red reading at twenty-five degrees, daylight 10.1 and the LED-type sources about 4. The second is how much the chart’s readings vary under that lamp. Under tungsten the chart’s blue readings spread only 0.23 as far as its red ones, against 0.78 under daylight, because a tungsten lamp puts little light in the blue for any surface to vary. A least-squares fit with most to rebuild and little spread in one input to rebuild it from will lean on that input with a large coefficient, because a large coefficient on a quantity that hardly varies costs the fit little on the chart. Least squares prices a coefficient only by what it does to the patches in front of it — the same objective the objective nobody chose found under every camera profile — and nothing in that price looks at a lamp the chart was not photographed under.

Where the residual goes #

A row corrects the patches whose error the gain map left, and leaves others.

Under daylight, with a row fitted under daylight, the patches are a mean of 0.93 off and a worst of 2.1, and the worst lies towards the red end of the peak-wavelength order — a matrix can use green and blue to estimate a red loss, and cannot recover the shape of a band that was never measured, so a surface whose reflectance rises most steeply through the lost band is still the hardest. Under tungsten, with the same daylight row, the mean is 1.18 and the worst 2.5, and the worst has moved to the blue end: the row’s blue coefficient, fitted for daylight’s balance of light, is now adding red to surfaces whose blue reading a tungsten lamp supplies differently.

That is still better than the gain map, which leaves 1.87 under tungsten with a worst of 4.7. No matrix is right everywhere, and a matrix confined to a row is no exception; it moves the error rather than removing all of it, and where it moves the error is decided by the lamp.

Carried to another lamp #

A converter photographs its calibration chart under a few lamps and meets many. Every cell of the table below is a row fitted under one lamp and used under another.

In eleven of the twelve cells off the diagonal the row leaves the grey closer than a gain map fitted under the same lamp. Some of the improvements are large. A row fitted under tungsten and used under the white LED leaves the grey 0.44 off, where a tungsten gain map leaves 7.98. A daylight row used under tungsten leaves 1.03 where the daylight gain map leaves 2.63. On the coloured patches the count is the same, eleven of twelve.

The twelfth cell is the one that matters most in practice, because it is the ordinary case of a camera calibrated indoors and taken outside. A row fitted under tungsten and used in daylight leaves the grey 9.17 colour differences off, against the gain map’s 5.46, and its worst coloured patch 17.7 off, against 5.5. The row’s blue coefficient, 0.63, was cheap on a tungsten chart whose blue readings hardly varied. In daylight, which supplies three times as much blue relative to red, the same coefficient adds red that was never lost.

The asymmetry is the one a matrix is fitted under one light found for a camera’s whole colour matrix, and it has the same cause: a least-squares fit is decided by the light it was given, and a light that starves one input decides that input’s coefficient badly. Here it arrives in a correction whose job was only to undo a filter.

Dividing the lamp out makes it worse #

The coefficient that fails looks like a matter of units. A number that turns a blue reading into a red one has units of red counts per blue count, and the ratio of a lamp’s red counts to its blue counts is the lamp’s colour — so a row fitted under tungsten seems to carry tungsten’s colour inside its blue coefficient. The obvious repair is to fit the row on balanced values instead, each channel divided by the frame’s own response to its lamp, which a converter has already computed for its white balance.

Under its own lamp the balanced row is the raw row exactly, because rescaling a least-squares fit’s inputs and target by constants changes its coefficients and not its predictions. Under every other lamp it is worse than a gain map — in all twelve cells. A daylight row used under tungsten goes from 1.03 to 3.14 on the grey; a tungsten row used under the white LED from 0.44 to 10.02. The one cell it improves, the tungsten row in daylight, it improves from 9.17 to 6.65, which is still above the gain map.

So the blue coefficient is not a statement about units. It is a statement about how much of the lost band’s light arrives with how much blue under that particular spectrum, and dividing by the white balance treats a spectral relationship as though it were a scale. The row belongs to the lamp it was fitted under, as surely as the chart decides the profile, and no rescaling of the readings makes it belong to another.

Off the chart #

A correction fitted on twenty-four patches can be excellent on those patches and poor elsewhere. The test is a second set of surfaces the fit never saw.

Under daylight the row leaves the held-out surfaces a mean of 1.94 off against the gain map’s 2.23, and under the LED and the tube 1.16 and 1.06 against 1.25 and 1.21. Under tungsten it leaves 3.25 against 2.32, and its worst surface is 11.6 against 5.2 — on the chart it was fitted to, the same row under the same lamp leaves 0.99.

That is the tungsten row’s large coefficients showing themselves without any change of lamp. The held-out surfaces are more saturated than the chart’s — and a chart’s saturation decides what a camera scores for a whole profile as well as for a corner — so they supply the blue variation the chart did not, and the coefficient the chart could not constrain meets surfaces that exercise it. A tungsten chart leaves the row’s blue coefficient nearly unidentified, and the residual on the chart says nothing about it — the gap a fit can be exact and empty names, between how well a fit matches its data and how much of the fit the data decided. The gain map has three coefficients that each see only their own channel, and has nothing to overfit.

How many positions #

A chart photographed at every position across a frame is not a calibration anybody performs. A grey card photographed flat across the whole frame is — it is the gain map — and it gives the size of the correction at every pixel for free. The question is whether the shape can be measured at a few positions and carried between them.

With the row fitted at the centre, at 17.5 degrees and at 35 degrees, interpolated linearly between them and rescaled at each position by the grey map, the patches at 22.5 degrees are 0.86 off — against 0.74 for a row fitted at that position and 1.49 for the gain map. At 27.5 degrees the three are 1.27, 1.14 and 2.28. Near the centre the interpolation is slightly worse than a gain map, 0.10 against 0.02 at two and a half degrees, at sizes nobody could see. With five rings the interpolated rows are within three hundredths of a row at every position.

The grey map is not optional. Interpolated between three rings without it, the same rows leave the grey itself 4.7 off at 27.5 degrees, because the shift grows as the square of the sine of the angle and a straight line between rings misses the curve’s size even where it gets its shape. With only two rings, at the centre and the corner, the grey is 10.1 off at 17.5 degrees without the map, and even with the map the patches there are 1.21 against the gain map’s 0.88. Shape from a few rings, size from every pixel, is the combination that works, and either half alone is worse than the gain map it was meant to improve on.

A shape costs noise that a size does not #

A correction multiplies the noise as well as the signal, and a row multiplies more of it than a gain does.

A gain map multiplies the corner’s red reading by one number, so it multiplies the red reading’s noise by the same number: at twenty-five degrees, 1.11 under daylight and 1.18 under tungsten. A row adds the other two channels’ readings into the red one, and their noise comes with them. With the read noise the same size in counts in all three channels, the corner’s red noise after the row is the square root of the sum of the squared coefficients times the noise — 1.38 under daylight and 1.75 under tungsten, a quarter and a half again what the gain map costs. Under the LED-type sources, with their small coefficients, the row costs 1.19 against 1.04. At thirty-five degrees under daylight it is 2.06 against 1.26.

So the improvement in colour is paid for in the corner’s noise, and paid most under the lamp whose row leans hardest on the other channels — the arithmetic correcting colour costs noise measured for a whole colour matrix, arriving again in a correction three numbers long. At the corner of a phone frame, where the light is weakest because the lens vignettes it and the gain map has already raised the noise, the row is a trade rather than a free repair: half the colour error for a quarter to a half more noise in one channel.

What a converter would hold #

The calibration the arithmetic supports is modest, and one part of it is a prohibition.

A dense grey map per lamp family, which phone camera pipelines already hold as illuminant-dependent lens-shading tables. It carries the size of the correction at every position.

A red row at three to five rings, per lamp family, fitted under that family’s own light. Three numbers at each ring, a dozen positions at most. The rows for the LED and the fluorescent tube are nearly interchangeable — the tube’s row under the LED leaves the grey 0.07 off — and daylight and tungsten each need their own.

Never a tungsten row in daylight. Where a converter must fall back to a row from another lamp, a row fitted under daylight carries to every other lamp better than a gain map does; a row fitted under tungsten carries well to the LED sources and badly to daylight. If only one row can be calibrated, it should be calibrated in daylight.

And a lamp estimate good enough to choose between them, which is the same dependence the gain maps already had, now with a larger price on getting it wrong in one direction. The highlight is the white balance when a scene has a highlight, and an image does not determine the light when it does not.

How the corrections were computed #

The sensor is the modelled silicon sensor of the essay on the corner filter, with the infrared-cut filter’s edge moved by the interference condition at each angle and nothing else changed. The chart is twenty-four constructed coloured surfaces and six neutrals; the held-out set is twenty-four more saturated surfaces with narrower bands.

Each correction takes the corner’s raw values to the centre’s under the calibration lamp. The gain map is the ratio of the centre’s to the corner’s response to an eighteen per cent grey card. The full matrix is the least-squares three-by-three over the chart; the red row is the least-squares fit of the centre’s red reading on the corner’s three readings, with the green and blue channels given the grey card’s gains. Each is scaled so the grey card comes out exactly under the calibration lamp. The balanced row divides each raw channel by the centre’s response to the lamp before fitting and multiplies back by the response to the lamp in use afterwards. Errors are ΔE₀₀ between the centre’s and the corrected corner’s rendering through the converter’s pipeline for the lamp in use. Interpolation between rings is linear in the angle, entry by entry.

What the model does not contain #

The filter is moved rigidly and every photosite at an angle sees a single chief ray; a real lens delivers a cone of angles, and a real stack changes its shoulder and splits its edge between polarisations. Each would change the row’s coefficients and none would move the change out of the red channel, which is what makes it a row.

The chart and the held-out set are constructions, and the held-out set’s greater saturation is a choice. A measured chart and a measured library would give their own numbers, and the tungsten row’s failure off the chart is exactly the kind of result whose size depends on how far the test surfaces differ from the chart.

And the only departure across the frame is the filter’s. Microlens efficiency, pixel crosstalk and vignetting also vary with position; a gain map absorbs whatever part of them is a scaling, and whatever part is a change of shape would add to what the row has to rebuild.

The habit: a fit’s free parameters show what the defect touched #

A correction fitted with more freedom than the defect needs will spend the extra freedom, and where it spends it is information. A full matrix that returns the identity in two rows is saying the defect lives in the third; a coefficient that grows large on an input the calibration barely varied is saying that input was not measured.

The move is to fit the general correction once, read which parameters moved, and hold only those — then test what is held on data the fit did not see, under conditions the fit did not see.

The failure mode is to trust the reduction in error on the chart. A correction that halves the error on its own chart has told nothing about the chart’s absent colours or the calibration’s absent lamps, and for the corner of a frame the absent lamp is the daylight most photographs are taken in.

Where this comes from #

Illuminant-dependent lens-shading correction is a standard part of mobile camera pipelines, and colour-dependent shading from the angle-dependence of interference filters is well documented in the design of small camera modules. Least-squares colour correction matrices and their dependence on the calibration illuminant and training set are textbook colour-imaging practice.

That the per-position correction reduces to one row, that its transfer between lamps is asymmetric with the tungsten-to-daylight case failing badly, that balancing the row makes every transfer worse, and that three rings with a grey map approach a correction at every position, are computed here on one modelled sensor.

Still open: a correction fitted under every lamp at once #

Each row here was fitted under one lamp. A row fitted on charts photographed under several lamps together would trade some accuracy under each for robustness under all, and the tungsten row’s failure suggests where the trade lies: a daylight chart in the fit set would pin the blue coefficient that a tungsten chart leaves free. How much accuracy under each lamp that costs, and whether a single pooled row per position could replace the per-lamp rows a converter otherwise has to choose between with an uncertain lamp estimate, is the same least-squares fit on a larger set of patches, with the grey held under a stated lamp rather than under the lamp in use — the condition that decided every result above.