The corner of the frame has another filter

The filter that makes colour possible is the infrared-cut filter in front of every colour camera’s sensor. Without it the three channels collapse towards one another under a tungsten lamp, because silicon keeps responding to light long after the dyes have stopped distinguishing it. That filter is almost never a dye. It is an interference stack — a sequence of thin dielectric layers whose reflection band is decided by their thicknesses — and an interference stack is a different filter for light arriving at a different angle.

A corner is a different sensor #

At the corner of a frame the infrared-cut filter’s edge has moved, the red channel has lost part of its response, and no single gain map corrects it for more than one lamp.

• The filter’s half-transmission edge moves from 665 nm at the centre to 658 at 15 degrees, 646 at 25 and 630 at 35.
• At 25 degrees the corner’s red channel loses 10.1 per cent of its response to daylight, 15.3 to tungsten and 4.1 to a white LED; the green channel loses about one per cent and the blue none.
• A gain map made from a grey card under daylight leaves that grey exactly right under daylight at every angle. Under tungsten the same grey at 25 degrees is 2.6 colour differences off, under a white LED 4.2 and under a fluorescent tube 4.0.
• Under its own lamp the map still leaves coloured patches wrong: a mean of 1.9 colour differences at 25 degrees under daylight and a worst of 3.7, because moving an edge changes the shape of a channel and a gain can only change its size.

Why the edge moves #

An interference filter reflects the wavelengths for which light reflected at its many layer boundaries arrives back in step. The condition depends on the optical path through each layer, and a ray crossing a layer obliquely inside a medium of index n has a shorter phase path per unit of thickness than one crossing it straight on. The edge wavelength therefore scales as the square root of one minus the squared sine of the angle over the square of the stack’s effective index.

For an effective index of 1.8, which is typical of the high- and low-index oxide stacks such filters are made from, that puts the edge 7 nm to the blue at 15 degrees, 19 nm at 25 and 35 nm at 35. It is the same physics that makes a thin film change colour as it is tilted: a path length and an angle inside the condition, and no pigment anywhere.

Why a corner sees a steeper angle #

At the centre of a frame the light from the lens arrives at the sensor nearly square. At the corner it arrives at the chief ray angle — the angle of the central ray of the cone of light forming that part of the image — and a short lens that sits close to the sensor has a large one. Phone cameras and compact cameras have chief ray angles of twenty to thirty-five degrees at the corner of the frame, because the lens has to be thin and the sensor large relative to it.

So the corner of a phone photograph is taken through a filter whose edge sits twenty or thirty nanometres bluer than the filter the camera was designed around. The rest of the stack — silicon, dyes, microlenses — changes with angle too, but the interference filter’s change is a shift of an edge, and an edge in the red is where the channels are least separated.

What the corner’s channels lose #

The red channel’s response to a lamp is the lamp’s power weighted by the red channel’s sensitivity, and the part the moved edge removes is the part between the two edges.

At 25 degrees the red channel loses 10.1 per cent of its response under daylight, 15.3 under a tungsten lamp, 4.1 under a phosphor white LED and 4.5 under a fluorescent tube. The green channel loses 0.7, 1.5, 0.4 and 0.4, and the blue nothing at all. At 35 degrees the red losses are 20.8, 30.1, 10.9 and 12.3.

The four losses differ because the four lamps put different amounts of power in the band the edge crosses. A tungsten lamp’s power rises steeply towards the red, so it has most there; a phosphor LED’s broad yellow emission falls off before 650 nm, so it has little. The loss is a property of the corner and the lamp together, and that is the whole reason one correction cannot serve.

The correction, and the lamp it belongs to #

A converter corrects the corner by photographing a flat grey card, dividing the centre’s channel values by the corner’s, and multiplying every later frame’s corner by that ratio. The ratio is a gain map: three numbers per position. A gain above one multiplies the corner’s noise along with its signal, which is one more way correcting colour costs noise, and the corner of a frame is where the gains are largest.

Under the lamp the map was made with, the grey is exactly right — the computed error is zero at every angle, because the map was built to make exactly that true. Under a tungsten lamp the same grey is 0.9 colour differences off at 15 degrees, 2.6 at 25 and 5.3 at 35. Under a white LED it is 1.5, 4.2 and 7.9; under a fluorescent tube 1.5, 4.0 and 7.1.

A grey four colour differences off in the corner of a picture is a visible cast, and its direction follows from the losses. A daylight map puts back the tenth of the corner’s red that daylight lost. Under a white LED the corner lost only a twenty-fifth, so the map puts back too much red; under tungsten it lost a sixth, so the map puts back too little. One gain map tints the corners warm in one room and cool in another, which is the colour shading familiar from the corners of phone photographs taken indoors.

Every lamp against every calibration #

If one map is right for one lamp, a converter can hold several and pick one. The question is how many it needs and how close together the lamps have to be.

The diagonal is exactly zero. The white LED and the fluorescent tube correct each other to 0.25 colour differences, because their spectra have similar power in the red band the edge crosses. Daylight and tungsten correct neither each other nor either of the other two: a tungsten map used under daylight leaves 5.5 on the grey, a daylight map under tungsten 2.6, and a tungsten map under a white LED 8.0.

The table is not symmetric, and the asymmetry has a direction. A map made under the lamp with the larger red loss over-corrects every lamp with a smaller one, and a map made under a smaller loss under-corrects the larger. The magnitudes grow with the difference in loss, so a converter needs a map for each distinct amount of red power near 650 nm, which in practice means one for thermal sources, one for daylight and one for the LED and fluorescent family — and it has to know which lamp it is under before it can choose.

The colours a grey card cannot fix #

A gain map is three numbers, and three numbers can make one colour right. The grey is that colour. Every other colour is a question about the shape of the channels, and a moved edge changes their shape.

Under daylight, with a daylight map, the twenty-four coloured patches are a mean of 1.86 colour differences off at the corner and a worst of 3.65. Under tungsten, with the same map, the mean is 1.87 and the worst 4.66. The patches whose reflectances peak in the red are worst in both cases, because they are the ones whose signal depends most on the band the edge removed.

The grey is right under daylight because the map was made on it. A red patch is not, because the corner’s red channel has lost a different fraction of a red patch’s light than of a grey’s: a red patch puts more of its light near the edge. No per-channel gain can make every reflectance right when the change is to the shape of a channel’s sensitivity — which is the same statement a matrix is fitted under one light made about a camera’s colour matrix, arriving at a different stage.

Under a white LED with a daylight map the patches are a mean of 3.28 off and a worst of 4.74. Correcting under the LED instead brings the grey to zero and the patches to a mean of 0.80. Down the diagonal of the table, where every lamp has its own map, the patches are a mean of 1.86 off under daylight, 1.52 under tungsten, 0.80 under the LED and 0.85 under the tube. Even the right map for the lamp leaves coloured patches one to two colour differences wrong, and the order is not the order of the red losses: tungsten, which loses the most, sits between daylight and the two LED-type sources, because a patch’s error depends on the whole shape of its light and not only on the band the edge crossed.

Filters that move less #

The shift is a property of interference, and a camera can avoid most of it in two ways.

An absorbing filter — a blue glass that takes out the infrared in its bulk — has its edge set by the glass’s chemistry rather than by layer thicknesses. Light crossing it obliquely travels a slightly longer path through the glass, which deepens the absorption a little without sliding the edge along the spectrum. Small cameras commonly pair a thin interference coating with a blue absorbing glass for this reason: the glass sets the visible edge and does not move, and the coating blocks what the glass lets through further into the infrared, where a shift matters less.

The other remedy is to reduce the angle. A lens whose chief rays leave it parallel to the axis — an image-side telecentric design — delivers the corner’s light to the filter square on, and the edge stays where it was designed to be. Such lenses are large, which is why they are found on measurement and machine-vision cameras rather than on phones. Shifting each microlens towards the centre of the frame, which small sensors do to catch oblique light, does not help the filter at all: the filter sits in front of the microlenses, and the angle through it is the chief ray’s whatever happens beneath it.

What a converter would need #

Three things, and the third is the expensive one.

A gain map per lamp family. The table says three families cover the ordinary cases to a quarter of a colour difference on grey, and that a single daylight map is several colour differences wrong indoors. Phone camera pipelines that carry lens-shading tables per illuminant are doing exactly this, and the table above is why.

A reliable guess at the lamp. Choosing the map needs the illuminant, and estimating the illuminant is what the highlight is the white balance and every white-balance algorithm attempts from the image itself. An error in the estimate becomes an error in the corners before it becomes an error anywhere else.

And a correction with shape, not only size, for colour. A three-by-three matrix per corner position, fitted to coloured patches rather than to a grey card, would correct the coloured patches under its lamp. It would need a chart photographed at every position and every lamp, which is a calibration most cameras cannot afford, and a matrix still cannot correct a change of shape exactly, only best fit it — the same limit that means no matrix is right everywhere.

What was computed, and how #

The sensor is a modelled silicon sensor: a silicon quantum efficiency, three dye transmittances with a common infrared leak, and a logistic infrared-cut filter with a 665 nm edge and a 34 nm shoulder, on a grid running to 1,100 nm. The corner’s sensor is the centre’s sensitivities with the filter’s transmittance at the centre replaced by its transmittance at the steeper angle, which is the only element changed.

A lamp’s response is the lamp’s spectrum times each sensitivity, summed; a surface’s is the lamp times its reflectance, with the surface’s reflectance taken as nothing beyond 780 nm. The converter balances by the centre’s response to the lamp and applies a matrix fitted at the centre under that lamp, a tone curve and a clip. The gain map is the centre’s response to an eighteen per cent grey card divided by the corner’s, under the calibration lamp. The error is ΔE₀₀ between the centre’s and the corrected corner’s rendering of the same surface.

Where the measurement stops #

The filter is a logistic curve moved rigidly; a real interference stack also changes its shoulder with angle, splits its edge between polarisations at steep angles, and has ripple in its pass band. Each would change the numbers and none would make the shift a gain.

The chief ray angle is taken as a single number per position. A real lens delivers a cone of angles around the chief ray, so each photosite sees an average of shifted filters, which softens the edge at the corner without moving it back.

And only the infrared-cut filter changes with angle here. Microlens efficiency, dye path length and pixel crosstalk — a pixel has an aperture too — also vary across the frame and add their own colour shading, which a real calibration corrects in the same gain map and which is not modelled.

The habit #

The habit is about a correction measured on one input that is applied to all of them.

A flat-field, a gain map, a white balance: each is measured on a reference — a grey card, a white tile, a lamp — and is exactly right for that reference. Whether it is right for anything else depends on whether the error it corrects has the same form for every input or only for the reference. An error that is a pure scaling has the same form for everything. An error that changes the shape of a response does not.

The move is to ask what physical change the correction is standing in for before trusting it on other inputs. A moved edge is a change of shape.

The failure mode is to validate the correction on the reference it was made from. A grey card corrects the grey card, and the corners of every indoor photograph are where that shows.

Who noticed it first #

That interference filters shift towards shorter wavelengths with angle of incidence is standard optics, and filter manufacturers publish the shift for their stacks. That short lenses with large chief ray angles produce colour shading at the corners of small sensors is well known in mobile camera design, and illuminant-dependent lens-shading correction tables are a standard part of phone camera pipelines.

That a grey gain map is exact under its own lamp and leaves several colour differences under another, the per-lamp asymmetry, and the residual on coloured patches under the calibration lamp itself are computed here on one modelled sensor.

Still open: the corner’s own matrix #

A per-position colour matrix fitted under each lamp family would correct coloured patches as well as grey, to the extent a matrix can. How many positions it needs across the frame, how many lamp families, and how much of the residual it removes on real reflectances is a calibration experiment with a colour chart and a real phone lens, and it would say whether the corners of indoor photographs can be fixed in software or only reduced.