Filling in a highlight is a claim about the surface

Two converters and one highlight measured where a raw converter clips a highlight and found the two defensible placements twenty-one colour differences apart. It said plainly what it had left out: no highlight reconstruction was modelled, and every serious converter attempts one. A reconstruction puts a number back where a raw channel stopped counting. The number has to come from somewhere, and where it comes from is an assumption about what kind of surface the highlight is on.

The fill is right when the model is #

A reconstruction is exact when its model of the highlight is the right one and enough channels survive to feed it, and several colour differences wrong otherwise — and which highlights the right model can reach is decided by the lamp.

• An over-exposed matt surface keeps its colour, and filling the clipped channel from that colour is exact with one or two channels clipped.
• A glossy highlight adds a reflection of the lamp, and the same fill leaves it a mean of 7.6 colour differences off with one channel clipped under tungsten. Solving for the surface’s colour and the lamp’s together is exact there.
• With two channels clipped on a glossy highlight neither fill has enough to work with: both leave 8.6. Clipping to white leaves 3.7, the best of the four.
• Under tungsten a highlight climbs a mean of 0.86 stops with only one channel clipped; under daylight 0.39.

Why a clipped channel needs a model #

A photosite whose well has filled reports the same number for any greater light, so the reading is not a measurement of the highlight but a floor under it. Of all the decisions raw is not a picture leaves to a converter, this is the one where the converter has to supply data rather than choose an operation. A blown highlight turns because the channels that are still counting carry on climbing and the one that stopped does not, and the ratio between them — which is all colour ever is in a raw file — drifts towards whatever is still rising.

A converter has two things it can do with no model at all. It can clip every channel to the level of the lowest ceiling after the white balance, which sends a highlight towards neutral; or it can carry the clipped readings through unchanged, which keeps the drift. Either is a statement about the highlight made by not making one, and the placement between them is what the earlier essay priced.

Anything better has to supply the missing reading, and a missing reading can only be supplied from something known. What is known is the channels that are still counting and what is around the highlight — the same surface a little further from the brightest point, where nothing clipped. What connects the two is a model of how a surface’s colour changes as the light on it rises, and there are two physically different such models.

A matt surface is its own colour, brighter #

A matt surface returns light by scattering it inside the paint, so every photon it returns has met the pigment. Brighter light on it is the same spectrum scaled, and its three raw readings keep the ratio they had in the shade. The unclipped surroundings supply that ratio, the channels still counting supply the level, and the clipped channel follows — one open channel is enough.

Under tungsten, filling from the surface’s ratio is exact on every surface at every exposure until a surface has clipped all three channels. At two and a half times the exposure that first clips, carrying the clipped values through leaves a mean of 28.4 colour differences and clipping to white 11.0; the ratio fill leaves none. The fill that solves for surface and lamp is exact too, because a matt highlight is a special case of its model with no lamp in it.

That is the case highlight reconstruction is usually demonstrated on, and it is the easy one. Nothing about it tells a converter whether the highlight in front of it is matt.

A glossy highlight is the lamp arriving #

A glossy surface returns light two ways — which is why gloss changes the measurement of its colour. Some enters the paint and scatters out coloured; some reflects at the surface before it enters, the same reflection that takes colour out of a bounce in a glossy room, and the highlight is the lamp — that part carries the lamp’s spectrum almost unchanged, because the reflectance of a clear interface barely varies with wavelength. A glossy highlight is therefore the surface’s colour plus a growing amount of the lamp’s, and as it brightens its colour moves from the paint’s towards the light’s.

Filling a glossy highlight from the surface’s ratio assumes the highlight is still the surface’s colour, so it restores the paint’s chroma to a highlight that has lost part of it. On the saturated surface above, the true chroma at equal lightness falls from 52.6 to 42.1 as the reflection grows to 1.2 times the lamp’s own level; the ratio fill stays at 52.6. Carrying the clipped readings through wanders between 38 and 43 as the channels reach the ceiling in turn, and clipping to white holds it at 37.9. Only one response follows the truth.

That response writes each channel still counting as an unknown amount of the surface’s colour plus an unknown amount of the lamp’s, which is the dichromatic model of a highlight that is the white balance turned to a different use. The surface’s colour comes from the unclipped surroundings and the lamp’s from the white balance, so two channels still counting give two equations in the two unknowns, and the third channel follows exactly.

Sorted by what survived #

The ramps mix surfaces that clip one channel at a time with surfaces that clip two at once. Sorting every surface at every step by how many channels it had clipped separates what each fill can and cannot do.

With one channel clipped — 86 cases under tungsten — the fill from surface and lamp is exact on every one, the ratio fill leaves a mean of 7.64 and a worst of 13.7, carrying the clipped value 11.9 and clipping to white 5.57. Under daylight the one-channel cases number only 31, and the fill from surface and lamp is exact on all of them too, while the ratio fill leaves 8.69.

With two channels clipped — 58 cases — there is one reading left for two unknowns. The fill from surface and lamp cannot separate how much of the reading is paint and how much is lamp, falls back on the surface’s ratio, and both leave 8.62. Clipping to white leaves 3.69, the best of the four, because a glossy highlight bright enough to clip two channels really is mostly lamp, and the lamp is white after the balance. With all three channels clipped nothing is left at all.

On a matt highlight the count runs differently: the ratio fill is exact with two channels clipped as well, because a matt surface needs only one channel to fix its level. A matt highlight needs one reading and its surroundings; a glossy one needs two readings, its surroundings and the lamp.

The lamp decides how much can be rebuilt #

Every exact result above lives in a band — the exposures at which a highlight has clipped one channel and not yet a second. How wide that band is depends on how far apart the three raw channels sit, and that is set by the lamp.

A white balance exists because a sensor responds unequally to a lamp’s light, and the inequality is the spread. Under daylight this sensor’s raw white is 1.00, 0.86 and 0.54 across its three channels; under tungsten, 1.00, 0.55 and 0.16. A surface under tungsten reaches its second channel’s ceiling a long way after its first, because the lamp starves the other channels. The mean band is 0.86 stops under tungsten and 0.39 under daylight, with a median of 0.88 against 0.35, and on the least favourable surface under daylight it is two hundredths of a stop — the two channels clip together.

So the same reconstruction, run on the same scene, can rebuild more than twice as much of a highlight’s exposure range indoors under tungsten as outdoors in daylight. That runs against expectation, because daylight is the easy light for almost everything else a camera does. It is easy because it fills the three channels evenly, and a highlight in evenly filled channels loses them together.

The exact fill is the noisy one #

A fill is exact on a model and a model is exact on noiseless readings. The two channels still counting carry noise, and each fill passes some of it into the channel it writes.

Under tungsten the ratio fill passes a median of 4.1 times the open channels’ noise into the filled channel, and the fill from surface and lamp 7.0 times. Under daylight the two medians are 1.4 and 2.8. The ratio fill scales two readings by the surface’s own ratio, and under tungsten the readings it scales are the green and blue a warm lamp starves, so a small reading’s noise is multiplied up to fill a large channel. The dichromatic fill does that and more: it separates two colours using two readings, and separating two nearly parallel directions multiplies whatever error the readings carry.

What makes it worst is not how alike the surface and the lamp look but how alike they are in the two channels still counting. On one surface under tungsten whose green and blue readings hold the lamp’s own balance to three hundredths of a degree the fill multiplies noise 840 times, while the surface whose overall colour is nearest the lamp’s passes on 1.7 times, because its open channels differ. Across the twenty-four surfaces the multiplier’s rank correlation with the angle between the two balances in the open channels is −0.82 under tungsten and −0.87 under daylight. On such a surface a matt highlight and a glossy one give exactly the same two readings, so there is nothing for the fill to separate; the sensible response is the ratio fill, which the two models agree on there.

So the lamp that widens the band also thins what the band is built from. Tungsten gives a highlight more than twice daylight’s exposure range with one channel clipped and passes on between two and a half and three times daylight’s noise into whatever fills it — the arithmetic by which correcting colour costs noise, arriving in a channel nothing measured. A fill’s noise is also not the zero-mean noise a later average removes once a clip has been involved, as clipped noise does not average away found at the other end of the range: a filled channel written as at least the ceiling has its noise cut off from below.

Under daylight, the band is where the difference is #

The daylight ramp shows the narrow band in practice.

At a reflection four tenths of the lamp’s level, with most surfaces clipped in one channel, the fill from surface and lamp leaves 0.60 and the ratio fill 6.57. By 0.7 most surfaces have clipped a second channel, and the two leave 7.74 each. Clipping to white leaves 9.78 there and carrying the clipped values 15.65. The model that is exact under tungsten across three steps of the ramp is nearly exact under daylight at one, and only because a few surfaces there have already clipped a second channel.

Two clipped channels under daylight are a common state rather than an edge case: 60 of the ramp’s cases have two, 53 have three and only 31 have one.

What the placement was measuring #

The earlier essay compared clipping at the sensor with clipping after the matrix and found them twenty-one colour differences apart at the worst. Those are two of the four responses here, the two without a model.

A reconstruction does narrow that gap, and only on the highlights its model fits. On glossy highlights under tungsten with one channel clipped, the fill from surface and lamp removes the whole of what the placements disagree about, and the ratio fill replaces it with an over-coloured highlight of its own. On a matt surface the ratio fill removes it with one channel counting or two. On any highlight with all its channels clipped, every placement and every fill is the same white. The disagreement between converters that the earlier essay attributed partly to reconstruction is, more exactly, a disagreement about which surfaces highlights are on.

The luminance is a separate matter and every response underestimates it. Clipping to white on a glossy tungsten highlight at twice the lamp’s level loses 1.44 stops; on the same ramp the fills stay within a sixth of a stop, and on a matt surface clipped in all three channels they lose two thirds of one, because a filled channel is set by a model and a model’s level is only as good as the channels it read.

Telling the surfaces apart #

The choice between the two models can be made from the image, because the two kinds of highlight leave different traces in the pixels around them.

A matt highlight keeps its chromaticity from the shade to the clipping point; a glossy one moves towards the lamp’s as it brightens. Before any channel clips, the unclipped approach to a highlight already shows which is happening: the ratio of the channels either holds or slides towards the white balance’s ratio. A converter could read that slide on the way up and choose the dichromatic fill where it sees one and the ratio fill where it does not.

It could also be wrong. A matt surface whose colour genuinely varies across the highlight — a gradient in the paint, a fold in fabric — slides for reasons that have nothing to do with the lamp, and an image does not determine the light in general, let alone the surface under it. Whichever it chooses, a fill needs to know which channels stopped counting, and a converter knows that only while it still holds the raw readings and the sensor’s white level — so the fill has to run before anything clamps, the placement one step has no choice and the order is not in the documentation found unrecorded for every other operation in the chain.

What a converter cannot do is make no claim. Clipping to white claims a highlight is entirely lamp; carrying the readings claims nothing is missing; each fill claims a model. The one it chooses should be the one its evidence supports, and a file that recorded the choice would let a later edit revise it.

How the highlights were computed #

The camera is the modelled raw pipeline of the essays on raw conversion: a silicon sensor with dye filters and an infrared-cut filter, a white balance by the lamp’s raw white, and a colour matrix fitted to a chart. The ceiling is one level in counts for all three channels, placed where the lamp’s own white reaches it in its strongest channel.

The surfaces are twenty-four constructed chart reflectances. A matt highlight is the surface’s raw reading scaled so that its strongest channel reaches the ceiling at an exposure of one. A glossy highlight is the surface’s reading at seven tenths of the ceiling plus the lamp’s own raw white scaled by the reflection’s strength, so a strength of one puts the lamp’s strongest channel at the ceiling. The fills take the surface’s colour from its unclipped reading. Colours are compared at equal lightness — the delivered linear values’ chromaticity scaled to an eighteen per cent grey’s luminance — in ΔE₀₀, and luminance errors separately in stops.

What the model assumes #

The surroundings are assumed to give the surface’s true colour and the white balance the lamp’s. A real converter estimates both, from pixels that carry noise and from a white balance that may be wrong, and each estimate’s error passes straight into the fill. The fills are exact here in the way a model is exact on the data it describes.

The glossy highlight’s reflection is taken as exactly the lamp’s colour. A real interface’s reflectance varies by a few per cent across the band, and a metal’s reflection is coloured by the metal, which puts metals outside both models.

And the ceiling is the same in all three channels. Real sensors saturate at different levels per channel and a raw file’s recorded white level is not always where the well filled, which moves the band’s edges without changing what happens inside it.

The habit: a missing value is filled by a model whether or not one is named #

Every method of putting a value where a measurement stopped assumes something about what the measurement would have been. Zero, the last good value, a ratio, an interpolation: each is a model, and the ones that look like no model at all — clip it, carry it — are the ones whose assumptions are least examined.

The move is to write down which kind of object the fill is right for, check that the object in hand is that kind, and count how many independent readings the model needs against how many survive.

The failure mode is to demonstrate a fill on the case its model fits and ship it for every case. A fill tested on matt highlights has been tested on the highlights that need it least, and a glossy highlight under daylight is the ordinary case where it is wrong.

Where reconstruction comes from #

Highlight reconstruction in raw converters goes back to the early open-source raw decoders, which offered modes for clipping, blending towards white and rebuilding a channel from its neighbours’ ratios, and it is a standard feature of every current converter. The dichromatic reflection model is Shafer’s, from 1985, and is the basis of a large literature on separating specular and diffuse reflection in images.

That a fill from surface and lamp is exact on glossy highlights with one channel clipped and a ratio fill is not, that neither survives a second clipped channel, and that the lamp’s spread of raw channels sets how much of a highlight’s range can be rebuilt, are computed here on one modelled sensor.

Still open: whether a converter can read the slide #

The choice between the models rests on seeing whether a highlight’s chromaticity slides towards the lamp’s on its way up. How reliably that slide can be measured from real pixels — through noise, through the sensor’s own mosaic, and on surfaces whose colour varies across the highlight for reasons of their own — and how often a converter reading it would choose the wrong model, is a question about real images rather than about the arithmetic, and it would say whether a converter can choose its fill surface by surface or only once per picture.