A grey edge arrives coloured

Everything else in this field is spectral. The arguments are about three curves, three integrals and what the collapse from a function to three numbers throws away.

This one has no spectrum in it at all. It is about where on the sensor each measurement was made, and it produces colour out of a scene that had none.

The claim #

Each sensor site measures one channel, the other two are interpolated from neighbours that were somewhere else, and across a sharp edge that reconstruction produces colour in a scene that had none.

Nothing is broken. The sensor measured what was in front of it; every number in the raw file is correct; and the colour is created by the arithmetic that turns one channel per site into three.

Why one channel per site #

A colour filter array puts a single dye over each photosite, in a repeating pattern — most commonly two greens, one red and one blue in a two-by-two tile, an arrangement patented by Bryce Bayer in 1976 and still dominant.

The alternative is three sensors and a beam splitter, which triples the cost and the size, or three stacked photodiodes at different depths exploiting silicon’s wavelength-dependent absorption depth, which has been built and has its own problems. The mosaic won, comprehensively, on cost and light efficiency.

The price is that two thirds of the data is missing at every location and is invented, and the inventing is the stage called demosaicing. It is not a small correction: a twenty-four-megapixel camera writes twenty-four million measurements and outputs seventy-two million numbers.

Where the colour comes from #

The mechanism is easiest to see in one dimension, which is what the figure above is.

Take a row alternating green and red sites, and a scene that is bright on the left and dark on the right with a single sharp step between them. Green is sampled at the even positions and red at the odd, so the two channels are sampled at different places.

Interpolate each independently. Near the step, the red value at a green site is estimated from the two nearest red samples, which straddle the edge — so it comes out somewhere between bright and dark. The green value at that site was measured, so it is exactly bright or exactly dark. The two disagree, and their ratio is what colour is.

The result is a red-cyan fringe running along an edge that had no colour in it. Measured here, the peak chroma reaches 144 per cent of the local mean, which is not a subtlety at the edge of the effect but the dominant thing happening at the edge of the object.

The assertion is what makes this an argument rather than an illustration. The generator checks that the scene contains exactly two distinct levels before it draws anything, so the caption’s claim that nothing in the scene was coloured is a fact about the drawing rather than a promise.

The repair, and what it assumes #

The standard fix is not to interpolate the channels. It is to interpolate the colour differences.

Reconstruct green first — it is sampled twice as often, so its reconstruction is better — and then interpolate $R - G$ and $B - G$ across the gaps rather than $R$ and $B$, adding the reconstructed green back at the end. Since the edge is achromatic, $R - G$ is zero on both sides of it, and a quantity that is zero on both sides interpolates to something near zero in between.

Measured, that takes the peak chroma from 144 per cent to 112. A real improvement and not a repair, and the shortfall is instructive.

The method assumes hue varies more slowly than luminance — and brightness is not luminance either — which is true of most scenes and is false at a step edge — a step edge is precisely where the assumption has no room to work, because $R - G$ at a red site is a correct red minus an interpolated green, and the interpolated green is already wrong there. Better algorithms estimate the green using gradient information from the red and blue channels, which is a stronger assumption about scenes and does considerably better.

The sampling density is not what decides it, which is worth showing rather than asserting.

Four times as many sites is the other direction, and it is the one an instinct about resolution expects to fix the problem.

Two more samplings say that the repair and the sampling are separate arguments, and that neither of them removes the fringe.

Why more pixels do not help #

A natural expectation is that this is a resolution problem and that a denser sensor solves it. It does not, and the reason is worth being precise about.

The fringe is not caused by too few samples in an absolute sense. It is caused by the three channels being sampled at different positions, and doubling the sensor’s resolution halves the spacing without changing that fact. The fringe becomes narrower in pixels and stays the same in scene terms relative to the sharpest edge the optics can deliver.

What actually removes it is either sampling all three channels at every site — which is what a three-sensor or stacked design does — or blurring the image slightly before it is sampled so that no edge is sharper than the mosaic can handle. The second is what an optical anti-aliasing filter does: a birefringent layer that splits each ray into two slightly separated ones, deliberately degrading resolution to keep the scene’s spatial frequencies below the mosaic’s limit.

A camera with such a filter has traded sharpness for the absence of false colour, and the trade is a design decision that manufacturers have reversed twice in twenty years — filters were standard, then widely removed as sensor resolution rose past most lenses’ ability to feed them, and are now returning in some designs.

The same argument the eye already made #

This site has already met a mosaic sampling three channels in different places, and the essay about it reached a different conclusion for a good reason.

The cone mosaic is not the observer: the retina’s L, M and S cones are interleaved, in a ratio that varies enormously between people, and each location samples one type. That is the same arrangement, with two additional features that change everything — the mosaic is irregular rather than periodic, and there is a great deal of processing afterwards.

Irregularity matters because a periodic sampling grid aliases high spatial frequencies into low ones coherently, producing the structured false colour this essay is about. An irregular one scatters the same energy into noise, which is far less objectionable. That is an argument made about retinal mosaics and adopted deliberately in some sensor designs.

Two failures that look the same and are not #

False colour from demosaicing is routinely confused with two other artefacts, and separating them is worth a paragraph each because the remedies differ completely.

Chromatic aberration is optical. A lens focuses different wavelengths at slightly different distances or magnifications, so the three channels are imaged at slightly different scales and an off-axis edge acquires a coloured fringe. It grows towards the corners of the frame, it is absent at the centre, and it is corrected by a geometric warp per channel — a correction that is entirely useless against demosaic false colour, which is uniform across the frame and has nothing to do with position.

The eye has the same problem and cannot correct it: longitudinal chromatic aberration in the human eye spans about two dioptres across the visible band, which is far more than any camera lens, and nothing in the visual system warps one channel against another.

Purple fringing is a third thing again, arising from a mixture of lens aberration, sensor blooming near saturation and the demosaic behaving badly on a clipped edge. Because it appears at high-contrast boundaries it looks like this essay’s subject; because it is worst wide open and improves on stopping down, it behaves like the lens’s.

The diagnostic that separates them is position and sharpness. Demosaic false colour is uniform across the frame and lives on the sharpest edges. Chromatic aberration grows radially. Purple fringing needs a blown highlight next to it. Getting the diagnosis right decides whether the fix is a different converter, a different lens, or a different exposure.

Where the model stops #

One dimension is not two. A real Bayer pattern is a two-dimensional tile, greens sit on a quincunx and reds and blues on square lattices at half the pitch in each direction, and the geometry of the failure is correspondingly richer — false colour appears as coloured maze patterns on fine repeated detail rather than as a simple fringe. The one-dimensional case has the mechanism and none of the pattern.

The repair modelled here is the simplest one. Production demosaicing uses edge-directed interpolation, iterative refinement and increasingly learned models, and all of them do better than 112 per cent. None of them removes the effect, because none of them can measure what was not sampled.

And nothing here is about noise or about the optics. Both interact with demosaicing in practice — noise is what makes gradient estimation unreliable, and lens blur is what determines whether the scene contains edges sharp enough to trigger the effect at all.

What the fringe costs downstream #

The fringe is a local artefact and its effect on the rest of the pipeline is not local, which is worth one measurement.

Every stage after demosaicing operates on the reconstructed triple. The colour matrix multiplies it, the tone curve maps it, the encoding quantises it — and none of them can distinguish an invented value from a measured one. So a false-colour edge is carried forward as though it were a real colour, amplified by the matrix’s off-diagonal terms along with everything else, and the amplification is not small.

That interaction is the reason modern converters demosaic and denoise jointly rather than in sequence, and the reason white balance is now usually applied before demosaicing: multiplying the three channels to near-equality first means the interpolation is done on more nearly equal numbers, which makes the colour-difference assumption hold better. Both are repairs to a chain that was drawn as a chain and is not one.

The generalisation #

The transferable statement is about sampling several quantities in different places and then comparing them, and it is more common than the imaging case.

A ratio between two quantities sampled at different locations is not a ratio of anything, wherever the quantities vary. Where the field is smooth the interpolation is harmless and the ratio is nearly right. Where it is not, the error in the ratio can exceed either quantity’s own error by a large factor — because the numerator and denominator are wrong in different directions, and a ratio amplifies exactly that.

The specific instances are everywhere once the shape is visible: two instruments on a moving platform sampling at different times, two sensors at different points in a gradient, two survey questions asked of overlapping but different populations, a rate computed from a numerator and a denominator collected on different schedules. In every case the derived quantity is more fragile than either input and the fragility concentrates at the discontinuities.

And the standard repair is always the same one demosaicing uses: interpolate the derived quantity rather than the inputs, on the assumption that it varies more slowly. That is usually a good assumption, it is an assumption, and it is worth saying so — which is why the measurement in this essay reports what the repair removes rather than that it works.

Why green gets two sites, and what that argues #

The Bayer pattern’s asymmetry is the part most worth understanding, because it is an argument about the eye written into the sampling geometry.

Green gets half the sites; red and blue get a quarter each. The reason given is that green carries most of the luminance signal, and it does: the $\bar{y}$ matching function is by construction the photopic luminous efficiency function, it peaks at 555 nanometres, and the luminance row of the sRGB matrix weights green at 0.7152 against red’s 0.2126 and blue’s 0.0722.

The second half of the argument is about acuity. Human spatial resolution for luminance is several times better than for chroma — the visual system’s chromatic channels are low-pass in a way the achromatic channel is not — so a sampling scheme that spends its density on luminance and economises on chroma is spending it where a person can tell.

This is the same reasoning every image codec uses, and it is worth noticing that a camera applies it at the sensor and a codec applies it again at the file: chroma subsampling in JPEG and in video encoding discards three quarters of the chroma resolution on exactly this argument. A photograph has therefore usually had its chroma sampled sparsely twice, by two independent systems each appealing to the same fact about vision.

The consequence for this essay’s subject is that the false colour is worst where the pattern is sparsest. Blue and red are the undersampled channels, and blue-red fringing on fine detail is the characteristic signature — which is the same reasoning read backwards, and is why the demonstration above uses a green-red row rather than a green-green one.

Who noticed, and when #

Bryce Bayer’s patent was filed in 1975 and granted in 1976, at Eastman Kodak, and its central argument is worth recovering because it is often misremembered as being about colour. It is about luminance: Bayer gives green twice as many sites as red or blue on the grounds that green carries most of the luminance signal, and that the eye’s spatial acuity for luminance far exceeds its acuity for chroma. The pattern is an argument about human vision embedded in silicon.

False colour arrived with the pattern and the literature on suppressing it is enormous — bilinear interpolation, then colour-difference methods in the nineteen-eighties, then edge-directed and frequency-domain approaches, then joint denoising-and-demosaicing, and since about 2016 learned reconstructions that outperform all of them on the metrics anyone has agreed to measure.

What has not changed is the structure of the problem. Every one of those methods is a stronger prior about what scenes look like, applied to recover measurements that were never made, and the best of them are better priors rather than better measurements.

Where it shows up outside a photograph #

Two consequences reach beyond the picture itself, and both matter more than the fringe does.

Every colour measurement taken from an image is taken from interpolated data. Machine vision inspecting a printed sheet, a colorimetric app reading a chart from a phone, a scientific camera measuring a stained slide: unless the region sampled is large and flat, the numbers include invented values, and the invention is worst exactly at the boundaries where somebody has drawn a region of interest. Averaging over a large uniform patch removes it almost completely, which is why every practical procedure says to do that and few say why.

And the artefact survives into the training data of anything learned from photographs. A model trained on demosaiced images has learned the demosaic’s characteristic fringes as a property of edges, and there is a small literature on identifying which camera took a photograph precisely by the residual signature its demosaicing left. That signature is a forensic tool and a contaminant depending on what one is doing.

The general form is the one this collection keeps arriving at from different directions: a reconstruction is not a measurement, and the distinction survives every downstream stage that treats them alike.

Where the ladder goes next #

Downward, this rung sits on raw is not a picture, which lists demosaicing as one of six decisions, and on the mosaic is not the observer, which is the retina’s version of the same arrangement.

Upward, the field returns to the spectral thread. The highlight is the white balance takes up the next decision in the chain, and a photograph is not a measurement collects this failure with the others.