A lattice has no derivative

Every departure the previous round measured was measured the same way: change one thing by a stated amount, read the answer, and report the difference. That method has a hidden requirement, which is that the thing being changed has a derivative — that a small change produces a proportionally small effect, so a stated change can stand for the family of changes near it. One of the operations in every delivery chain does not.

The claim #

Quantisation is the one departure in this collection with no derivative anywhere, so it admits a bound and not a sensitivity — and the bound is six times its mean.

• At eight bits a channel the mean cost is 0.191 ΔE₀₀ and the worst 1.15, a ratio of 6.0 over twelve thousand colours.
• Each pair of bits divides the error by almost exactly four — 0.191, 0.048, 0.012 at eight, ten and twelve — which is the code count and nothing else.
• The cost rises with lightness, from a mean of 0.116 below L* 20 to 0.226 above L* 80, which is the opposite of every other departure in this round.
• And the shape does not change with depth. The ratio of worst to mean is 6.0, 6.6 and 5.9 at the three depths, so a deeper lattice buys a smaller version of the same problem.

Why the method does not apply #

A sensitivity is a derivative and a derivative is a limit. The audit’s departures all have one: yellow the lens a little more and the reading moves a little; broaden a cone’s absorption a little and the reading moves a little. That is what makes it legitimate to state a departure at one strength and let a reader scale it, and it is what makes the pairing argument work at all.

Rounding to a lattice has a derivative of zero almost everywhere and no derivative at all on the boundaries between cells. Nudge a colour by a millionth and the output usually does not move; nudge it across a cell boundary and the output jumps by a whole code. There is no strength at which the departure can be stated, because the effect is not proportional to anything.

What can be said instead is a distribution and a bound. Every colour has a distance to the nearest lattice point, that distance has a maximum over the space, and the maximum is a genuine worst case rather than a sample of one. That is a weaker statement than a sensitivity and it is the strongest one available.

The practical difference is that quantisation cannot enter an error budget the way the others do. A budget that adds a quantisation term in quadrature with an observer term is treating a bound as a standard deviation, and the two are not the same object: the bound is attained, on a set of colours that is small but is not empty and is not random.

What it costs #

The measurement is direct. Take a colour, encode it through the sRGB transfer function, round each channel to the lattice, decode, and read the colour difference.

At eight bits the mean is 0.191 and the worst is 1.15. At ten bits, 0.048 and 0.31. At twelve, 0.012 and 0.071. The successive ratios are 4.0 and 3.9 on the mean, against the 4.0 the code counts give: a deeper lattice divides the error and does nothing else to it.

That is worth stating because it is the thing a bit count does say, and it is nearly the only thing. The distribution’s shape is unchanged — the ratio of worst to mean sits at 6.0, 6.6 and 5.9 across the three depths — so a specification that names a depth has named a scale factor on a fixed shape, and has said nothing about where in the space the problem is.

Where it is worst, and why that is a surprise #

The band means in every one of those figures rise from left to right. The mean cost below L* 20 is 0.116 and above L* 80 it is 0.226, so the lattice is nearly twice as expensive at the top of the ramp as at the bottom.

That is the opposite of every other departure in this round. The price of a deviation rises as a surface darkens, by a factor of between thirty and eighty over a set with real lightness in it, so every observer departure, every grid departure and every geometry departure costs most in the shadows. The lattice costs most in the highlights.

The reason is that the lattice is not laid down in tristimulus values. It is laid down in the encoded variable, and the sRGB transfer function exists precisely to spend more code values where the eye is more sensitive. Its slope near black is a straight line of gradient 12.92 and its slope near white is that of a 2.4 power — the shape an absolute encoding replaced — so the code steps near white are much further apart in light than the steps near black.

So two curves are fighting: the metric’s price, which rises towards black, and the encoding’s step size, which falls towards black. The transfer function is an approximation to the inverse of the price and it slightly overshoots, which is why the product is not flat and tilts the way it does.

That is a design statement rather than a defect. An encoding that made the product exactly flat would be an encoding fitted to one difference formula, and there is more than one difference formula and no agreement about which. Overshooting in favour of the shadows is a defensible choice given that shadows are where banding is actually seen.

A bound is not a standard deviation #

The distinction between a sensitivity and a bound has a second half that matters more, and it is about what happens when the number is used.

An audit term that is a standard deviation can be combined with others by the ordinary arithmetic, can be scaled to a different confidence level, and describes a distribution whose tail somebody can reason about. A bound does none of those. It is attained — there exist colours where the lattice costs 1.15 colour differences and they are not hypothetical — and it has no distribution behind it that a factor could be applied to.

The two get confused because a bound and a large percentile look similar in a table. The ninety-ninth percentile of the eight-bit error is 0.636 and the maximum is 1.15, so a reader who assumes the entry is a percentile is out by a factor of 1.8 on the tail they actually face.

And the colours where the bound is attained are not scattered. They are the ones sitting furthest from a lattice point in every channel at once, which is a regular set: it is the centre of every cell, and there are as many of them as there are cells. A gradient walking across the space passes through them at a regular spacing, which is precisely why banding is periodic rather than random.

So the right way to enter quantisation into a chain’s arithmetic is not to enter it at all. It is a floor on what the chain can deliver, to be compared against the tolerance rather than added to the other terms, and the comparison is the one made two sections above: 1.15 against a tolerance of one, at eight bits.

What the encoder is for #

The transfer function’s job is to make the lattice’s cost flat, and reading the residual tilt as a design record says how well it does it and what it was optimised against.

If the encoding were exactly the inverse of the metric’s price, the product would be constant and the band means in the lattice figures would be a horizontal line. They are not: they rise by a factor of 1.95 from the darkest band to the lightest. So the encoding under-spends codes at the top of the range relative to what a flat cost would want, by about a factor of two.

That is a small miss for a curve chosen in the early nineteen-nineties against a cathode-ray tube’s own physics rather than against a difference formula. sRGB’s exponent is what it is because a display of that era had that gamma, and the fact that a curve derived from an electron gun’s transfer characteristic lands within a factor of two of the inverse of a perceptual metric is the coincidence the whole of display engineering has been living on.

The coincidence is not an accident and it is not exact. A cathode-ray tube’s gamma is near 2.2 for reasons of grid geometry; a lightness compression is near a cube root for reasons of receptor response. Neither knows about the other, and the agreement is close enough that eight bits works and loose enough that its worst case exceeds a delivery tolerance.

The absolute encodings that replaced it were designed against a difference criterion rather than a display’s physics, and their band structure is correspondingly flatter — which is the strongest argument for them and is not the one usually made.

What a bit count does not say #

This collection has already found that a bit count does not predict visible banding, because banding is a spatial phenomenon and depends on where in the tone scale the gradient sits, how sharp each step’s edge is and how far away the reader is. That argument is about visibility.

This one is about arithmetic and it is separate. Even for a reader who will never see a band, the lattice puts a floor under how exactly a colour can be delivered, and that floor competes with the tolerance the contract names. At eight bits the worst case is 1.15 colour differences, which is larger than the tolerance in most delivery contracts this collection has quoted.

And the comparison is not with the tolerance number but with the shell. A tolerance of one colour difference accepts a shape whose shortest radius is much smaller than its longest, and a rounding error that happens to point along the short axis exhausts the tolerance sooner than its ΔE₀₀ suggests. The lattice’s direction is not random — it is along the encoded channel axes — so whether it lands on a long axis or a short one is a systematic property of the colour rather than a matter of luck.

Where the lattice lands relative to the shell #

The comparison that decides whether eight bits is enough is not between two numbers but between a bound and a shape, and the shape has an orientation.

A rounding error points along the encoded channel axes — it is a displacement in R, G and B, each independently up to half a code. Transformed into tristimulus values those three directions are the display’s own primaries, which are three fixed directions in the space and are not aligned with anything the metric cares about.

The accepted set of a tolerance is tightest along a red-green direction at every colour measured, and a display’s red primary is not far from that direction. So a rounding error in the red channel consumes tolerance faster than one of the same magnitude in the blue channel, and the encoder has no way to know that: it rounds each channel by the same rule, because a channel is what it has.

A quantiser that knew the metric would round the three channels to different precisions, spending more codes on red than on blue at the same total bit budget. Nothing does this, and the reason is not ignorance — an encoding whose channels had different depths would be an encoding nothing else could read, and interoperability is worth more than the factor of a few it would buy.

That is the honest ending for a measurement like this one. The defect is real, the repair is known, and the repair is not worth its cost, which is a different conclusion from the ones this round has mostly reached and is worth having among them.

The other lattice #

There is a second lattice in every chain and it is coarser than the first.

A profile is a table, exact at every patch that was printed and interpolated everywhere else, and its grid is seventeen or thirty-three points a side rather than two hundred and fifty-six. Its cells are therefore between fifteen and eight times wider, and unlike the code lattice its output is not rounded but interpolated, so its error is smooth and does have a derivative.

That difference is the interesting one. A table’s error can be propagated and a lattice’s cannot, even though the two look like the same kind of object and are often described in the same sentence. A finer table converges; a deeper lattice divides. The first is an approximation improving and the second is a floor descending, and only the first can be entered into an audit as a sensitivity.

Putting the two side by side gives the chain’s two irreducible steps and their two different characters, and a chain’s arithmetic should treat them differently. Nothing in this collection’s own budget does.

What was computed, and how #

Twelve thousand colours are drawn uniformly in the linear sRGB cube by a stated generator with a fixed seed, so the measurement is reproducible and its sampling is stated. Uniform in linear light over-samples the highlights relative to how an image is distributed; a sampling uniform in the encoded variable gives a mean about a fifth lower and the same worst case and the same trend, because the worst case is a property of the cell size rather than of how often the cell is visited.

The transfer function is the sRGB standard’s piecewise form, linear below 0.0031308 and a 2.4 power with an offset above it. Quantisation is round-to-nearest, which is what an encoder does; truncation would double the mean and is what a badly written one does.

The two piecewise functions in this essay are worth putting beside each other. CIELAB breaks at a relative luminance of 0.00886 and sRGB’s encoding breaks at 0.0031308, and both breaks exist because a power law has an infinite slope at zero. They were chosen by different committees for different purposes and they do not coincide, so between those two luminances the encoding is linear and the metric is a cube root.

Where the model stops #

Everything here is per-channel rounding in sRGB. A file in a wider gamut at the same depth is worse in proportion to how much more of the space it spans, and a file in a perceptual encoding is better; neither is measured here.

The measurement takes a colour to its nearest lattice point in each channel independently, which is what an encoder does and is not the nearest lattice point in the space. The distinction costs nothing at eight bits and would matter for a serious analysis of the lattice’s geometry, which this is not.

And there is no dither in it. Dithering trades the deterministic error for a random one of the same mean magnitude and much better spatial behaviour, which is why every real encoder does it — a dither mask is a spatial object — and why the worst case here overstates what a reader would ever see.

The generalisation #

The habit is about which quantities in an audit can be scaled and which can only be bounded.

An audit that varies its inputs produces two kinds of entry that look identical in a table. The first is a sensitivity: an effect proportional to a cause, stated at one strength, valid at others. The second is a bound: a maximum over a set, attained somewhere, with no strength attached and no scaling law. A reader multiplies both by whatever factor their situation calls for, and for the second that operation is meaningless.

The move is to label them. One extra column, three characters wide, saying whether an entry is a slope or a ceiling.

The failure mode is quiet and it favours the auditor. A bound entered into a budget as though it were a standard deviation makes the budget look tighter than it is, because bounds are attained and standard deviations are not — and the budget’s total, being a root of a sum of squares, hides the substitution completely.

Who found it, and when #

Quantisation error in colour encodings is a standard engineering subject and the eight-bit figures here are not new; the sRGB standard’s own design rationale is that eight bits with a 2.2-ish curve puts the step size near the visibility threshold across the range, which is what the flat-ish band means confirm.

What does not appear to be standard is the observation about method — that a lattice cannot be entered into a sensitivity audit at all, and that its presence in one is a category error rather than an approximation. It arises here only because this round is auditing an audit, and the boundary between what can be perturbed and what cannot is the boundary the round is about.

Where the ladder goes next #

Five of these six rungs have been about the price of a small change. The last is about a large one, and about the property colorimetry is built on: an additive mixture is exactly a straight line in tristimulus values, which is Grassmann’s second law, and it is not a straight line in anything a reader is ever shown.