A tolerance has a grain

A colour tolerance is written as a single number, and what that number accepts is a closed surface in tristimulus values whose volume changes by five orders of magnitude across the sRGB cube and whose longest axis runs between three and thirty times its shortest. Those are sizes and ratios. A long thin shape also has a direction, and the direction is the part a supplier can do something with, because a delivery does not drift at random.

One direction, nearly everywhere #

The shortest axis of a ΔE₀₀ tolerance points the same way across almost the whole of the sRGB cube, and it is the way in which X rises while Y falls.

• Over sixty-four colours the shortest axis sits a median of 15 degrees from one common direction, ninety per cent of them within 32 degrees.
• That common direction is 3.1 degrees from X up and Y down, which is the gradient of a*, and a median of 9 degrees from each colour’s own a* gradient against 45 for L* and 54 for b*.
• The longest axis lies near the white: a colour can move furthest by becoming more or less of the same light.
• So drifts of the same relative size are allowed very different amounts. A one-unit tolerance on an ordinary surface allows the light on it to change 9.3 per cent, a gain error confined to Z 5.1 per cent, and a gain error confined to X 1.4.

Why the tight way is a* #

The explanation is in the definition, and it is not a subtle one.

CIELAB builds its three coordinates from cube roots of the tristimulus values divided by the white’s, and then takes differences with three different coefficients: L* is 116 times the compressed Y, a* is 500 times the difference between compressed X and compressed Y, and b* is 200 times the difference between compressed Y and compressed Z. A small step in tristimulus values moves L*, a* and b* by amounts proportional to how the step lines up with those three gradients, weighted by those three coefficients.

The a* gradient carries the largest coefficient, so a step along it moves the colour further than a step of the same length along either of the others. The shortest radius of a tolerance is the direction along which the least tristimulus change uses up the allowance, and that is the direction of the fastest-moving coordinate. The grain of a colour tolerance is the number 500.

ΔE₀₀ does not use CIELAB’s coordinates as they stand. It divides the lightness, chroma and hue differences by weights that depend on where the colour is, and it rotates the hue term in the blues. Where the formula is not smooth is one consequence of that. Those weights bend the grain without replacing it: the chroma weight rises with chroma, which loosens a tolerance along whichever direction chroma lies, and the lightness weight rises away from the middle of the scale. The measured spread — 15 degrees at the median and 55 at the extreme — is the weights working against a coefficient they are not strong enough to overturn.

The loose way is the white #

The longest axis has an explanation of the same kind. Scaling all three tristimulus values together changes L* and, for a colour that is not neutral, changes a* and b* by amounts proportional to how far the colour already sits from grey. For a neutral it changes nothing but lightness, and lightness has the smallest coefficient.

So the loose direction lies close to the direction of the white itself: a median of 21 degrees from it over the sixty-four colours, and 6 degrees for the common axis. A tolerance is most generous to a change that makes a colour more or less of the same light, and least generous to a change that trades X against Y.

A tolerance is a shape made the point that a total colour difference and a set of component tolerances accept different sets of samples. The grain is a finer version of the same observation, stated in the coordinates a process actually drifts in rather than in the coordinates a specification is written in.

Five drifts, each with a direction #

Processes drift in characteristic directions, and each of those directions can be written down as a shape before any sample is measured.

More light, or a denser print, scales a sample’s reflectance, which scales all three tristimulus values together — the loose direction. A veil of white — flare in an instrument, show-through from a backing — adds a fixed amount of the white to every reading. A thicker ink film deepens the absorption bands a surface already has, which changes its spectrum’s shape and mostly its level. An instrument’s X filter changing its transmission scales X and nothing else, and so does a drift in its Z filter. A warmer booth lamp, the sixth, changes the spectrum the sample is lit by.

Driven to exactly ΔE₀₀ 1 on forty-two surfaces from a family with four reflectance levels, the medians are these. The light may change by 9.3 per cent. The Z filter may drift 5.1 per cent. An ink film may thicken by 3.2 per cent. The X filter may drift 1.4 per cent. A veil may reach 0.46 per cent of white. A warmer lamp, measured against an instrument calibrated on the nominal one, is allowed 353 kelvin.

Two of those numbers are directly comparable, because they are the same kind of fault in the same instrument. A drift in X and a drift in Z are both a gain error in one filter, of the same relative size, and the X filter is allowed a quarter of what the Z filter is. Nothing about the filters decides that. The tolerance does, by being thin along the direction an X drift travels and thick along the direction a Z drift travels.

Size does not predict the allowance #

The drifts do not sort themselves by magnitude. What sorts them is how much of each lies along the tight axis at the colour it acts on.

Changes of light, film and Z sit within a degree or two of ninety degrees from the tight axis, which is to say almost wholly across it. The X filter’s drift sits at a median of 45 degrees, with half of its length along the direction the tolerance is least forgiving in. That single angle accounts for most of why one gain error is allowed four times another.

The veil is the useful exception, because it shows that direction is not the only thing. A veil is almost perpendicular to the tight axis and is still allowed less than half a per cent of white. The reason is its size relative to the sample rather than its direction: a fixed addition of white is a small fraction of a light surface and a large fraction of a dark one, and on the darker members of the family half a per cent of white is several per cent of the surface. Direction decides the price of a drift of a given relative size, and whether a drift is relative or absolute decides its relative size. A specification needs both.

Filters that age together are cheap #

The two gain errors above are faults in one filter. The allowance for the light level says what the same kind of fault costs when all three channels share it.

Some instrument faults act on every channel at once: the instrument’s own lamp dimming, a dusty aperture, a detector losing sensitivity evenly across the spectrum. A gain error shared by the three channels scales X, Y and Z together, which is exactly what more or less light on the sample does, so it lies along the loose axis and is allowed 9.3 per cent at the median. The same loss confined to the X channel would use up the tolerance several times over.

So the grain gives an instrument a cheap way to fail and an expensive one. A fault common to the channels is nearly free. A fault confined to X is the most expensive of the single-filter faults measured here, and the same fault in Z costs about a quarter as much.

That changes what a standardisation schedule protects. Most instruments standardise on a white tile, which divides out a pure gain in every channel exactly, so a gain drift only matters in the interval between one standardisation and the next. The channel that decides how often an instrument must be standardised is then the channel lying across the grain: at a drift of a tenth of a per cent a day, an X channel uses its allowance in two weeks and a Z channel in seven. The rate is an illustration; the ratio between the two intervals is the grain’s, and a schedule set by checking whichever channel is easiest to check has been set by the wrong one.

Where the grain bends #

The common direction is common, not universal. At ninety per cent within 32 degrees, a tenth of the cube has its tight axis turned further, and the worst case is 55 degrees away.

The turned colours are the saturated ones, and among those the reds. A red at full drive is the most turned colour on the lattice at 55 degrees, a darker red 51 and an orange 46. The greys are the least turned, all four within 3.6 degrees of the common axis, and so is the darkest colour on the lattice, even though it sits inside the straight piece where the compression has stopped compressing. Across the sixty-four the angle correlates with chroma at 0.43 and with lightness not at all.

The reason is ΔE₀₀’s chroma weight. On a neutral there is no chroma to weight, the coefficients decide the grain on their own, and the grain sits on a*. On a saturated colour the chroma weight is large, which loosens the tolerance along that colour’s own chroma direction, and a red’s chroma direction lies close to a* — so the loosening falls on the very axis that was tight, and the tight axis turns away from it. The weights that make a tolerance fairer to saturated colours are the weights that turn its grain.

The turning comes with a rounding. The most turned shells are the least elongated — the angle correlates with the axis ratio at −0.51 — and a shell eight times longer than it is wide has less difference between its directions than one twenty-eight times longer. The correlation with volume is −0.01. So where the grain turns most it also matters least, and the practical advice holds across the whole cube: a drift that trades X against Y is the expensive one.

What a supplier could do with it #

Three consequences, and they get more useful as they go.

Instrument checks are weighted wrongly by default. Inter-instrument agreement is usually reported as a single ΔE over a set of tiles, which averages over directions that are not equally expensive. An instrument whose X channel reads one per cent high costs as much, on an ordinary surface, as one whose Z channel reads four per cent high. A calibration report that listed the error per channel would say which of those an instrument has; a report in ΔE cannot.

Some process drifts are nearly free and some are not. A press that runs slightly heavy delivers more ink of the same colours, which is close to the loose direction, and a tolerance written in ΔE₀₀ is forgiving of it. A press whose ink has drifted in hue — a slightly redder magenta, say — delivers a change with a large a* component, which is the tight direction. The two can be the same size in density and a factor of several apart in what the tolerance makes of them.

And a tolerance could be stated with its grain. A delivery tolerance is three tolerances already, and a fourth parameter — the ratio between the allowance along the white and the allowance along a* — would let a supplier plan against the drifts that process actually has. That is close to what the textile industry’s CMC formula does with its lightness-to-chroma ratio, and the constants nobody quotes in ΔE₀₀ are a version of the same idea left at their defaults.

What was computed, and how #

Each colour’s tolerance is found by bisection along a direction: walk outwards from the colour in tristimulus values until ΔE₀₀ reaches the stated tolerance, and record the distance. That is done along two hundred directions spread evenly over a sphere, and the shortest and longest of them are then walked downhill on the sphere with a halving step until they hold still. The sweep alone resolves about fifteen degrees, which is the size of the spread being measured, so the refinement is not optional.

The common axis is the principal direction of the set of shortest axes — the eigenvector of the sum of their outer products — which does not care that an axis has no sign. The angles are between axes, so they run from nought to ninety.

The drifts are computed on forty-two surfaces from a family of banded reflectances at four levels, under a 6500 K thermal source, through the 1931 observer. Each drift is a stated function of its size: the reflectance scaled, a multiple of the white added, the reflectance raised to one plus the size, a filter’s reading scaled, or the lamp’s temperature lowered with the reading still divided by the nominal white. The allowance is the size at which ΔE₀₀ reaches one, found by bisection.

A shell is star-shaped — walking outwards crosses the tolerance once — and that is checked rather than assumed, because ΔE₀₀ has non-monotone terms and a bisection converges either way.

Where the measurement stops #

Everything here is ΔE₀₀ against the 1931 observer. A difference formula’s weighting is the thing that bends the grain, so another formula would bend it differently; ΔE*ab, with no weights at all, would put the grain on a* almost exactly.

The drifts are shapes stated in advance, at one reflectance family and one lamp. Real processes drift in combinations, and the direction of a combination is a weighted sum of the directions here — which is exactly why two sizes do not compose without an angle and why the angle is the quantity worth publishing.

And a tolerance is a matching tolerance. Whether an observer judging two samples side by side is also most sensitive along a* is a question about people rather than about a formula, and the answer is only as good as the data the formula was fitted to.

The habit #

The habit is about a scalar limit placed on a vector quantity.

A tolerance, a threshold, an error budget: each reduces a change in several coordinates to one number and accepts or rejects on it. The reduction always has a shape, and a shape that is not round always has an orientation, and the orientation is fixed by the coefficients inside the reduction rather than by anything the person setting the limit chose.

The move is to find the direction along which the limit is tightest before deciding what to control. It costs one bisection per direction.

The failure mode is to control whichever quantity is easiest to measure, and to discover that it was the one the limit barely notices while the one it punishes went unwatched. A single number does not treat all directions alike, and nothing in the number says which direction it favours.

Who noticed it first #

That colour discrimination is anisotropic in a consistent direction is as old as MacAdam’s ellipses, which are long and thin and whose orientations follow a pattern across the chromaticity diagram. CIELAB’s coefficients were chosen to make those shapes rounder, and the choice of 500 for a* against 200 for b* is where a*'s dominance comes from.

That the resulting tolerance shape has one dominant direction in tristimulus values, and that the direction is the gradient of a*, follows from the definition and is not usually stated. The drift allowances are specific to the reflectance family and the lamp used here and should be read as an illustration of an effect whose sign is general and whose size is not.

Still open: which drifts real processes have #

The allowances above are for drifts of stated shape. What decides whether the grain matters to a particular press, dye house or display line is which of those shapes its process actually produces, in what proportion — a covariance of process variation expressed in tristimulus values. That is a measurement a supplier already has the data for and almost never computes, and it would say directly how much of a process’s variation lies along the one direction a tolerance does not forgive.