The eye counts a corner's error, not its peak

The eye keeps the lightness errors took four defects that come from averaging stored values instead of light — a blend, a resize with negative weights, the corner of a magnified patch, and an unsharp mask — and put each through a model of the visual system’s three spatial channels. The ranking by colour difference per pixel inverted. The mask, the largest error per pixel, lost 62 per cent of its peak on a printed page because half of its error is colour and the eye’s colour channels blur early. The corner, the smallest along the line measured, rose from 7.14 to 9.87, because its error is lightness and the lightness channel keeps it.

That essay measured each defect along a line, and it said what a line could not reach. The corner of a resized patch is lighter than its edges had found the magnified corner’s worst error — 14.5 colour differences — in a small neighbourhood off the diagonal, where the kernel’s negative lobes act in both directions at once, and a filter along the diagonal saw 9.87 of it. Its prediction was that a filter over the plane would leave the corner near 14.5, because the corner’s error is nearly pure lightness wherever it is measured.

The prediction is half right, and the half it gets wrong is the more useful half.

A corner that peaks is not a corner that is seen #

Filtered by the eye over the plane, the Lanczos-magnified corner is seen at 16.5 on a printed page — above both its per-pixel 14.5 and the diagonal’s 9.87 — and it is seen at exactly the level of its own edge. The unsharp mask’s corner, which per pixel is no worse than its edge, is seen at 2.4 times its edge on a print. The eye judges a corner by how much error it holds, not by how high the error peaks.

• Per pixel, the magnified corner is 1.19 times its edge; seen, it is between 0.99 and 1.03 times its edge at a print, a display at 60 centimetres and a display at 25.
• Per pixel, the sharpened corner is 1.01 times its edge; seen at a print it is 2.41 times, 17.1 against 7.1, and at 25 centimetres 1.25 times.
• Along the middle of an edge, the plane and the line agree to within a colour difference for both defects at every distance; the two readings part only where the error has structure in two dimensions.
• The magnification’s lightness error changes sign — lighter, then darker, then lighter — and the mask’s is lighter on every pixel. That is the whole mechanism.
• On a print, the sharpened corner is now seen above the magnified corner, so the inversion the line reading found holds at an edge and fails at a corner.

The two corners, per pixel and as seen #

The figure at the top of the page is the sharpened patch’s upper-left corner, drawn twice on one scale: the colour difference between the mask taken on stored values and taken on light, pixel by pixel on the left, and as the eye’s filter leaves it at print distance on the right.

Per pixel it is two thin bars of error meeting at a right angle, and nothing marks the corner out: the error at the corner, 16.8, is within a per cent of the error anywhere along the bars, 16.6. As seen, the bars have faded and the corner has not. The middle of each bar is seen at 7.1, the corner at 17.1, and the map shows a single dark blot where the two faint bars meet.

The magnified patch does the opposite. Per pixel its corner stands out: 14.5 against 12.2 along the edges, a small bright knot where the kernel’s two sets of negative lobes cross. As seen, the knot has dissolved into the edges, which are now seen at 16.3 along their length, and the corner at 16.5 is no more visible than any point on them. The eye’s filter has not removed the magnified corner’s error — it has raised it — but it has raised the edges’ error by the same amount, and what made the corner a corner is gone.

Corner against edge, at every distance #

The comparison is cleanest as a ratio: the corner’s error divided by the error at the middle of an edge, per pixel and as seen.

The magnified corner’s ratio collapses from 1.19 to 1.00 at every distance: 1.01 on a print, 0.99 on a display at 60 centimetres, 1.03 at 25. Whatever the viewing geometry, the eye does not see the knot the kernel ties at the corner.

The sharpened corner’s ratio rises from 1.01, and by how much depends on the distance. On a print, at 82 pixels a degree, it is 2.41. On a display at 60 centimetres, at 41 pixels a degree, it is 1.03 — the edge is seen almost whole there, so there is no gap for the corner to open. At 25 centimetres, 17 pixels a degree, it is 1.25. The corner survives the eye best exactly where the edge survives it worst, which is the print, where the eye keeps the lightness errors reported the mask’s edge losing 62 per cent.

Why a line misread both #

The line reading was not wrong about the lines it measured. Along the middle of an edge the plane agrees with it: the magnified edge seen at 16.35 along the line and 16.34 over the plane on a print; the sharpened edge 6.29 and 7.09. The disagreement is at the corner.

For the magnified corner, the diagonal reported between 55 and 60 per cent of what the plane sees: 9.87 against 16.50 on a print, 7.65 against 13.73 on a display, 7.27 against 12.30 at 25 centimetres. The diagonal passes through the corner’s centre, where the kernel’s lobes partly cancel, and misses the pixels a little way off it where they do not. A filter along the diagonal then averages the diagonal’s pixels only with each other; a filter over the plane pools them with the pixels beside the diagonal as well, which is where more of the corner’s error sits.

For the sharpened corner no diagonal reading was made, and the line across the edge measured the edge alone. A line through the mask would have shown a single thin spike and filtered it away, as it did at the edge. What makes the corner survive is not on any line through it: it is the overlap of two bars that run in different directions.

Ringing cancels, a halo does not #

The mechanism is in the sign of the error, and the lightness difference itself — stored minus light, with its sign kept — shows it directly.

The magnification’s error rings. Across an edge it runs from +17.5 through −7.3 and back through positive values — the same fringe how fine a colour edge can be measured the eye’s tolerance for, arriving from a kernel rather than from a lens — because a Lanczos kernel’s negative lobes put a dark fringe beside the light one; along the diagonal it does the same with the lobes of both directions combined. A filter that averages a few neighbouring pixels adds light and dark fringes together, and much of the error cancels in the average.

The mask’s error does not change sign. Across an edge it is lighter on every pixel, peaking at +17.8 and falling to nothing on both sides, and along the diagonal the same. Averaging it averages positive numbers, and what survives is roughly how much of it there is in the patch of image the filter spans.

The windowed means make it quantitative. For the magnification, the corner and the edge hold the same mean error from a half-width of four pixels outward — 1.17 against 1.19, then 2.05 against 1.96 — and at smaller windows both are dominated by the dark fringe. For the mask, the corner holds more at every window: 10.0 against 4.7 in a three-pixel square, 9.5 against 6.6 at a half-width of two. A corner of the mask is where the halo from the top edge and the halo from the left edge occupy the same few pixels, all of them lighter, so a window there holds up to twice the error a window on either edge does. At a print’s resolution the eye’s filter pools over a few pixels in each direction — fewer for lightness than for colour — and a pool that sits on the corner collects the overlap, which is why the corner survives where the edge does not.

The per-pixel composition confirms that nothing else distinguishes them. At the mask’s corner the error is 18.2 units of lightness and 13.6 of colour; at its edge, 17.8 and 13.6. The two places are made of the same error. They differ only in how it is arranged.

What this changes about the earlier ranking #

The line reading’s headline was that the ranking inverts: the largest per-pixel error, the mask’s, becomes the smallest seen, and the resize defects rise. That remains true at an edge. At a corner it is not. On a print the sharpened corner is seen at 17.1 and the magnified corner at 16.5, so the mask’s corner is the worse of the two, as it was per pixel.

That matters for the advice the earlier essay implied. A sharpening step is not made safe for print by the eye’s colour blur, as the edge reading suggested — the same blur a halftone is a luminance object relies on to hide a screen’s colour — it is made safe along straight edges and left visible wherever two edges meet — at the corners of type, of window frames, of any rectilinear detail. A resize with a negative-lobed kernel, conversely, is not made more visible at corners than at edges, whatever its per-pixel corner knot suggests; a resize with a negative weight in it described the lobes that tie that knot, and the eye does not see the knot as a separate feature.

What a pipeline could do with it #

Two things, and both are cheap.

Test a sharpening step at corners, not only at edges. The standard check for halos is a straight edge, because halos are an edge phenomenon, and a straight edge is exactly the configuration in which a one-sided halo is seen least on a print. A corner target — a square patch — finds the case a straight edge hides, at no extra cost.

And do linear-light sharpening where corners matter. The mask’s error is the gap between sharpening on stored values and sharpening on light, and it is one-sided because the encoding’s curvature pushes every sharpened pixel the same way. An average on the stored values is where that curvature was first priced, and the repair it named — take the average on light — removes the one-sided error that makes corners survive, whereas it removes a ringing error the eye was cancelling anyway.

How the fields were filtered #

The magnified field is a 32-pixel image with a 16-pixel square of skin against a background of its own shadow, magnified four times with a three-lobe Lanczos kernel on sRGB-encoded values and on linear light, sampled on the magnification’s own output grid into a 128-pixel field. The sharpened field is a 64-pixel image with a 32-pixel square of the same pair, sharpened by an unsharp mask of amount one and radius 1.5 pixels, separable, on encoded values and on light, at one page pixel per image pixel — the geometry the line reading used.

Each field is converted to XYZ and to the model’s three opponent channels, and each channel is filtered in two dimensions by its own low-pass transfer function, including the model’s reduced sensitivity to oblique orientations that a pattern has a direction measured, at the viewing geometry’s pixels a degree, with mirrored boundaries. The colour difference is ΔE₀₀ between the stored and light fields, pixel by pixel, before and after. A corner’s error is the largest in a square around the patch’s corner; an edge’s is the largest in a square of the same size at the middle of an edge.

What this leaves out #

The spatial model is a model. It is a set of contrast-sensitivity functions measured with gratings at threshold, applied as linear filters to errors far above threshold, and every threshold was measured with a grating is the standing caveat about that. The finding here rests on the filter’s spatial extent, which is its most robust property, rather than on the exact shape of any curve; but whether people see a sharpened corner at more than twice its edge on a print is a prediction, not a measurement.

The fields are one pair of colours and one kernel and one mask. A pair whose error is more chromatic would lose more at the edge and possibly at the corner too, and a mask of larger radius spreads its halo over more pixels, which should change the distance at which the corner and edge part.

And a corner of a square patch is the simplest two-dimensional structure. A diagonal edge, a curve and a thin line each arrange the same one-sided error differently, and none is measured here. A diagonal edge is the nearest case and the most informative one to run next, because it has no corner at all and its halos are closer together along the eye’s weaker oblique direction: if the pooling argument is right, a sharpened diagonal edge should be seen slightly more than a horizontal one on a print and much less than a corner.

Still open: whether a corner of type is seen #

The case that matters most for print is text, where every stroke has ends and every letter has corners, and where sharpening is routinely applied to scanned pages. The prediction is sharp: a sharpened glyph at print resolution should show its halos at its corners and stroke ends and hardly along its straight stems.

The computation is the same filter applied to a rendered glyph at a few point sizes. The quantity worth recording is the share of the glyph’s seen error that sits within two pixels of a corner or stroke end against its share of the glyph’s length, and the prediction is that the first is several times the second. If it is not, then the square patch’s corner is a special arrangement and the finding here does not generalise to shapes a reader actually meets.

A peak is not an amount #

The habit is about which summary of an error a reader’s eye computes.

A per-pixel difference map has a maximum, and the maximum is what gets reported: the corner errs by 14.5, the mask by 16.6. The eye does not report a maximum. It pools error over a region whose size is set by the viewing geometry, and in pooling it adds the error’s sign. An error that alternates in sign within that region reports little however high its peak; an error of one sign reports its whole amount.

The move is to look at the signed error before trusting any unsigned summary of it, and then to ask how much of it falls within the eye’s pooling region at the distance in question. Both steps are cheap: the signed map is the unsigned map without an absolute value taken, and the pooling region is a few pixels at any ordinary viewing distance. Neither needs a model of the eye to be run in full; they need only the question to be asked in the right order. Here that turned a corner that peaked into one that disappeared, and a corner that did not peak into the worst feature on the page.

The failure mode is to rank defects by their largest per-pixel difference and to test them in the configuration that makes their difference largest. The configuration that makes an error largest is not always the one that makes it most visible, and for a one-sided error it is often the one that makes it least.