A family's line runs through its neighbours

A phone’s ambient-light sensor reads a few narrow channels, and the question a run of essays here has put to it is whether they can tell the camera — a fourth observer, agreeing with no eye — how wrong its colour will be under the lamp in the room. Two stages catch the tube and bill the lamps they misplace tried the obvious design: recognise which family the lamp belongs to — radiator, white LED, halophosphate tube, other tube, three-emitter source — from how the sensor’s 450, 500 and 640 nm channels depart from what the camera predicts, then apply that family’s own line from the departures to the error — the line one ratio reads a peak and a trough built from the gap between the 450 and 500 nm departures, with the 640 nm departure beside it. Told each lamp’s family, the design caught the halophosphate tube and brought the held-out error over fifty-four lamps from the single ratio line’s ±0.141 to ±0.133. Left to find the families by each lamp’s nearest neighbour, it misplaced five and paid ±0.162.

A classifier that abstains gives up the tube let the vote say when it was unsure. It abstained on the wrong lamp. The misplaced lamps had three agreeing neighbours, all white LEDs; the lamp it gave up was the halophosphate tube, whose two nearest neighbours are its siblings and whose third is a cyan-filled LED, because the tube’s own family does not form a cluster the sensor can see whole — it runs through the cyan LEDs. Chosen without looking at the lamp it estimates, the abstaining rule came to the ratio line’s ±0.141 exactly.

That essay’s closing section noticed what “runs through” implied. The halophosphate tubes differ from one another by one continuous number — where the main phosphor band sits, from 558 to 598 nm across the tube and its four siblings — and a family generated by one number is, to a first approximation, a line in the departures. A classifier that fitted each family’s line and gave a lamp to the family whose line passes nearest would see the halophosphate family whole where a vote sees it in pieces. The prediction was that it keeps the tube, because the tube sits on its siblings’ line however many LEDs lie closer; that it still misplaces the pink-tinted radiator, whose neighbours are LEDs on every side; and that the census’s held-out error falls below ±0.141 with the rule fixed in advance.

The line keeps the tube and claims the LEDs #

Every line classifier that measures plain distance keeps the halophosphate tube with its siblings and estimates it within a hundredth and a half, as predicted. None of them improves the census. Given every family’s infinite line, eighteen white LEDs — the cool LED and most of the violet- and cyan-pumped ones — lie nearer the radiators’ line than their own, the radiators’ estimate is extrapolated to them and to three other lamps, and the census’s held-out error is ±1.363. Clamped to the stretch each family’s members span, the lines still give twelve violet LEDs to the radiators and cost ±0.285. A line for the halophosphate family alone, beside the nearest-lamp rule for everything else, keeps the tube and claims a cyan LED instead, and costs ±0.179. Measuring each distance in its family’s own scatter comes in at ±0.139, two thousandths under the ratio line, having placed the tube with the LEDs.

• The prediction is right about the tube. Its siblings’ line passes within 0.06 of it, a quarter of the distance to any other family’s line.
• It is right about the pink-tinted radiator, which lies nearer the LEDs’ line than its own.
• It is wrong about the error, and the reason is not the tube. A line is a model of a family, and a model extends past the lamps it was fitted to. The radiators’ line runs along the LEDs’ own.

The picture is the abstaining essay’s map with the three lines drawn on it. The halophosphate line is short and steep: five tubes stacked almost vertically by their 640 nm departure, which is the channel the main band crosses. The LEDs’ line is long and shallow, from the violet-pumped LEDs at the right to the cyan-filled ones on the left. The radiators’ line is the one that does the damage. Six radiators — tungsten, a 4,000 K Planckian radiator, two daylights and two tinted sources — lie close together near the origin, and the line through them points almost the way the LEDs’ line does. Over its own short stretch it lies on top of the LEDs’; extended, it runs off to the right through the violet LEDs, and every lamp in its path is nearer it than to the LEDs’ own line, which bends away.

Why two lines lie on top of each other #

The pink-tinted radiator lies 0.005 from the LEDs’ line and 0.014 from its own. That is the prediction’s misplaced radiator, and it is misplaced for the reason the vote misplaced it: in the departures a slightly pink radiator and a white LED look alike. But it is more than one lamp. The radiators’ principal direction and the LEDs’ agree to within about sixteen degrees, and where the radiators sit, the LEDs’ line passes within five hundredths of every one of them — about as close as their own line does. A line fitted to the radiators is therefore very nearly a short piece of the LEDs’ line, and a line fitted to the LEDs passes straight through the radiators.

That is a statement about what the three departures can see, and it is worth putting plainly. The sensor’s departures place a smooth radiator and a phosphor-converted LED along the same direction, because both are broad, smooth spectra whose departures from the camera’s prediction vary mostly with how blue or yellow the light is — the same smoothness the lamps two channels miss are smooth to the camera found hiding lamps from the narrow channels. The families differ in where along that direction they sit, not in which direction they run, and a classifier that asks only which line is nearer cannot use where along a line a lamp sits.

The violet LEDs make the other half of the point. The violet-pumped LEDs at 435 to 475 nm sit at the right-hand end of the LEDs’ cloud, where the cloud bends away from the straight line fitted through all forty-four of them — the cloud is not a line, as the prediction feared. The radiators’ line, straight and fitted over a short stretch, does not curve, and passes nearer them: the 465/80 LED lies 0.026 from the radiators’ line, beyond the end of the radiators’ own stretch, and 0.043 from its own. Eighteen LEDs are closer to a line fitted to six radiators than to the one fitted to them.

What the misplaced lamps cost #

A misplaced lamp is estimated with the wrong family’s line from the departures to the error, and how wrong that is depends on how far the lamp sits from where the wrong family’s line was fitted. The radiators’ estimate is fitted over six lamps close together, and applied to lamps far along the radiators’ line it is an extrapolation of a three-coefficient fit well beyond its data. The triphosphor tube, placed on the radiators’ line, is missed by 7.6 ΔE00; the three-emitter source by 4.4; the cool LED by 2.8; the cyan LED at 475 nm with the lightest filter by 2.1. Those four carry nearly nine tenths of the census’s squared error, and they are lamps a vote placed correctly, or at least harmlessly.

This is the step the prediction did not take. It asked where the lines would send each lamp, and expected the answer to be decided near the lamps the families were fitted on. Two stages catch the tube and bill the lamps they misplace had already found that a misplacement costs in proportion to how badly the wrong family’s line fits the misplaced lamp. An infinite line misplaces lamps far from the family it belongs to, which is exactly where its family’s estimate is worst — so a line classifier’s errors are larger than a vote’s in kind, not only in number.

Clamping, scaling, and one line alone #

Three repairs suggest themselves, and the census ran each.

Clamp each line to its members’ stretch, so that a lamp beyond the radiators’ last member is measured to that member rather than to the line’s continuation. That removes the triphosphor tube’s and the three-emitter source’s worst misses, and it keeps the tube. It does not help the violet LEDs, which lie beside the radiators’ line within its stretch as well as beyond it; twelve of them still go to the radiators, three cyan LEDs go to the tubes, and the census costs ±0.285, twice the ratio line’s.

Measure each distance in its family’s own scatter, so that the tight radiators’ and halophosphate lines are not favoured over the loose LED line merely for being tight. The LEDs scatter about their line by 0.168 in the departures, the radiators by 0.020 and the tubes by 0.049; divided by those, the LED line becomes the nearest for almost everything. The violet LEDs return to their own family. All six radiators leave theirs for the LEDs’, and so does the halophosphate tube: its siblings’ line, fitted without it, passes 1.73 of their scatters away, and the LEDs’ line 1.55 of theirs. Scaled, the census comes to ±0.139, two thousandths below the ratio line, with the tube missed by 0.40. That is the vote’s cost structure again, arrived at by a different route, and two thousandths over fifty-four lamps is well inside what choosing the scaling after seeing the census could buy by accident.

Give a line only to the family that is one. The halophosphate family is a line by construction; the radiators and LEDs are not, or not usefully. So keep the nearest-lamp rule for every lamp, and override it only when the halophosphate line’s stretch passes nearer than the lamp’s nearest other lamp. The tube is 0.076 from its siblings’ line and 0.141 from its nearest other lamp, and is kept. But the cyan-filled LED at 485 nm is 0.140 from the line and 0.145 from its nearest lamp, and it is taken too — placed with the tubes, estimated with their line, and missed by 0.57. The census comes to ±0.179, worse than the plain vote’s ±0.162, because the line that catches the tube catches the lamps the tube’s family runs through, which is what the abstaining essay found in the first place.

No line does both #

Every line classifier that keeps the tube costs more than the ratio line alone. The only one that costs less has given the tube up, and costs less by two thousandths. The prediction’s condition — keep the tube and come in under ±0.141 with the rule fixed in advance — is met by none of the four. The nearest-lamp vote, which the lines were meant to improve on, sits between them at ±0.162, and the ceiling the whole design is reaching for, every family given, is ±0.133.

The pattern is the same one the vote showed, seen from the other side. A classifier that reaches far enough to catch the tube — whose own family runs through the cyan LEDs — reaches far enough to catch LEDs for the tube’s family, or for the radiators’. A classifier cautious enough to leave the LEDs alone leaves the tube with them. What separates the halophosphate tube from its LED neighbours in the three departures is where it sits along a direction they share, and neither a vote nor a line reads position along a direction.

What a sensor designer should take from it #

A family’s model is not its boundary. Fitting each family well — a line for a one-parameter family, a cloud for a family that varies in several ways — says nothing about where one family stops and the next begins. The lines here are good models of their families, and they are bad boundaries because two of them share a direction. The tube a third channel misses has no family found the same sensor unable to see a family for the tube; it can see the family, as a line, and cannot keep other lamps off it.

Three departures do not separate a smooth radiator from a phosphor LED by direction. The two families lie along one direction in the departures — the way the light is blue or yellow — and differ in where along it they sit. A channel that responds differently to a Planckian spectrum and a phosphor-converted one of the same colour, in the way two channels estimate what one can only sort found for the LEDs’ pump and fill, is what would give the radiators a direction of their own. Until then the radiators’ own line is a liability, and the estimate is safer without it.

Choose the rule before looking. Every classifier here was fixed before the census was scored, and the one that came in under the ratio did so by an amount a choice made afterwards could buy by accident; a classifier that abstains gives up the tube found a rule chosen after the fact looking better than a known family and worth nothing when chosen honestly. Two thousandths over fifty-four lamps is a result to report and not to act on.

How the census was taken #

The lamps, departures and estimates are those of the two-stage essay: fifty-four lamps and four halophosphate siblings, each lamp’s camera error and its 450, 500 and 640 nm channels’ signed departures from the camera’s prediction, and each family’s estimate a least-squares line from the ratio gap’s magnitude and the 640 nm departure to the error, fitted over the family’s other members if it has four or more and otherwise replaced by the census-wide ratio line. Each lamp is left out of every line it is classified by and of the estimate it is then given. A family with three or more other members — radiators, LEDs, halophosphate tubes — gets a principal line in the three departures: the members’ mean and the leading eigenvector of their scatter matrix, found by power iteration, with the stretch between its outermost members and the root-mean-square distance of the members from it. Distances are Euclidean in the three departures, to the infinite line, to its clamped stretch, or to the stretch divided by the family’s scatter; the halophosphate-only rule compares the distance to that line’s stretch with the distance to the lamp’s nearest other lamp.

What the census leaves out #

The families are the collection’s labels. The tinted radiators are called radiators because of how they were made; in the departures they are indistinguishable from LEDs, and a sensor has no way to know the label. A classifier trained on different labels — radiators and LEDs as one family of smooth spectra — would face a different problem, and this census does not try it.

A line is the simplest curve. The LED cloud bends at its violet end, and a curve fitted to it — a quadratic in the departures — would follow the bend and could keep the violet LEDs off the radiators’ line. With forty-four LEDs and three coordinates such a fit has room; with six radiators it does not.

The lamps are modelled. Every spectrum, the camera and the sensor are the collection’s models, and the departures carry no measurement noise. Noise would widen every family’s scatter and make every boundary here worse.

Still open: whether radiators and LEDs are one family #

In the three departures the radiators’ line is a short piece of the LEDs’, and the pink-tinted radiator lies nearer the LEDs’ line than its own. The labels treat them as two families; the sensor sees one.

The calculation is the two-stage estimate with radiators and white LEDs merged into one family of smooth broadband sources, classified by the nearest lamp as before and by a quadratic curve through the merged family, with the merged family’s own estimate fitted over fifty lamps rather than six and forty-four. The prediction is that merging removes the pink-tinted radiator’s and the daylights’ misplacement entirely, since there is no longer a boundary for them to cross, and that the merged estimate costs the radiators little, because their errors are small and the LEDs’ fit passes near them — bringing the census within a hundredth of the known-family ±0.133. If the merged fit costs the radiators more than the misplacement did, the families are genuinely two, and the sensor needs a fourth channel to tell them apart.

A model is not a map #

The habit here is about what fitting something well entitles a classifier to.

Each family’s line was a good description of its family — the tubes’ tight, the radiators’ tight, the LEDs’ loose — and the classifier used them as if a good description were also a good territory. It is not. A description says where a family’s members are; a territory has to say where nobody else is, and two families sharing a direction can each be described perfectly while overlapping everywhere their descriptions are extended.

The failure mode is to use a model of each class as a model of the space between classes. The test is to draw every class’s model on one map and look at where each one runs when it leaves its own members — because the lamps that get misplaced are the lamps out there.