A classifier that abstains gives up the tube

A phone’s ambient-light sensor reads a few narrow channels, and the question a run of essays here has asked of it is whether those channels can tell the camera how wrong its colour will be under the lamp it is standing under. Two stages catch the tube and bill the lamps they misplace tried the obvious design. Recognise which family of lamp this is — radiator, white LED, halophosphate tube, other tube, three-emitter source — from how the sensor’s 450, 500 and 640 nm channels depart from what the camera predicts, then apply that family’s own line. Told each lamp’s family, the two stages worked: they caught the halophosphate tube, which every one-line estimate had missed by 0.445 ΔE₀₀, to within 0.011, and brought the census’s held-out error from the ratio’s ±0.141 to ±0.133. Left to find the families by each lamp’s nearest neighbour, they misplaced five lamps and paid ±0.162 for them — worse than not classifying at all.

The five were placed confidently: each had a nearest neighbour, and each neighbour belonged to another family. The repair that essay proposed is the standard one for a classifier that is sometimes wrong. Let it abstain. Look at a lamp’s three nearest neighbours, and only if all three share a family use that family’s line; otherwise fall back on the census-wide ratio line, which is the safe estimate for a lamp the classifier cannot vouch for. The prediction was specific: the pink-tinted radiator, the two small-family tubes and the three-emitter source would abstain, because their neighbourhoods are mixed; the violet LED at 465 nm would still land among the radiators; and the census’s error would fall to about ±0.135, most of the way to the known-family ±0.133, with the tube keeping its small miss.

The classifier abstains on the lamp it was built for #

With three neighbours required to agree, three of the five lamps the nearest-neighbour classifier misplaced are still misplaced — the pink-tinted radiator, the broadband tube and the three-emitter source — and among the lamps that abstain is the halophosphate tube: its two nearest neighbours are its siblings and its third is a cyan-filled LED. Its miss goes back from 0.011 ΔE₀₀ to 0.445, and the census’s error comes to ±0.142 — the ratio alone’s, to within a thousandth. Two agreeing neighbours instead of three give ±0.128, better than knowing every lamp’s family. But that number is chosen after seeing the census; chosen for each lamp from the other lamps alone, the rule gives ±0.141, and when the tube is the lamp left out, the other lamps choose six neighbours and the tube is missed.

• Confidence is not where the errors are. The misplaced lamps sit inside a cloud of LEDs and have three agreeing neighbours; their mistakes are invisible to any vote among neighbours.
• The tube’s family is spread thin. Its siblings are further from it than several cyan LEDs are, so any rule strict enough to be safe abstains on it.
• The best rule is learned from the tube. Only the tube’s own presence makes two neighbours better than six, so a rule chosen without it abstains on it.
• The prediction was right about two lamps and wrong about the rest: the triphosphor tube abstains as predicted, and the violet LED abstains where it was predicted to stay misplaced; the census’s error with an abstaining classifier is the ratio’s.

The figure is the whole census of rules. One neighbour is the earlier classifier, lamp for lamp. Two agreeing neighbours dip below the dashed line where every family is given. From three on, every rule that asks all its neighbours to agree loses the tube and settles near the ratio alone; majority rules do no better, and at three neighbours a majority rule abstains on nothing and misplaces eight lamps.

Why the misplaced do not abstain #

The pink-tinted radiator’s three nearest lamps are three violet-pumped LEDs, at distances of 0.041, 0.051 and 0.052 in the three departures; the nearest radiator is further than all three. The broadband tube’s three nearest are violet LEDs at 0.159 to 0.185. The three-emitter source’s are a cool LED, a neutral LED and a cyan-filled LED, far away — 0.40 to 0.48 — because nothing in the census is like it, and the nearest things to nothing are LEDs, because the census holds forty-four of them. Every one of the three has neighbours that agree with each other and disagree with it, so a vote among them places it with the LEDs as firmly as one neighbour did.

That is not a flaw in the rule. It is what a nearest-neighbour vote measures: whether a lamp’s neighbourhood is mixed, not whether the lamp belongs to it. A lamp that sits inside another family’s cloud has an unmixed neighbourhood. The pink-tinted radiator is a radiator whose tint filter moves its short-wave departure exactly where a violet pump moves an LED’s, and to three narrow channels it is a violet LED — the same kind of blindness a narrow channel has to be read on its own found when a channel was pooled with the others instead of read as a direction. Flicker sorts lamps the wrong way found the same kind of thing with another classifier: a property that sorts lamps cleanly and sorts them by something other than what the camera needs.

The violet LED at 465 nm with the longest phosphor tail is the one prediction that half holds. Its nearest neighbour is the pink-tinted radiator, and the nearest-lamp classifier put it with the radiators and missed it by 0.717. With three neighbours its neighbourhood is mixed — the radiator and two violet LEDs — and it abstains, and the census line misses it by 0.101. It was predicted to stay misplaced and instead it is rescued.

The tube’s own family runs through the LEDs #

The halophosphate tube’s family is itself and four siblings: the same mercury lines on the same two broad phosphor bands, with the main band moved to 558, 568, 588 and 598 nm instead of the tube’s 578. The tube a third channel misses has no family built them so the census would have something to learn the tube from. The two with the main band ten nanometres either side are the tube’s two nearest lamps. The two with it twenty nanometres away are its eighth and ninth, behind five cyan-filled LEDs at 0.165 to 0.252.

The siblings move along a line in the sensor’s departures as their band moves — the long-wave departure one ratio reads a peak and a trough showed the tube’s error following — and the tube sits in the middle of that line — which is why, told its family, a line through four siblings interpolates the tube’s error to 0.011. The cyan LEDs sit close to the same line, because a cyan filler puts light between 470 and 500 nm where the halophosphate coating’s short band does. So the family is a thread and the LEDs are a cloud it passes through, and any rule that asks three or more neighbours to agree finds the thread interrupted. A vote cannot tell a family spread thin along a line from a lamp with no family at all.

The same is true the other way round. With three neighbours the cyan LEDs that sit nearest the thread abstain too, because two of their three neighbours are halophosphate tubes; they lose the LED family’s line and take the census line, and for them that is slightly better — their misses fall from 0.30, 0.10 and 0.11 to 0.27, 0.07 and 0.08, because the LED line serves the cyan-filled edge of the LED cloud poorly.

Where the rule abstains #

Twelve lamps abstain under three agreeing neighbours: five violet-pumped LEDs on one side of the LED cloud where it meets the radiators, three cyan-filled LEDs on the other where it meets the halophosphate thread, two daylights at 4000 and 5000 K whose third neighbour is an LED, and both of the census’s tubes. The map shows what abstention does in general: it gives up the lamps at the boundaries between families. The misplaced lamps are not at a boundary. They are inside the LED cloud, and the three circles are the ones a vote can never reach.

For most of the violet LEDs and the daylights abstaining costs a few hundredths: their family lines had served them to within a few hundredths, and the census line serves them to within about a tenth. Two of the seven — the violet LED at 475 nm and the 5000 K daylight — do slightly better on the census line. For the triphosphor tube it changes nothing, because its family of two was too small to fit a line and it was always given the census line. For the halophosphate tube it costs everything the two stages were for.

Two neighbours, and what choosing them costs #

Looking across the sweep, two agreeing neighbours give the census’s smallest error, ±0.128, better than the ±0.133 of being told every lamp’s family. The rule abstains on seven lamps — the triphosphor tube, three violet and three cyan LEDs — misplaces the same three the vote can never reach, and keeps the tube, whose two nearest neighbours are both siblings. It beats the known families because it abstains on the lamps the LED family’s line fits worst: the cyan edge and the violet edge, where the census line happens to do better, and the long-tailed violet LED, rescued from a miss of 0.717.

That number was chosen by looking at the census. Six rules were tried, the one with the smallest error over the fifty-four lamps was picked, and the error it is quoted with is the same error it was picked by. Two channels estimate what one can only sort and every estimate since have been careful to leave each lamp out of the line that estimates it; picking the rule is a fit as well, and a lamp has to be left out of that too.

Chosen honestly, the rule is worth nothing. For each lamp in turn, the number of neighbours was chosen by the error it gives over the other fifty-three, each of them left out in turn from what remains, and only then applied to the lamp. Fifty-one lamps are estimated with two neighbours and three with six, and the census’s error is ±0.141 — the ratio alone’s, to the thousandth.

The three are where the whole difference lies, and one of them is the tube. With the tube left out, the other lamps no longer reward a rule for keeping it: over the other fifty-three, two neighbours give ±0.128 and six give ±0.124, because the one lamp six neighbours miss badly is no longer among them. The tube is then estimated with six neighbours, whose votes include LEDs, and it abstains and is missed by 0.445. The rule that catches the tube is learned only from the tube. That is the finding the tube a third channel misses has no family made about a third channel, arriving again one stage later: a lamp left out of a census is estimated from what the census holds without it, and without the tube the census holds no reason to protect the tube.

What a sensor designer should take from this #

Abstention is a remedy for a classifier that is unsure, and this one is sure and wrong. The misplaced lamps are the ones whose spectra make them look like another family to three narrow channels, and more neighbours only make the look more certain. What would separate them is a channel that sees the difference — a pink tint filter’s long-wave shape, a tube’s mercury lines — not a stricter vote.

A thin family needs to be recognised by its shape, not its neighbours. The halophosphate thread is easy to describe: a line through the departures, parameterised by where the main band sits. A classifier that fitted such lines and asked whether a lamp falls on one would catch the tube where a vote gives it up. That is a model of the family, which a nearest-neighbour vote deliberately does without.

Any rule chosen on the census has to be chosen without the lamp it estimates. The difference between ±0.128 and ±0.141 is the whole of what an abstaining classifier appears to buy, and the whole of it is the selection. A sensor designer tuning a classifier on a lamp library should expect the same: the library’s error is optimistic by whatever the tuning learned from the lamps it is scored on, and with one halophosphate tube in the library, that can be all of it.

The LED family is too broad for one line, and has been since two matrices do not reach a white LED found a camera profile’s daylight blend failing under LEDs by amounts that depend on the pump. Seven of the twelve lamps that abstain under three neighbours are LEDs at the violet and cyan edges of their cloud, and for most of them the census line is as good as or better than the LED line. The lamps two channels miss are smooth to the camera found the camera treating LEDs as one kind of error and the sensor seeing several; that is still the case, and the two-stage estimate’s second stage should probably hold more than one LED line.

How the census was taken #

The census is the earlier essay’s exactly: fifty-four lamps — radiators and tinted radiators, white LEDs including generated violet- and cyan-pumped ones, the halophosphate tube and two other tubes, and a three-emitter source — and the tube’s four siblings, which are used for fitting and classifying and never scored. Each lamp’s camera error is the ΔE₀₀ a camera’s fixed matrix makes on a set of surfaces under it, against a standard observer, which is the disagreement a camera is a fourth observer measures; its departures are how far each of the sensor’s 450, 500 and 640 nm channels reads from what the camera’s own white predicts. The family line is the camera’s error on the absolute ratio departure and the 640 nm departure, fitted on the family’s other members; the census line is the error on the ratio departure alone, fitted on the other census lamps. A family with fewer than four other members is not fitted and its lamps take the census line.

The classifier takes a lamp’s k nearest other lamps by Euclidean distance in the three departures, with k from one to six; under the unanimous rule the family is used only if all k share it, under the majority rule if more than half do. Every lamp is left out of its own neighbours and of every line applied to it. With one neighbour the estimate equals the earlier nearest-lamp estimate to the last digit, which is checked. The honest choice of k is nested: for each lamp, every k is scored by the held-out error over the other census lamps, each estimated with the lamp and itself both left out, and the best k is then applied to the lamp.

What this leaves out #

The census is small and its families are unequal, and its lamps were chosen to span the kinds a lamp is not a blackbody set side by side rather than in the proportions they are sold in. Forty-four LEDs, six radiators, five halophosphate tubes counting the siblings, two other tubes and one three-emitter source. A vote among neighbours in a census this unbalanced is a vote the LEDs usually win, and a larger library of tubes and tinted lamps would change which lamps are sure.

The siblings are constructed. Four halophosphate tubes with the main band moved in ten-nanometre steps are a family by design. Real halophosphate coatings differ in more than one band’s position, and a real family’s thread would be thicker; whether it would still run through the LEDs is not known.

Only one kind of abstention was tried. Falling back on the census line when the vote is split is the simplest. A classifier that abstained by distance — when its nearest neighbour is further than some threshold — would treat the three-emitter source, whose neighbours are all far away, differently from the pink-tinted radiator, whose neighbours are all close.

Still open: whether a family’s line can be the classifier #

The halophosphate family is a line in the departures, and so, to a first approximation, is every family whose members differ by one continuous parameter — a phosphor band’s position, a filter’s strength, a pump’s wavelength. A classifier that fitted each family’s line in the departures, and assigned a lamp to the family whose line passes nearest it, would see the halophosphate family whole where a vote sees it in pieces.

The calculation is that classifier on this census: each family’s members fitted with a straight line through the three departures, a lamp left out assigned to the family whose line passes nearest, and the two-stage estimate scored as before. The prediction is that it keeps the tube, because the tube sits on its siblings’ line whatever LEDs are nearer, and that it still misplaces the pink-tinted radiator, because the LED family’s cloud is too wide for a line and the radiator sits in it. If it keeps the tube and the census’s error falls below ±0.141 with the rule fixed in advance, the family structure is usable; if the LED cloud’s line swallows the tube’s, the families need more than three channels to be told apart at all.

A vote measures its voters #

The habit is about what a confidence measure is a measure of.

An abstaining classifier is sold as one that knows when it does not know. What it knows is whether its neighbours agree, and that is a property of the neighbourhood rather than of the lamp. A lamp that has wandered into another family’s territory has a quiet, unanimous neighbourhood and is placed wrongly with complete agreement; a lamp whose family is sparse has a noisy one and is given up though it was right.

The failure mode is to read agreement as correctness. The test is to ask, before trusting a confidence measure, which of the known mistakes it would have caught. Here it caught none of the five, and gave up the one success the method was built for.