Radiators and LEDs are one family to the sensor

A phone’s ambient-light sensor reads a few narrow channels, and a run of essays here has asked whether they can tell the camera how wrong its colour will be under the lamp in the room. The design that has come closest works in two stages. Two stages catch the tube and bill the lamps they misplace recognised a lamp’s family — radiator, white LED, halophosphate tube, other tube, three-emitter source — from how the sensor’s 450, 500 and 640 nm channels depart from the camera’s prediction, and applied that family’s own line from the departures to the error. Told each lamp’s family, it caught the halophosphate tube, which the single census-wide ratio line misses by nearly half a unit, and brought the held-out error over fifty-four lamps from ±0.141 to ±0.133. Left to find the families itself, from each lamp’s nearest neighbour, it misplaced five and cost ±0.162.

Two essays have tried to find the families better. A classifier that abstains gives up the tube let the vote abstain when unsure, and it abstained on the tube. A family’s line runs through its neighbours gave each family a line through its members and classified by the nearest, and found the reason none of this was working: in the three departures the radiators’ short line lies on the LEDs’ long one. The pink-tinted radiator sits 0.005 from the LEDs’ line and 0.014 from its own. The families differ in where along a shared direction they sit, not in which direction they run, and every classifier that tried to put a boundary between them misplaced lamps across it.

That essay closed by taking the sensor’s side. The labels treat radiators and LEDs as two families because of how the lamps are made; the sensor sees one. The prediction was that merging them into one family of smooth broadband sources removes the radiators’ misplacements entirely — there is no longer a boundary for a lamp to fall across — and costs the radiators little, because their errors are small and the LEDs’ fit passes near them, bringing the census within a hundredth of the known-family ±0.133. If instead the merged family’s line cost the radiators more than the misplacements had, the families were genuinely two and the sensor would need a fourth channel to tell them apart.

One family estimates better than two, known or found #

With the six radiators and forty-four white LEDs merged into one family of fifty, each lamp’s family found by its nearest other lamp as before, the two-stage estimate’s held-out error over the fifty-four lamps is ±0.127. That is below the ratio line’s ±0.141, below the separate families found by the same rule at ±0.162, and below the separate families given — every lamp’s true family handed to the estimate — at ±0.133. The halophosphate tube is still caught, within a hundredth and a half. No radiator or LED is placed outside the merged family; the only lamps placed elsewhere are the two other tubes and the three-emitter source, whose own families are too small to fit a line. And the radiators, which the prediction expected to pay a little for the merge, are estimated better inside it than inside their own family: ±0.177 against ±0.215.

• The prediction is right that the misplacements go, and right that the merge keeps the tube.
• It is wrong about the size, in the good direction. The merged estimate does not approach the known-family ±0.133; it passes it.
• It is wrong about the radiators. They gain from the merge, because their own family’s line was never good for them.
• The sensor’s grouping beats the label’s grouping. What looked like a classification problem was a labelling one.

The four bars are four ways of turning three departures into an estimate of the camera’s error. The ratio line uses one number, the gap between the 450 and 500 nm departures, for every lamp. The two-stage designs recognise a family first and then use that family’s line, which adds the 640 nm departure; they differ in what the families are and how a lamp’s is found. The merged design is the only one that is better than giving the estimate the right answer to the classification question — because the right answer, the lamps’ labels, was the wrong question.

Why the radiators do better with the LEDs #

A family’s estimate is a least-squares line with three coefficients — a constant, the ratio gap’s magnitude and the 640 nm departure — fitted over the family’s other members with the lamp being estimated left out. For a radiator that means three coefficients fitted over five lamps, and five lamps close together in the departures pin a three-coefficient line poorly: it passes near them and tilts wherever their scatter happens to point. Held out, it misses the radiators by ±0.215. The census-wide ratio line, fitted over fifty-three lamps with only two coefficients, misses them by ±0.164. A fit can be exact and empty put the general case: a fitted object reports its residual on its own data, and says nothing about how many of its parameters the data determine. Five radiators determine a three-coefficient line barely, and the chart decides the profile found the same thing for a camera’s matrix fitted on a chart with too little range: an error that is small where it was fitted and large everywhere else.

In the merged family the same line is fitted over forty-nine other smooth sources, and it misses the radiators by ±0.177 — close to the ratio line and far better than their own. The LEDs, who were forty-three in their own family and are forty-nine now, barely notice: ±0.112 against ±0.111. The merge is free for the LEDs and a gain for the radiators, because the radiators’ errors do lie near the LEDs’ line, as the prediction supposed; the prediction’s mistake was to assume the radiators’ own line was the baseline to beat.

Why a radiator looks like an LED #

The three departures are what the sensor’s narrow channels see beyond what the camera’s own broad channels predict. A lamp whose spectrum is smooth between 440 and 660 nanometres departs from the camera’s prediction mostly according to how blue or how yellow it is: a warmer lamp has less at 450 and more at 640 than the camera’s reading of its white implies, a cooler one the reverse, and the 500 nm channel follows the 450 at a slightly different rate. A Planckian radiator and a phosphor-converted LED of the same colour differ in their spectra’s fine structure — the LED has a pump peak and a phosphor hump — but the narrow channels at 450, 500 and 640 sample that structure only where it is broad, and the collection’s LEDs’ departures vary with their colour in almost the way the radiators’ do. The lamps two channels miss are smooth to the camera found the same thing from the other side: the lamps a two-channel sensor could not tell apart were the ones whose spectra were smooth where the camera looks.

That is also why the camera’s error follows one line across both kinds. The camera is a fourth observer, with broad channels of its own, and its colour error under a smooth lamp is set by how that smooth spectrum weights its channels against the eye’s — a quantity that varies with the lamp’s colour, again, much the same way whether the smoothness comes from a hot filament or a phosphor. The merge works because the physics that separates the two families is not the physics the camera’s error depends on. The pink-tinted radiator is the exception that shows it: its tint is a bump in the spectrum the narrow channels do see, and the camera’s error does not follow it.

Which lamps the merge helps #

The largest single gain is the violet-pumped LED at 465 nm with the widest pump, which its nearest neighbour in the departures placed with the radiators. Estimated with the radiators’ line — five lamps, extrapolated to a departure none of them reached — it was missed by 0.72; in the merged family by 0.07. That one lamp is most of what the separate families’ nearest-lamp classifier cost. The next largest are radiators: the 4,000 K radiator from 0.087 to 0.016, the 5,000 K daylight from 0.092 to 0.035, the green-tinted radiator from 0.030 to 0.004. Then a handful of violet LEDs whose miss falls by two or three hundredths, because the merged line through fifty lamps is a little better placed for them too.

Radiator by radiator the merged line is better than their own for four of the six. It is worse for the tungsten lamp and the 4,000 K daylight, by two or three hundredths, which the radiators’ own line happened to fit. The pink-tinted radiator is missed by four to five tenths whatever line estimates it — 0.51 by its own, 0.43 by the merged, 0.40 by the ratio line — and that is the census’s hardest lamp for reasons no family structure touches: its tint puts a departure in the sensor’s channels that the camera’s error does not follow.

What the sensor sees #

Drawn as one family, the radiators are simply part of the cloud: they sit near the smooth family’s line, among the LEDs, at the end where the light is warm. Nothing in the three departures sets them apart, and the earlier essays’ trouble was the attempt to make it. Two channels estimate what one can only sort found the LEDs’ pump and phosphor as a structure the narrow channels could read; a Planckian radiator has no such structure, and to the channels it looks like an LED with a very gentle pump.

The halophosphate line stands apart from that cloud as it always did, and the tube is caught by it in the merged design exactly as in the others — the merge changes nothing outside the smooth sources. The two tubes and the three-emitter source are the only lamps the nearest-lamp rule still places in another family, and the families they belong to have too few members to fit a line in any design; they fall back on the ratio line or are estimated with a neighbour’s, and cost what they cost before.

What those three cost is worth stating, because they are the census’s remaining classification errors. The triphosphor tube’s nearest lamp is the three-emitter source, a family of one, so it falls back on the ratio line and is missed by 0.14 — exactly as when its family is given, since its own family of two also falls back. The broadband tube’s nearest lamp is a smooth source, and the smooth family’s line misses it by 0.10 against the 0.08 its fallback would have given. The three-emitter source goes to the smooth family too and is missed by 0.33, a thousandth better than its fallback. Between them they add a few thousandths to the census’s error, and no grouping of radiators and LEDs could have changed them: they are lamps with no family large enough to learn from, and they will stay that way until the census holds more tubes and more narrow-emitter sources.

What a sensor designer should take from this #

Group lamps by what the sensor sees, not by how they are made. A family is useful to an estimate if its members’ errors follow the same line in the sensor’s own readings. Radiators and LEDs do; the label that separates them is about physics the three channels cannot resolve, and an estimate built on it pays for the resolution it cannot make.

A family needs members. Six radiators were never enough to fit a three-coefficient line honestly, and the census’s own “family given” design — the benchmark every classifier was measured against — had a weak member in its radiators all along. The halophosphate family works with five members only because the tube and its siblings lie on a clean line that one parameter generates.

Choose the grouping from the inputs, and score it on held-out errors. The merge was proposed from the departures alone — where the lamps sit, before any error was looked at — and every estimate here is made with the lamp left out. That is not the same as a grouping chosen before the census existed: the departures are this census’s, and a new lamp of a kind none of the fifty-four represents could sit anywhere. A classifier that abstains gives up the tube found a rule chosen after the fact worth nothing; this one was chosen on the inputs, and its advantage over the ratio line, fourteen thousandths over fifty-four lamps, is modest enough to state with that caveat.

How the census was taken #

The lamps, departures and estimates are those of the two-stage essay: fifty-four lamps and four halophosphate siblings; each lamp’s camera error and the signed departures of the sensor’s 450, 500 and 640 nm channels from the camera’s prediction; each family’s estimate a least-squares line from the ratio gap’s magnitude and the 640 nm departure to the error, fitted over the family’s other members if it has four or more and otherwise replaced by the census-wide ratio line from the gap alone. The only change is the labels: the six radiators and forty-four white LEDs carry one label. A lamp’s family is found by its nearest other lamp in the three departures, or given. Every lamp is left out of its own classification and of the line it is estimated with. Outside the merged family, with families given, every lamp is estimated exactly as the separate families estimated it, which is checked.

What the census leaves out #

The lamps are modelled. The radiators are Planckian and daylight spectra with two tinted variants; the LEDs are pump-and-phosphor models. Real radiators include incandescent lamps behind coloured glass, and real LEDs include violet-pumped and multi-phosphor designs outside the census’s range. Either could sit off the smooth family’s line.

Fifty-four lamps. Every comparison here is between held-out errors over the same fifty-four, and differences of a few thousandths are within what a different set of lamps could move.

Noise-free departures. A real sensor’s channels are noisy, and noise would scatter both families about their line; the merge’s advantage, which comes from fitting one line over more lamps, should grow with noise rather than shrink.

Still open: whether a new lamp falls off the smooth line #

The merged family’s line was fitted to the lamps the census has. The case that would break it is a smooth-looking lamp whose camera error does not follow that line — a tinted incandescent, a two-phosphor LED — placed confidently in the smooth family by its nearest neighbour.

The calculation is this census with a second set of lamps held out entirely: a dozen spectra the census has not seen — incandescent lamps behind blue and pink glass, two- and three-phosphor LEDs, a filtered daylight — estimated by the merged design fitted on the fifty-four. The prediction is that the new smooth lamps are estimated within the census’s own error, ±0.13, because they share the smooth family’s structure, while the tinted incandescents fall off it as the pink-tinted radiator does, by four tenths or more. If so, the smooth family is a real family for any lamp whose spectrum is smooth, and the tint is the one thing the three departures cannot read.

A benchmark is a design too #

The habit here is about what a comparison is measured against.

Every classifier in this run of essays was scored against the families given — the error the two stages reached when told each lamp’s true label — as if that were the ceiling a classifier could approach. It was a design choice like any other, and not a good one for the radiators: their own label gave them a line too short to be trusted.

The failure mode is to treat the best case of one design as the best case of the problem. The test is to ask whether the benchmark’s own assumptions — here, that the labels are the right groups — have been tested, before measuring anything against it.