Flicker sorts lamps the wrong way

Two matrices do not reach a white LED found that the blend a raw profile computes from the scene’s colour temperature is exact along the daylight family and about twice as wrong as it needs to be under any lamp with lines in it, and that the information deciding which case a photograph is in is exactly the information a white balance reading discards. Three numbers cannot see a line is why: a tube and a smooth lamp of the same chromaticity give a camera the same three numbers.

So the information has to come from somewhere else, and an image does not determine the light rules out getting it from the picture itself. That essay named three candidates. The first was flicker, because the shutter samples the lamp and a camera can measure it for nothing.

Two properties of a lamp, chosen by different people #

Flicker is measurable and it measures the wrong property. How a lamp delivers its power and how its spectrum is shaped are independent, so a classifier reading the first is wrong about the second in both directions — and the commonest indoor lamp is one of the mistakes.

• Of twelve fixtures the classifier gets seven right, misses three and false-alarms on two.
• The three misses are structured lamps that do not flicker: a white LED and a warm LED on constant-current drivers, and a three-emitter fixture. Handed the blend they were not built for, they cost 1.06, 1.18 and 0.37 colour differences more than the better of the two matrices.
• The worst single mistake is a false alarm. A halogen lamp on mains flickers through its own filament, is called structured, and is handed a matrix fitted under a white LED — 1.99 colour differences worse than the blend it should have had.
• Raising the threshold removes every false alarm and no misses, because a miss is a lamp whose flicker is exactly zero and no threshold is below zero.
• At its ordinary setting the classifier is worse than not choosing at all: 0.435 of regret against 0.419 for always using the third matrix.

What a shutter can see #

A lamp fed from the mains is fed a full-wave rectified sine, so its power arrives at twice the supply frequency. What reaches a camera depends on what converts that power into light.

A filament has thermal inertia, and eight milliseconds of it against a ten-millisecond cycle, so it smooths the supply almost flat — a spread of 26 per cent where the raw supply is 151. A discharge tube’s phosphor has a persistence of about three milliseconds and smooths it partly, to 62. A diode has neither, so an LED wired straight across a rectifier delivers the supply itself, and one on a pulse-width dimmer delivers a square wave. A constant-current driver delivers a flat line, and so does a window.

A camera measures this by comparing readings at two shutter durations. It does not choose its shutter phase against the mains, so the observable is not one reading but the spread over every shutter phase it might have opened at — which is what a burst of frames at a fixed exposure actually shows.

The difference between the two is the classifier’s statistic, and it is a clean measurement: exactly zero for a constant driver and 0.99 for a dimmer, with the filament at 0.26 and the phosphor at 0.62 in between. Nothing here is a limitation of the measurement. It works.

The measurement is also cheap in a way that matters for whether anybody would do it. Two shutter durations at a fixed aperture is a pair of frames, it costs no extra hardware, and a camera that runs an autofocus burst has already taken them. A classifier’s cost is not why this one fails, which is worth saying before the failure, because the usual reason a proposed measurement is not adopted is its cost and that is not the reason here.

The two classifications, side by side #

The measurement works and the classification does not, and the reason is that a fixture is two choices made by two people. Somebody chooses the emitter, which fixes the spectrum. Somebody else chooses the driver, which fixes the flicker. A shop sells every combination, which is the same shape of trouble a lamp is not a blackbody found in the colour temperature printed on the box.

A white LED on a constant-current driver is steady in time and strongly structured in wavelength. Its flicker signature is exactly zero, the classifier calls it smooth, and it gets the two-matrix blend — 1.96 colour differences against 0.89 for a matrix fitted under it. That is the commonest indoor lamp in a new building, and it is the case the third matrix exists for.

A halogen lamp on mains is the opposite mistake. Its spectrum is a thermal radiator’s, which is the smoothest light there is; its filament still passes a quarter of the supply’s modulation; the classifier calls it structured and hands it a matrix fitted under an LED. That costs 1.99 — the largest regret in the table, from a lamp the blend handles nearly perfectly.

And a cheap LED on rectified mains is a hit. It flickers at 151 per cent and its spectrum has a phosphor trough in it, so both classifications agree, and it gets the third matrix and 1.01 instead of 1.92.

So the classifier is not noisy. It is reading a different variable, and it is reading it accurately.

Why the two properties are independent, and not merely observed to be #

The cross-tabulation is a measurement over twelve fixtures, and a measurement over twelve of anything invites the question of whether a thirteenth would behave. The answer is that the independence is structural.

A lamp’s spectrum is fixed by what emits the light. A tungsten filament emits a thermal continuum because it is hot; a discharge emits mercury’s lines because that is what excited mercury does; a phosphor emits a broad band because that is what a phosphor does with a photon it has absorbed. All three are properties of a material.

A lamp’s flicker is fixed by what feeds it. A rectifier, a filter capacitor, a constant-current regulator and a dimmer are circuit elements, and any of them can be put in front of any emitter. The only coupling between the two is through the emitter’s persistence — a filament smooths its own supply and a diode cannot — and that coupling runs the wrong way for a classifier: the emitter with the smoothest spectrum is the one with the most persistence, so the lamp least in need of a third matrix is the one most likely to look steady, and the lamp most in need of one is a diode that follows whatever its driver does.

That is why raising the threshold helps at all — it is exploiting the one real correlation, that filaments smooth — and why it cannot reach the misses, which are on the other side of a coupling that does not exist.

No threshold fixes it, and the reason is an asymmetry #

The obvious response to a classifier that is wrong in both directions is to move its dividing line. Half of that works.

Every threshold from a two-hundredth to two fifths leaves the same three misses. Raising it past a filament’s 0.26 removes both false alarms and halves the regret, from 0.435 to 0.218 — which is better than always using the third matrix, so a tuned classifier is worth having. And it never removes a miss, because a miss is a lamp whose flicker is exactly zero and no threshold sits below zero.

That asymmetry is the shape of the failure. A false alarm is a smooth lamp whose flicker is above the line, and the line can be moved above it. A miss is a structured lamp with no flicker at all, and nothing about where the line sits reaches a measurement of zero. The floor of what tuning can achieve is exactly the regret of the lamps that do not flicker, computed directly and equal to the best threshold’s remaining regret to machine precision.

At its untuned setting the classifier is worse than not choosing: 0.435 against 0.419 for always using the third matrix, because the false alarms cost more than the hits save. That is worth stating plainly, since a classifier is usually adopted on the strength of being better than nothing.

What the mistakes cost, one by one #

The regrets are not all the same size and the pattern in them is useful.

The false alarms cost more than the misses. A halogen lamp handed the LED’s matrix reaches 3.19 against 1.20 for the blend; a tinted radiator, 1.81 against 1.20. The misses cost 1.06, 1.18 and 0.37. A matrix is fitted under one light is where those two matrices come from and why the blend between them is exact where it is. A smooth lamp given a matrix fitted under a structured one is further from home than a structured lamp given the blend, because the blend was built from two smooth calibrations and a smooth lamp is what it is for.

And the three-emitter fixture is the interesting miss. It is the most structured light in the set — three narrow bands and nothing between them — its own matrix reaches 0.33, the best of any lamp here, and it sits on a constant driver. The classifier calls it smooth and it costs 0.37, which is small only because the third matrix is not much better for it than the blend is: a matrix fitted under a phosphor LED is not a matrix for three narrow bands. A third matrix is a class, not a lamp is the point that essay made, and it limits what any classifier is worth here: even a perfect classifier only chooses between two matrices, and neither of them is right for that fixture.

What would work instead #

The measurement to make is the one the classification actually needs, and the list is short.

A spectral sensor with more than three channels reads the structure directly. Six or eight channels cannot reconstruct a spectrum — the colour is right first is the standing limit on that — and they do not have to: distinguishing a smooth spectrum from one with narrow bands is a much easier question than recovering one, and a handful of channels with different widths answers it.

The camera’s own white against a second sensor’s is the same idea assembled from parts a phone already has. Two devices with different spectral sensitivities reading one lamp disagree in a way that depends on the lamp’s structure, and the disagreement is a statistic with no extra hardware in it.

And a photographer naming the lighting is the one that has always been available. It is the only route on the list that carries no measurement error at all, and it is the one every camera has quietly removed from its interface.

What flicker gives, after tuning, is a genuine improvement on always guessing — half the regret — and it is not the measurement the question asks for. A classifier that is right about a different question can still be useful, and the right way to describe it is as a different question rather than as a noisy answer to this one.

How the fixtures and the signature were computed #

The twelve fixtures pair a spectrum from the two-matrix work with a drive of their own. The spectra are closed-form: thermal radiators, daylights, a radiator with a broad tint, white LEDs with and without a red phosphor, three phosphor mixtures with mercury lines, and a three-emitter source. The drives are a constant current, a pulse-width dimmer at 480 Hz, rectified mains at 100 Hz, and rectified mains through a filament or a phosphor, each modelled as a one-pole smoothing at that emitter’s own time constant.

The shutter reading is the waveform integrated over a window of the stated duration, taken at every shutter phase the record holds, and the observable is the spread between the largest and smallest of those readings as a share of their mean. The classifier’s statistic is that spread at half a millisecond minus the spread at forty.

The cost of a choice is read from the two-matrix work directly: blend is what the profile’s two matrices give at the weight the lamp’s colour temperature chooses with the neutral held, and third is what a matrix fitted under a neutral white LED gives on the same surfaces. Regret is the chosen matrix’s error minus the better of the two, which is zero whenever the classifier is right.

What this leaves out #

The drives are models. Real drivers cover a continuum from a well-filtered supply to a bare rectifier, and a real fixture’s modulation depends on its phosphor, its current and its age. What the model fixes is the ordering — a filament smooths more than a phosphor, which smooths more than a diode — and that ordering is physics rather than a parameter.

The classifier is a threshold on one statistic. A better one could use the modulation’s frequency as well as its depth, and a pulse-width dimmer’s 480 Hz is distinguishable from a mains 100 Hz. That would separate the drives further and would not move a single lamp across the spectral line, because the spectral line is not a fact about drives.

And the twelve fixtures are a set. A room lit by several at once is the ordinary case and is not modelled here; a room with two lights has no white is the appearance side of that problem, and its imaging side would ask which lamp’s flicker a camera reads when two are present.

Still open: whether two sensors disagreeing is enough #

The classifier this essay rules out was the cheapest of the three the earlier work named. The cheapest of the remaining two is the one that needs no new component: a phone already carries an ambient-light sensor with its own spectral sensitivity, quite different from the camera’s, and the two read one lamp.

The question is whether the disagreement between them separates the classes the matrix needs. It has the right shape — two devices whose sensitivities are not linear transforms of each other report different chromaticities for the same light, by an amount that depends on the light’s structure, which is observer metamerism with instruments in place of people. Whether it separates a phosphor LED from a smooth lamp of the same chromaticity, at what rate of mistakes, and whether the ambient sensor’s own calibration drift swamps it, is a computation on two measured sensitivity sets not in hand here.

The prediction worth recording is that it will work and will be fragile: the statistic is a difference between two small quantities, both of which drift with temperature and age, and a classifier whose input is a difference of two calibrations is a classifier that needs recalibrating.

A measurable property is not the property #

The habit is about what happens when a needed quantity is unmeasurable and a measurable one is nearby.

The reasoning goes: this decision needs X, X cannot be measured here, Y can be measured and Y is associated with X, so measure Y. Each step is reasonable and the conclusion is often wrong, because associated with is doing work that has not been checked — and the way to check it is not to ask how accurately Y is measured but how tightly Y and X are tied.

The move is to build the cross-tabulation before building the classifier. List the cases, mark each with its true class and with what the proxy says, and look at the off-diagonal. It takes an afternoon, it needs no implementation, and it answers a question no amount of improving the measurement can.

The failure mode is to improve the proxy. Every threshold, every extra statistic and every better shutter makes the flicker measurement better, and none of them moves a lamp across the line that matters, because what puts it on the wrong side is that somebody chose its driver and its emitter separately.