The corpus › Decisions

The widening rule

A decision record · the-widening-rule · cited by 2 pages

The projection band widens as horizon^0.65 rather than as a square root — the second parameter in this repository replaced by a fitted value, and the first fitted against what a band is for rather than against what it is made of.

Context Contents

project::series::Prior::spread draws every interval this repository publishes: z * sigma * horizon^k. sigma is the cross-sectional spread of district annual enrolled-ADM growth, recomputed from the panel with its provenance printed beside each interval. k was 0.5, and said why: “Growth errors compound, so the band widens with the square root of the horizon rather than staying fixed — the standard random-walk result.”

That is a claim about school districts, and it had never been checked. #289 checked it. The band is a floor, as the module doc says — realized out-of-sample error is wider than the interval at every horizon from one year to five, under the shipped method and the one it replaced. But it is a floor that sinks:

horizon    coverage under horizon^0.50
1 year               67.3%
2                    66.1%
3                    63.8%
4                    60.6%
5                    55.8%

against the 68.3% a one-sigma interval claims. At one year the prior is very nearly calibrated — a spread taken across districts turns out to be about the right size for one district’s one-year error — and everything the rule was doing wrong, it was doing to the years after the first.

The mechanism is that forecast errors are positively autocorrelated: a district whose trend is misjudged stays misjudged, so the random-walk assumption that each year’s error is drawn fresh is what a school district does not do.

The decision Contents

Set the exponent to 0.65, as project::series::HORIZON_EXPONENT, mirrored by HORIZON_EXPONENT in web/src/lib/project.ts.

horizon    horizon^0.50    horizon^0.65
1               67.3%           67.3%
2               66.1%           70.4%
3               63.8%           71.2%
4               60.6%           68.4%
5               55.8%           66.5%
worst gap      12.5 pts         2.9 pts

Fitted against coverage, not against dispersion. Matching the standard deviation of the error gives 0.60 and still leaves the band mis-covering, because the error distribution is not normal — the two targets would agree if it were. Coverage is what a band promises, so coverage is what was fitted.

sigma does not move. It is a measured quantity with a stated source, and replacing it with a backtest constant would trade provenance for fit. Only the shape in the horizon was wrong, and a band that is the wrong shape cannot be corrected by rescaling: it is wrong by a different amount in every year.

Consequences Contents

The published FY2032 bands widen. Central estimates do not move at all — the exponent touches only the interval:

current law         $7,233M   $7,017M-$7,494M  ->  $6,958M-$7,582M   ±3.3% -> ±4.3%
guarantee removed   $6,301M   $5,938M-$6,686M  ->  $5,831M-$6,809M   ±5.9% -> ±7.8%

The finding those figures carry survives: the guarantee-removed band is 1.80 times as wide as the current-law one, against 1.79 before, so “removing the guarantee nearly doubles the state’s exposure to enrollment forecast error” is unchanged.

Four of the figures #291 bound moved, and the manifest said so before anything else did — the_manifest_reproduces_its_pins failed naming the key, both values and the distance, on the first run after the exponent changed. That is the mechanism working on the next change after it was built, rather than in principle.

Math.pow and Rust’s f64::powf agree to twenty places on every integer horizon this feed publishes, so the exponent could be an exponent rather than a lookup table and the feed reproduction check is unaffected.

Amendment Contents

Settled, by #391. The fit used horizons one to five, and that was a choice rather than the panel’s limit: the same six origins reach eleven years, and the two the panel supports but the fit did not use — FY2011 and FY2012 — take it to thirteen. Run out there, coverage of the cross-district error stays within 3.0 points of 68.3% at every horizon from one to thirteen, and refitting to coverage over the longer clean range returns 0.65 again. The FY2032 leg and the feed’s ten-year horizon are inside the measured range now. Measured in crates/project/tests/the_horizons_the_backtest_stopped_short_of.rs.

The alternative this record rejected as “the obvious move and the wrong target” turns out to have been partly a window effect as well. The dispersion exponent refits to 0.573, 0.600 and 0.623 over one to five, seven and nine pre-closure years and 0.650 over the whole reach — it climbs toward the coverage exponent as the range extends. The two targets still differ, and for the reason given, but 0.60 was never a measurement of anything past five years.

The sentence below was the binding one, and it is settled too, by #423.

Whether the band should be asymmetric was not addressed. It is multiplicative, so it already is in dollars, but the log error has a growing positive mean — see the_bias_no_single_damping_can_remove — and an interval centered on a biased point estimate inherits that bias whatever its width. That was the whole of the long-horizon defect. Pooled across origins the band’s coverage sinks from 67.1% at five years to 60.1% at thirteen, and removing each origin’s own mean restores all of it: the width is right and the level is not. The origin effect is the shutdown — at five years the origins landing on FY2020 or earlier are very nearly unbiased while the two landing on FY2023 and FY2024 run +0.035 and +0.038 — and sigma, being a cross-sectional spread, cannot price a statewide level break at any exponent. Mean log error reaches +0.056 at the feed’s ten-year horizon, so the point sits about 5.8% high.

Neither the center nor the band moves, and the bias is published instead. crates/project/tests/the_bias_that_belongs_to_the_years.rs measured the four remedies #423 named. The 5.8% is the mean district’s; the statewide total’s is 3.2% across the closure and within 0.7% at every horizon to seven years before it, with a sign that changes from origin to origin, so one correction cannot serve the two figures the feed publishes from the same projections. Sorted on anything a forecast could see at its origin — the rate to the origin, the district’s size — every quarter of districts is forecast high by about the same amount: the bias is a year effect. A district’s long-run rate persists into its next three to nine years at 0.27 to 0.30, the three tenths the damping already carries, so decaying the rate toward a share of the long-run rate over-corrects at every share above five hundredths and five hundredths is worth a third of a percent; relative to the state the error-minimizing share is zero. A drift fitted on the pre-closure mean district removes half of what crosses the closure and puts the total a point low before it. An asymmetric band is that correction drawn as a band, and holds less — 62.5% against 65.6% at ten years — than shifting the center by the same amount.

Done, by #431, and recorded in “The bias published beside the point”. The feed publishes the bias beside the projection: four figures per horizon rather than the two the sentence this replaces asked for. The mean district’s and the total’s, each across the closure and restricted to targets before it — the restriction changes which origins are scored, so it is a different question rather than a cleaner sample of one. The computation is hoisted into project::backtest::bias_profile, on the same scored forecasts the band above is measured against, and /method draws all four against zero beside the point they describe. Nothing in this record moves: the exponent is 0.65, sigma is unchanged, and the center of every published interval is where it was.

Alternatives considered Contents

Leave it. The band was still a floor, which is the safe direction to be wrong in. Rejected because 55.8% coverage on a stated 68% interval is not a conservative interval, it is a mislabeled one, and the label is what a reader acts on.

Rescale sigma instead. Would fix one horizon and miss the rest, since the defect grows with the horizon. It also costs the prior its provenance.

Fit to the dispersion (0.60). The obvious move and the wrong target; see the decision.

0.64. Minimizes the worst horizon where 0.65 minimizes the average. The surface is flat from about 0.63 to 0.66 and the third digit is not identified, so the rounder value is taken — the same tie-break the-fitted-damping records.

Cited by Contents