The widening rule
A decision record · the-widening-rule · cited by 2 pages
The projection band widens as horizon^0.65 rather than as a square root — the second parameter in this repository replaced by a fitted value, and the first fitted against what a band is for rather than against what it is made of.
Context Contents
project::series::Prior::spread draws every interval this repository publishes:
z * sigma * horizon^k. sigma is the cross-sectional spread of district annual enrolled-ADM
growth, recomputed from the panel with its provenance printed beside each interval. k was
0.5, and said why: “Growth errors compound, so the band widens with the square root of the
horizon rather than staying fixed — the standard random-walk result.”
That is a claim about school districts, and it had never been checked. #289 checked it. The band is a floor, as the module doc says — realized out-of-sample error is wider than the interval at every horizon from one year to five, under the shipped method and the one it replaced. But it is a floor that sinks:
horizon coverage under horizon^0.50
1 year 67.3%
2 66.1%
3 63.8%
4 60.6%
5 55.8%
against the 68.3% a one-sigma interval claims. At one year the prior is very nearly calibrated — a spread taken across districts turns out to be about the right size for one district’s one-year error — and everything the rule was doing wrong, it was doing to the years after the first.
The mechanism is that forecast errors are positively autocorrelated: a district whose trend is misjudged stays misjudged, so the random-walk assumption that each year’s error is drawn fresh is what a school district does not do.
The decision Contents
Set the exponent to 0.65, as project::series::HORIZON_EXPONENT, mirrored by
HORIZON_EXPONENT in web/src/lib/project.ts.
horizon horizon^0.50 horizon^0.65
1 67.3% 67.3%
2 66.1% 70.4%
3 63.8% 71.2%
4 60.6% 68.4%
5 55.8% 66.5%
worst gap 12.5 pts 2.9 pts
Fitted against coverage, not against dispersion. Matching the standard deviation of the error gives 0.60 and still leaves the band mis-covering, because the error distribution is not normal — the two targets would agree if it were. Coverage is what a band promises, so coverage is what was fitted.
sigma does not move. It is a measured quantity with a stated source, and replacing it with a
backtest constant would trade provenance for fit. Only the shape in the horizon was wrong, and a
band that is the wrong shape cannot be corrected by rescaling: it is wrong by a different amount
in every year.
Consequences Contents
The published FY2032 bands widen. Central estimates do not move at all — the exponent touches only the interval:
current law $7,233M $7,017M-$7,494M -> $6,958M-$7,582M ±3.3% -> ±4.3%
guarantee removed $6,301M $5,938M-$6,686M -> $5,831M-$6,809M ±5.9% -> ±7.8%
The finding those figures carry survives: the guarantee-removed band is 1.80 times as wide as the current-law one, against 1.79 before, so “removing the guarantee nearly doubles the state’s exposure to enrollment forecast error” is unchanged.
Four of the figures #291 bound moved, and the manifest said so before anything else did —
the_manifest_reproduces_its_pins failed naming the key, both values and the distance, on the
first run after the exponent changed. That is the mechanism working on the next change after it
was built, rather than in principle.
Math.pow and Rust’s f64::powf agree to twenty places on every integer horizon this feed
publishes, so the exponent could be an exponent rather than a lookup table and the feed
reproduction check is unaffected.
Amendment Contents
Settled, by #391. The fit used horizons one to five, and that was a choice rather than the
panel’s limit: the same six origins reach eleven years, and the two the panel supports but
the fit did not use — FY2011 and FY2012 — take it to thirteen. Run out there, coverage of
the cross-district error stays within 3.0 points of 68.3% at every horizon from one to
thirteen, and refitting to coverage over the longer clean range returns 0.65 again. The
FY2032 leg and the feed’s ten-year horizon are inside the measured range now. Measured in
crates/project/tests/the_horizons_the_backtest_stopped_short_of.rs.
The alternative this record rejected as “the obvious move and the wrong target” turns out to have been partly a window effect as well. The dispersion exponent refits to 0.573, 0.600 and 0.623 over one to five, seven and nine pre-closure years and 0.650 over the whole reach — it climbs toward the coverage exponent as the range extends. The two targets still differ, and for the reason given, but 0.60 was never a measurement of anything past five years.
The sentence below was the binding one, and it is settled too, by #423.
Whether the band should be asymmetric was not addressed. It is multiplicative, so it already is
in dollars, but the log error has a growing positive mean — see
the_bias_no_single_damping_can_remove — and an interval centered on a biased point estimate
inherits that bias whatever its width. That was the whole of the long-horizon defect.
Pooled across origins the band’s coverage sinks from 67.1% at five years to 60.1% at thirteen,
and removing each origin’s own mean restores all of it: the width is right and the level is not.
The origin effect is the shutdown — at five years the origins landing on FY2020 or earlier are
very nearly unbiased while the two landing on FY2023 and FY2024 run +0.035 and +0.038 — and
sigma, being a cross-sectional spread, cannot price a statewide level break at any exponent.
Mean log error reaches +0.056 at the feed’s ten-year horizon, so the point sits about 5.8% high.
Neither the center nor the band moves, and the bias is published instead.
crates/project/tests/the_bias_that_belongs_to_the_years.rs measured the four remedies #423
named. The 5.8% is the mean district’s; the statewide total’s is 3.2% across the closure and
within 0.7% at every horizon to seven years before it, with a sign that changes from origin to
origin, so one correction cannot serve the two figures the feed publishes from the same
projections. Sorted on anything a forecast could see at its origin — the rate to the origin,
the district’s size — every quarter of districts is forecast high by about the same amount: the
bias is a year effect. A district’s long-run rate persists into its next three to nine years at
0.27 to 0.30, the three tenths the damping already carries, so decaying the rate toward a share
of the long-run rate over-corrects at every share above five hundredths and five hundredths is
worth a third of a percent; relative to the state the error-minimizing share is zero. A drift
fitted on the pre-closure mean district removes half of what crosses the closure and puts the
total a point low before it. An asymmetric band is that correction drawn as a band, and holds
less — 62.5% against 65.6% at ten years — than shifting the center by the same amount.
Done, by #431, and recorded in
“The bias published beside the point”. The feed
publishes the bias beside the projection: four figures per horizon rather than the two the
sentence this replaces asked for. The mean district’s and the total’s, each across the closure
and restricted to targets before it — the restriction changes which origins are scored, so it is
a different question rather than a cleaner sample of one. The computation is hoisted into
project::backtest::bias_profile, on the same scored forecasts the band above is measured
against, and /method draws all four against zero beside the point they describe. Nothing in
this record moves: the exponent is 0.65, sigma is unchanged, and the center of every published
interval is where it was.
Alternatives considered Contents
Leave it. The band was still a floor, which is the safe direction to be wrong in. Rejected because 55.8% coverage on a stated 68% interval is not a conservative interval, it is a mislabeled one, and the label is what a reader acts on.
Rescale sigma instead. Would fix one horizon and miss the rest, since the defect grows
with the horizon. It also costs the prior its provenance.
Fit to the dispersion (0.60). The obvious move and the wrong target; see the decision.
0.64. Minimizes the worst horizon where 0.65 minimizes the average. The surface is flat from
about 0.63 to 0.66 and the third digit is not identified, so the rounder value is taken — the
same tie-break the-fitted-damping records.