Residual-z: outcomes adjusted for who enrols

1. Method

• Model. For each outcome and year, weighted least squares of the school value on share of black students, share of hispanic students, share of white students, english learners: lower bound, share of economically disadvantaged students, log October enrollment, with dummies for sector, magnet, level. All covariates are from the same year as the outcome (race shares from the state race file, allocated as in the loader; the lower bound on English-learner share from the ELL file (hidden counts are bounded, not observed zeros); economic disadvantage from the School Grades file).
• Weights and eligibility. Each cell is weighted by its sampling N: the stored denominator where the file gives one (absenteeism enrollment, graduation cohort), otherwise October membership in the tested grades. Cells with N below 30 are not fitted; years without a complete covariate row are skipped.
• Standardisation. z = residual ÷ √(τ² + se²), where se is the cell’s own binomial (percent) or Poisson (rate) sampling error (zero for attendance, a share of student-days rather than of students, so its z is residual ÷ τ) and τ² is the between-school residual variance: the fit’s weighted mean squared residual (with an n ÷ (n − k) degrees-of-freedom correction) minus the weighted mean of the cells’ se², floored at zero. The raw residual spread already contains the sampling noise, so it is removed once rather than counted twice (a correction made after the 2026-09-16 review). A small school needs a larger raw gap to reach the same z. z is oriented so positive always means doing well.
• Universe. Every school with the outcome that year is fitted, including schools that have since closed; excluding them (as the first version did) shifted the expected line upward for exactly the schools the closure analysis cares about.
• Flags. |z| ≥ 2 is flagged. A flag is persistent when the same sign appears in 2 of the last 3 calendar years inside one comparability regime (the FAST switch in 2022-23 starts a new regime; 2019-20 and 2020-21 have no fitted outcome, so the window bridges neither).

Engine: pipeline_miami/analysis/residual_z.py · config: outcomes, covariates, comparability (config d87f0d4de921).

2. What the model explains

19,457 eligible school-years across 12 outcomes; 1,220 flagged. R² is enrollment-weighted; s is the model error in the outcome’s own units; the median sampling error shows how much of the denominator is the school’s own noise.

3. Pre-registered checks

Five checks were fixed in the engine before its first run. The one that matters is external: Florida publishes learning gains, a growth measure the state computes from each student’s own prior score, which is independent of the demographic regression. The expectation was that the adjusted residual, being the part of achievement not explained by enrolment, would track growth at least as well as raw achievement does.

• P1. rho(z, gains) > 0 and p < 0.001, every fitted year.
• P2. rho(z, gains) >= rho(raw, gains), every fitted year.
• P3. |rho(z, pct_econ_disadv)| < 0.10, every fitted year.
• P4. share of fitted outside [0,100] < 2%.
• P5. flagged share in [2%, 20%].

P2 fails in 8 of 8 fitted years. Raw achievement correlates with the state’s learning gains more strongly than residual-z does. This is an empirical comparison: a positive correlation between the fitted demographic component and gains does not make P2 fail by construction. These are Spearman rank correlations, and z uses each school’s sampling error; a Pearson identity for raw achievement = fitted value + residual does not establish their ordering. Readers should treat residual-z as a demographic adjustment, not as a growth measure. The reason gains carry a demographic gradient is unresolved by these correlations. Florida’s gain rules depend on prior performance and assessment combination: FAST ELA and FAST-to-FAST mathematics students maintaining levels 3 or 4 also need a scale-score increase, while maintaining level 5 can qualify by itself; some mathematics assessment transitions have different rules. See the state calculation guide, pp. 17–18 and 23–25.

Against the lowest-quartile gains the demographic gradient is weaker but not gone, and residual-z correlates more strongly than raw achievement in 2 of 8 fits. The lowest quartile is relative to each school’s prior-year assessment groups and uses the same gain rules; it can include students who qualify by maintaining a level. This is a sensitivity comparison with a different population, not a test that isolates or excludes the level-holding mechanism.

P3 asks whether the residual still tracks economic disadvantage in rank terms. A failed check means the absolute rank correlation remains at least 0.10; the linear adjustment has not removed that association. A passing check does not establish independence.

4. Which covariates carry the adjustment

Refitting ELA achievement for 2024-25 with one covariate removed at a time. The first row removes all of them (z is then the raw value standardised). Two numbers matter: how much the flags change, and whether the link to learning gains survives.

English-learner hidden-count sensitivity

Suppressed English-learner counts give a range of possible shares. The baseline uses each school’s lower bound. The table refits the same eligible schools with every upper bound, reporting the ELL coefficient and flag changes. This compares two endpoint choices; it does not bound all possible fits when schools’ hidden counts vary independently.

5. Disagreement with the peer-percentile method

Both methods answer “is this school unusual given its enrolment”, one by ranking within 40 nearest neighbours, one by a continuous adjustment. On eligible cells with both a peer percentile and a z, the table counts schools flagged by one method and not the other.

6. Bounded-rate guardrail

A linear fit to a 0–100 rate can predict impossible values. Each is counted rather than clipped; where the count is material the residual for those schools is unreliable at the boundary.

7. What this cannot show

• The covariates are the ones the state publishes at school level. There is no special-education share, no mobility rate and no prior-achievement control, so a residual partly reflects what those would have explained.
• The tested-grade membership used as N for the School Grades components is an approximation to the tested count, which the state does not publish with the component; using enrollment above the actual tested count understates the cell’s sampling error. That can make |z| too large at a fixed between-school variance; the net effect after refitting depends on how that variance changes.
• Economic disadvantage is not comparable across Community Eligibility status (decision M.8); it enters as a covariate anyway because omitting it is worse, and the ablation shows what it changes.
• Every fit is a cross-section. A school that changed grade span or merged appears as an outlier for reasons the events table, not the model, explains.

Per-metric definitions: English Language Arts Achievement (School Grades component), Mathematics Achievement (School Grades component), Science Achievement (School Grades component), English Language Arts Learning Gains (School Grades component), Mathematics Learning Gains (School Grades component), English Language Arts Learning Gains of the Lowest 25% (School Grades component), Mathematics Learning Gains of the Lowest 25% (School Grades component), 4-year graduation rate (federal uniform), Chronic absenteeism (absent 10%+ of days enrolled), Students absent 21 or more days, Attendance rate (% of days present), SESIR: all SESIR incidents per 100 students.

Evidence versions

Packet 4ccf92fcbe52 generated 2026-09-16 21:34:34 UTC · checks 4408b3f93567 (PASS) · packet JSON · per-cell output · check report

Corrections log

Accuracy review: removed the false claim that P2 must fail under a demographic gradient and corrected the lowest-quartile gain interpretation. The empirical correlations are retained. English-learner covariates now identify the lower bound explicitly, with an upper-endpoint sensitivity comparison.