What the review's fixes changed (2026-09-16)
Companion to 08_review_2026-09-16.md. Every correctness finding in that
review's Parts 1.1–1.4 has been fixed on the branch; the affected sources
were reloaded, the analyses re-run, the harness re-run, and the served
numbers re-published. The refactors in the review's Part 2 are not started.
This file records, per finding, what was done and what it changed, then the
re-validation results, the published conclusions that moved, and what is
still open.
Status. A second correctness review of this state (
10_review_2026-09-16b.md) found five further problems and retracts two claims made below: the invariants step would not have caught the first review's critical findings (§1.3), and the 2021-22 staff "substance abuse" item was not a reversed scale but one symptom of that workbook's values being shifted one item against their labels (§1.3). No fixes from the second review are applied yet. Those findings have since been applied;10_review_2026-09-16b.md§7 records what changed and what it measured.
Reading order for a sceptic: §3 (what moved), §2 (the checks that now hold), §4 (what is open), then §1 for the detail.
1. Fixes, by finding
1.1 Served numbers (C1–C13)
| # | Fix | Reload | Served effect (before → after) |
|---|---|---|---|
| C1 SESIR total doubled | sesir.ts skips the file's SCHOOL TOTAL row and uses it as a cross-check that fails the load on any disagreement. | SESIR 2024-25 | Mean sesir_total_per_100 3.24 → 1.62 (category sum 1.56; the four categories do not cover every type). All residual-z cells on this outcome refitted. |
| C2 survey extras from the Q99 row | climate_survey.py drops the Q00/Q99 lines before anything is aggregated; the respondent count is the item rows' median, else the Q00 line's count. | All 20 workbooks | survey_staff_response_rate 2022-23: 342 cells at exactly 100 → 322 cells, mean 71.9. survey_parents_response_rate 2020-21: mean 81.8 → 7.4. |
| C3 survey items above 100 | Percent agreeing = agree ÷ (agree + dk + disagree) where the three are published and sum further than 1.5 pp from 100 (the largest deviation three whole-percent roundings can produce); students 2024-25 shares come from the response counts on the same rows; any value still outside [0, 100] is invalid and skipped, counted in the load note. | All | 28 composite cells above 100 → 0. 2024-25 rows renormalised: students 0 (counts used), staff 666; invalid item rows dropped: 68 (staff 2018-19, no don't-know column), 57 (parents 2021-22), 8, 6, 2. |
| C4 parents 2022-23 counts as percents; recommend missing | Spec marks the file's extras as counts over Total; recommend = (DefinitelyYes_A + MaybeYes) ÷ Total. | Parents 2022-23 | survey_parents_grade_ab 2022-23 mean 48.7 → 86.9; survey_parents_recommend 2022-23 now served (368 schools). The pre-registered second cohort exists — see §3. |
| C5 2018-19 respondent count on Q00 only | Covered by the C2 rule (Q00 fallback). | Parents/staff 2018-19 | 8 parent schools served 0.0 → suppressed; 60 staff schools with n < 10 → suppressed; every 2018-19 composite now carries a denominator. |
| C6 whole domains suppressed by the core rule | Domain and reverse-coding are canonical-level in the crosswalk (they were per row, so a canonical could sit in "resources" one year and "teaching" the next, and the union count could never be met). | All | survey_staff_resources 2022-23 → 2024-25: 0 served → 411–414 of ~467; survey_students_resources 2021-22 → 2024-25: 0 → 383–430. Core items per domain now identical in every year. |
| C7 clamp hid garbage | climate_survey.ts never clamps; values outside [0, 100] are skipped and counted. | All | No served survey value above 100. |
| C8 2016-17/2017-18 total sheet missed | membership.ts matches `total (enrollment | membership)andstudents | membership`; fails loudly without a total sheet; requires all 14 grade columns. |
| C9 residual-z survivors only | The engine fits every school with the outcome that year; the packet reports n_closed_or_inactive_included per fit. | Engine, residual load | 18,297 → 19,457 fitted cells (32 closed schools in the 2018-19 ELA fit, 18 in 2024-25). |
| C10 2022-23 race small groups inflated | Two-pass loader: in a year where any school has more hidden cells than hidden students (the file's * marks zeros), U is split over hidden groups in proportion to the school's adjacent-year shares, capped at 9 per hidden cell, overflow re-spread by prior then equally. Rule chosen by measurement against CCD (probe/race_imputation_prior.py): 2022-23 mean error Black 2.39 → 0.89 pp, Hispanic 2.15 → 0.66, White 1.73 → 0.84, American Indian 1.64 → 0.08; the uncapped prior (1.21 / 0.94 / 1.04) left 2–4% of school-years outside the process bound, the capped one none. Applying the prior in normal years was measured worse (Black 1.10 vs 0.91) and is not done. | Race, all years; engine | pct_amer_indian 2022-23 median 0.958 → 0.000 (mean 1.67 → 0.10); 10 uniform-14.29% school-years → 0. Schools whose October total is suppressed now get suppressed shares instead of a stale CCD-derived value (6 school-years). |
| C11 directory re-run reverted FISH capacity | directory.ts seeds fish_capacity_permanent from GIS only on a new record. | Directory re-run | 0 of 756 records changed on re-run (was 126). Registry text corrected: GIS equals FISH for 200 of 326, not "7 of 9". |
| C12 ELL absence | Universe schools with October membership and no row are served pct_ell = 0 (no all-zero row exists statewide in any year, so absence is the file's zero); the load note lists them. | ELL, all years | 11–19 schools per year gain a value (School for Advanced Studies campuses, Teenage Parent Program, …). |
| C13 CCD teachers = 0 | A reported 0 at an operating school is served as not reported (suppressed), never 0; the note warns that charter-network reporting makes 25 of 474 ratios exceed 40. | CCD staff | 60 "Teachers (FTE) 0" cells → suppressed. |
Also from §1.4 of the review: essa_ppe.ts district filter strips .0; readLatestSourcePath sorts; school_events.ts replaces the table in one transaction; residual_z.ts resets every cell carrying residual columns; school_grades.ts loads college_career_acceleration (83–98 schools per year); derived.ts populates miami_schools.latest_enrollment (566 schools); CCD 2022-23 membership loaded (534 schools) so the enrollment cross-check now covers three years; compute_percentiles.ts kept its bare query (the reconnect loop is refactor #2 in the review).
New load-time guarantee. finishDataLoad deletes cells still owned by an earlier load of the same source, so a school or value that drops out of a re-run cannot survive as a stale served cell. The first survey reload after the fixes left ~11,000 such cells; the guarantee removed them and the invariants step now checks it (I3).
1.2 Method and harness (M1–M12)
| # | Fix |
|---|---|
| M1 R1 not independent on N | residual_checks.py re-derives N from the served grade columns under the n_rule with its own parser, re-derives eligibility, requires the eligible set to equal the engine's, then refits. Result: 46 fits, eligible sets equal, max |
| M2 survey reader shared the parser | source_file.climate_survey has its own column map, scale rule, renormalisation, respondent-count rule, suppression and bounds assertion; it imports nothing from the parser and consumes only the committed crosswalk. Base cases: 958 + 906 + 928 cells, 0 mismatches. |
| M3 variance double-count | z = r ÷ √(τ² + se²), τ² = max(0, s² − Σw·se²/Σw), s² with the n/(n−k) correction; packet carries tau_between, mean_sampling_se, k_parameters. |
| M4 science weight | grades:5,8,9 (measured: Biology testers ≈ 0.75–0.85 × grade-9 membership, 0.39–0.41 × grades 9–10, 0.20–0.21 × 9–12 in the SRS files); the old rule and 9–10 are in the ablation (n_rule= rows: 12–18 and 6–8 flag changes). |
| M5 crosswalk splits | Lead-ins dropped, stop-words and light stemming, plus a hand-reviewed alias table (survey_item_aliases.csv, 13 rows) for what wording cannot merge; the parser prints every disjoint-year pair it did not merge. Core items: parents 34, staff 31 (safety 3 → 6), students 25. |
| M6 students' response-rate denominator | Computed in the loader against the same year's October membership in grades 4–12; the workbooks' own eligible column is kept in the parsed table but not served. |
| M7 race bound looser than the loader | The harness counts hidden cells as the loader does (tokens in fully visible grades are zeros); adds a uniform-split guard and per-year reporting. |
| M8 suppressed PK | Schools whose PK cell is suppressed are in neither group (34 in 2022-23); no-PK group 298 → 264, exact 83%, within ±1 95.8%. |
| M9 regime windows | comparability.yaml separates boundaries (regime breaks) from events (excluded / flagged years); persistence counts calendar years, so a missing year is a gap, not bridged. |
| M10 served-only never failed | reconcile.py fails a component above its served_only_allowance (default 0; the four FISH components carry 3, the documented name-override cells). |
| M11 funnel after the join | Counted before the end-year join; the closed/merged drop is explicit (16 and 15 schools). A missing pre-registered predictor now stops the analysis. |
| M12 magnet cells too small | derive_peer_groups.ts relaxes the magnet filter to level |
1.3 New: served-value invariants (served_invariants.py, step E)
Hard rules that fail the closure verdict: ranges by unit (signed changes and
utilization may exceed the [0, 100] band), identities between cells (SESIR
total ≥ each category; race shares sum to 100; October total = sum of grades
where all are served), and ownership (every row belongs to its source's
latest load). Shift alarms: a year whose median moves further than both
3 × MAD of the other years' medians and half the metric's pooled within-year
IQR, or whose served-school count moves more than 3 × MAD (floor 2%), is
listed; an alarm counts as explained only when
served_invariants_allowlist.yaml records its cause (assessment regime
boundaries, ESSER expiry, the parent workbooks' own respondent counts, the
2025-26 capacity fallback, the CCD-verified 2018-19 race shares). Each of the
review's critical findings would have tripped one of these.
The invariants step found one more publisher anomaly the review had not:
the 2021-22 staff workbook's substance-abuse item shows 58% agreement
district-wide (Air Base K-8: 81%) against 0–4% in every other year and 0% for
the same school's violence and gang items that year. It is recorded in
survey_item_exclusions.csv and left out of the 2021-22 staff safety
composite (5 of 6 core items remain).
1.4 Published text
Stories, recipes, notes, registry descriptions and page text were rewritten
from the re-run packets: the cost-premium story now attributes its dollars to
the school-level series and cites Table 5; the closure recipe filters on the
four actions rather than a year and restricts to regular schools; the sector
recipe separates district-run regular from other; the ablation sentence names
school level and English-learner share (19–26 ELA flag changes) ahead of
economic disadvantage (16–17); the region sentence says central and south
lost more than north (median −18.8 / −17.3 / −14.7); the pre-K sentence says
two in three pre-K students are chronically absent; the CCD agreement is 99%
of 1,533 school-years over three years; k ÷ g5 before the pandemic is
0.85–0.88; the participation story uses like-for-like universes (76 of 130
charters, 310 of 314 district-run regular schools with grade-4+ students);
the pct_* note names the FLDOE race file and the allocation rule; the blank-
component note says blanks are not stored; the annual-enrollment note says
+4.9%; the "not here yet" and "ratings follow" sentences are gone; the
citywide tooltip names Miami-Dade; derived sesir/survey pages name their real
inputs; the outcomes verdict, metric index and invariants report are
versioned; the residual dropdown lists only fitted metrics and years and a
metric change clears the year; nullable sorts put NULLs last;
sources.yaml names survey_students_*.
2. Re-validation (all reports under docs/qa_reports/miami/)
| Check | Result |
|---|---|
| A base case, 30 components | PASS, 0 mismatches (enrollment_oct 5,099 common; the new enrollment_oct_g3 component confirms the grade-3 medians the invariants flagged are the file's) |
| B completeness, 10 years | every source school served, every served school in source |
| C FLDOE vs CCD October count | PASS, 98.9% exact over 1,533 school-years (2018-19, 2022-23, 2023-24) |
| D computed values | 0 mismatches |
| E invariants | hard rules 0 violations; 25 shift alarms, 13 with a recorded cause, 12 open (see §4); the outcomes summary therefore reads REVIEW while the closure verdict, which the hard rules gate, reads PASS |
| Absenteeism vs EDFacts | PASS, 95.8% of 264 no-PK schools within ±1; PK slope 0.66, r 0.86 |
| Race shares vs CCD | PASS, 100% within the (tightened) process bound for all three shares; mean error 0.88 / 0.77 / 0.93 pp; 2022-23 alone 0.89 / 0.66 / 0.84 |
| Residual checks | PASS: independent N and eligibility equal, max |
| Closure verdict | PASS |
| App | tsc clean, eslint clean on the touched files (the two pre-existing set-state-in-effect errors in the search dialog fixed), 51/51 unit tests |
3. Published conclusions that moved
- Parent sentiment → enrollment does not replicate. With the 2022-23 parent recommend measure loaded, the pre-registered second cohort gives +2.0% per SD (n = 265, p = 0.11) against +6.9% (p < 10⁻⁶) in 2021-22. The student recommend share holds in both (+5.5% and +5.3% per SD, p < 0.001), as do student safety (+5.0 / +4.2) and staff leadership (+3.4 / +3.2). The story, the climate page and the recipe now say so; the pre-registration's reason for two cohorts was exactly this.
- Residual-z: every z changed (universe, variance, science weights, race covariates, SESIR scale); the P2 negative result stands, ρ(z, gains) 0.45–0.60 against ρ(raw, gains) 0.76–0.85 in all eight ELA/math fits; ELA R² 0.68–0.76; flag counts moved by a few per fit.
- Enrollment packet: only the 2017-18 point (C8) and the 2022-23 five-year-change row count; the lift table, Fisher tests and cost figures are unchanged.
- Survey composites: every served value re-derived; resources domains exist; 2018-19 has no served zeros; nothing above 100.
- Race shares 2022-23: small groups back to their level; Black/Hispanic errors halved against CCD.
- SESIR total: halved; its 12 residual flags refitted.
4. Open items
- Twelve shift alarms without a recorded cause (
__served_invariants.md): eleven are served-school counts that move 8–23 between years on SRS and School Grades components (biology, geometry, grade 9–10 ELA, college and career acceleration, social studies, middle-school acceleration) and on the students' 2021-22 extras (430 schools against 393 typical); one is a median, the lowest-quartile ELA gains rising to 63 in 2024-25 from 55. Each needs a reader, not a loader: nothing in the re-reads disagrees with the files. - Survey response-rate history: on the same grades-4–12 denominator the students' median rate is 36–37% in 2018-19 and 2019-20, 50% in 2020-21 and 62–67% from 2021-22 (means 44–46%, 49%, 57–63%; corrected 2026-09-16b); the respondent counts are the workbooks' own. Not an artefact this pipeline can identify; reported as is.
- Charter staffing ratios above 40 are served as reported with a caveat; no external staff count exists to correct them.
- Refactors (review Part 2, items 1–10): none started. The two policy decisions to record first are unchanged: the reconcile suppression rule and the peer-group distance rule.