Find What Works: Schools
← All source documentsRaw file
Source document

docs/cities/miami/11_accuracy_fixes_2026-09-16.md

Served verbatim from the project repository. Internal working document conventions apply: documents may reference file paths, branch names, and findings-ledger anchors from the repo.

Accuracy corrections after 87dbbde

This record supersedes conflicting interpretations in review records 08–10. The changes were implemented on claude/admiring-darwin-9pzcjp following an independent adversarial review.

Corrections

  • Event identity: the August reassessment belongs to Carol City Middle, school 6051, not Carol City Elementary, 0681. The cited report names the middle school. The event table and enrollment packet were regenerated. An optional source-name check now rejects the former 0681/Carol City Middle mismatch before replacement; other rows without a source name are not claimed to have that check.
  • Censored ELL counts: pct_ell is exact only when the source identifies the total. Otherwise it is suppressed, and pct_ell_lower_bound / pct_ell_upper_bound preserve the interval. Each hidden race cell contributes zero to nine students, with the upper total capped at enrollment. The model and peers explicitly use the lower endpoint. The residual packet compares upper-endpoint fits on the same schools; that comparison is not a full interval over all allocations.
  • Survey validity and scoring: zero-total response profiles are unavailable, not ordinary zero scores. Composites average agreement on positive items and disagreement on negative items; unknown responses remain nonfavorable in both cases. The overloaded/overwhelmed staff item is explicitly aliased and assigned consistently to Teaching and Learning. All 20 original survey workbooks were downloaded with SHA256 matching the committed provenance and all 40 parsed tables regenerated. The independent reader checks 18,588 school/domain/year comparisons across 40 domain/file checks, with zero mismatches (docs/qa_reports/miami/survey_favorable_response_source_check.json).
  • Residual interpretation: removed fails_by_construction and its invalid proof. P2 is an empirical comparison of Spearman correlations. A positive demographic gradient and a larger fitted spread do not force raw achievement to win. A numerical counterexample is tested. The lowest quartile is relative to school/grade and can include proficient students; its gains use the ordinary rules. Maintaining level 5 can qualify; FAST-to-FAST levels 3–4 generally require an increased scale score. See the 2025 state calculation guide.
  • Movers: both UI and query reject reversed or incompatible windows, including FSA-to-FAST, pandemic exclusions and incompatible aggregate-grade rules. Multi-year windows are labelled as change over the selected period.
  • Atomic source loads: candidate facts and directory metadata are staged, then published together with provenance and pruning in one database transaction. A failed replacement preserves previous values. Real PostgreSQL regression tests cover rollback, metadata, empty candidates and competing load generations. Peer/percentile and residual derivation are subsequent steps, not one transaction over the entire pipeline.
  • Interpretations: sector totals describe net stocks, not individual transfers; the spending benchmark is not identified fixed cost or closure savings; an annual-count-gap regression on October PK membership does not identify PK prevalence; year-to-year survey correlation does not isolate measurement error or explain coefficient attenuation. Survey windows overlap and differ in duration. Historical grading notes now distinguish the optional 2020–21 grades and two-year gains from missing 2019–20 grades.
  • Peer sensitivity: removed the tautological comparison of the first ten neighbors in nested lists. k=20/30 memberships remain stored, but rating sensitivity to k is explicitly not evaluated.

Rebuild and checks

The core Miami source loaders, climate survey, peer groups, percentiles, residuals, survey regressions, metric catalogue and enrollment packet were rebuilt in an isolated local PostgreSQL database. Every harness step ran against this rebuild; the final summary aggregates those regenerated component reports. Production database publication remains a separate runbook operation; this branch update is not a production deployment.

  • 82 JavaScript tests and 12 Python tests passed, including five real-PostgreSQL transaction tests. TypeScript compilation and the final production build passed (786 generated pages).
  • Enrollment QA: PASS. All 35 configured served/source base-case components passed, with no numerical mismatches. Enrollment completeness, federal membership and race comparisons, federal absenteeism comparison and computed-value checks passed.
  • Residual checks: PASS. All 46 fits and 19,457 eligible cells were independently checked; maximum absolute z difference was 4.06 × 10⁻¹¹, with finite outputs and no served/CSV discrepancies. P2 still fails in all eight tested achievement fits; the L25 comparison favors adjusted scores in two of eight. These remain empirical results.
  • ELL sensitivity: all 46 upper-endpoint fits completed on the same schools. Across fits, 66 flags change (maximum six in a fit); the largest absolute z change is 1.1333. There are 576 exact ELL values and 2,499 appropriately withheld exact values across the 3,075 checked school-years; both bounds match the independent source reader throughout.
  • Survey: the isolated database contains 54,436 corrected survey facts. School 5007 has four unavailable staff composites; Miami Jackson Senior's 2024–25 student safety score is 37.9667. All five served survey checks pass (5,561 school-years). All 26 main regression fits agree with an independent statsmodels calculation (maximum coefficient difference 7.22 × 10⁻¹⁶; maximum standard-error difference 2.36 × 10⁻¹²). Seven of twelve tests survive Holm, down from eight: the 2022–23 student-safety association has p = .0134 and no longer survives that adjustment. Student-recommend associations remain positive in both windows.
  • Open review status: hard range, identity and source-ownership checks pass, but 26 year-to-year shift alarms remain unexplained (144 total alarms). Therefore the outcomes summary remains REVIEW, not blanket PASS. The corrections do not establish causal effects or eliminate every data limitation.

Source vintage and reproducibility

All 20 survey workbooks and the ingested static core files used in the rebuild matched the recorded SHA256 hashes, including the three NCES membership ZIPs and the board PDF. The live MSID directory had changed: the September 16 snapshot has SHA256 eafb8276aaac8a59adf1e9436ef6e8d6a209f395bb691c6fd1e4e850891e8304, 26 bytes smaller than the earlier file. Its actual new provenance is committed. State/Dade row counts, activity/charter counts and all 65 named schools available in the committed packets agree. The old raw bytes were unavailable, so exact row-level equivalence is not claimed; __msid_source_refresh.json records that limit.

NumPy/BLAS emitted floating-point warnings during matrix products on this local runtime. The independent fits explicitly require finite outputs and agree to the tolerances above; no nonfinite analytical result was accepted.

Corrected closure comparisons

The Carol City correction retains 260 eligible schools and 35 flagged schools. The under-half-capacity comparison remains 23/54 versus 12/206 (7.3×). Enrollment under 400 is now 27/69 versus 8/191 (9.3×, Fisher p ≈ 1.54 × 10⁻¹¹); the combined under-half-capacity/under-400 comparison is 18/30 versus 17/230 (8.1×). The $20,484,039 spending benchmark remains unchanged and is described as a benchmark, not savings.

Addendum (later on 2026-09-16): staff 2018-19 retired, ELL coverage explained

Three items raised on reading this record, applied on the same branch.

  • The 2018-19 staff workbook is retired, not re-scored. Favorable-response scoring exposed that the file is not comparable with the later staff workbooks: it publishes agreement and disagreement only, with no don't-know column, and the two sum to a median of 83% across item rows (10th percentile 50%), so a response category is missing from the file. On the negative-worded core items its published disagreement is 25% against 65–70% in every later year while agreement is stable (18% against 20%), so the disagreement column is not the measure the later files publish. Scored by disagreement, the 2018-19 staff safety, resources and teaching composites sat 13–20 pp below 2019-20 (six of the 26 unexplained shift alarms above); scored by 100 − agreement, the hidden category counted as favorable. Neither is comparable, so the staff series now starts in 2019-20: the file stays fetched with provenance and is marked role: retired in sources.yaml; RETIRED_SPECS in the parser records the measurement and removes its parsed tables; RETIRED_SOURCES in the survey loader deleted the 1,904 cells still served from it. The staff core item set is recomputed over six years (32 items, one more than before, teaching 6); the survey regression uses 2021-22 and 2022-23 only and is unchanged (7 of 12 tests survive Holm; parents on the common schools +7.1% then +3.5% per SD, bootstrap p = 0.003).
  • Exact pct_ell coverage is recorded, not changed. The exact share is served for 76–108 schools a year (those whose ELL file hides no race cell); its count series is explained in the allowlist with that cause. The model and the peer groups use the lower bound, whose coverage is complete.
  • Allowlist upkeep. The pct_econ_disadv 2024-25 entry sits at the alarm threshold (lift 3.1) and is kept with that note; the remote-year survey entry now includes 2019-20, because a first served year is compared with the next (the remote year); an entry with no year applies to every year of its metric.

Rebuilt in this session from the same source snapshots (the MSID directory re-fetched with the SHA256 above) with the reviewer's loaders: every base case, cross-publisher check and residual check passes; the regenerated residuals agree with the committed ones to 1e-11 (runtime floating-point noise) and the enrollment packet is identical, so those artifacts are kept as committed. Unexplained shift alarms: 16 of 146 (the small count and grade-cell moves listed in __served_invariants.md); the outcomes summary therefore stays REVIEW. Tests: 82 JavaScript (5 Postgres transaction tests skipped where no test database is configured), 12 Python; type check and production build pass. The production database still needs the runbook reload.