Miami build: adversarial review (2026-09-16)
Scope: everything authored on claude/admiring-darwin-9pzcjp through commit
bb0be6b (loaders, Python pipeline, harness, analyses, app pages, stories,
docs), reviewed for (1) correctness and (2) duplication / opportunities for
shared code across cities and outcome variables.
Method: four independent read-only reviewers, each scoped to one layer (TypeScript loaders + schema + enrollment packet; Python pipeline + harness + residual-z + survey; app pages + stories + docs; cross-city duplication), followed by verification of every critical claim with direct SQL against the served database and re-reads of the cited lines. Nothing was modified. No fixes have been applied; a proposed fix order is at the end.
Status (applied later on 2026-09-16). Every correctness item in Parts 1.1–1.4 has been fixed on the branch and the affected data reloaded, re-validated and re-published; the refactors in Part 2 are not started. The record of what changed, what the re-runs found and what remains open is in
09_review_fixes_2026-09-16.md.
Bottom line: the closure-track numbers (enrollment, utilization, PPE, events, lift table, Fisher tests) reconcile against the source cells and hold. The outcomes side has eleven defects that change served numbers, seven of them in the climate survey chain and one (SESIR total) a clean 2× error. The residual-z engine fits on survivors only, and two of its "independent" checks are not independent on the inputs that matter. Several story sentences contradict the packets they cite. On duplication: the schema is not what blocks sharing; copy-paste is, and there is a low-risk order in which to fix that.
Part 1 — Correctness
Severity scale: critical = a served number is wrong; major = a statistic, check, or published claim is wrong or biased; minor = latent, cosmetic, or confined to a probe. Confidence is stated where a finding rests on inference rather than a reproduced number.
1.1 Critical: served numbers are wrong
| # | Where | What | Evidence (served DB unless noted) |
|---|---|---|---|
| C1 | scripts/loaders/miami/sesir.ts:38-41 | sesir_total_per_100 is exactly 2× the true count. The file has a SCHOOL TOTAL row per school and the loop adds every row's count, including that one. Category rates are correct. | Mean sesir_total_per_100 across 519 schools 3.24 vs sum of the four category rates 1.56. School 0041: type rows sum 5, SCHOOL TOTAL 5, served total ×enr/100 = 10. Downstream: 442 residual-z cells fitted on the doubled scale (z is not scale-invariant because se depends on n). |
| C2 | pipeline_miami/parse/climate_survey.py:182-190 | School-level extras (n_responses, n_eligible, response rate, recommend, grade A/B) are aggregated with max over all rows, including the Q00/Q99 district/summary rows that line 179 drops only from the item table. | Staff 2022-23 school 0041: Q99 "Overall" TTL 2,368 vs 64 respondents → parsed response rate 3,289%. Parents 2020-21 school 0001: n_responses 43,168. Served survey_staff_response_rate 2022-23: 342/342 cells = 100 (after the loader clamp in C7); survey_parents_response_rate 2020-21 mean 81.8, 293/445 = 100. The n<10 suppression for recommend/grade_ab in those files uses the same wrong n. |
| C3 | climate_survey.py:119-124,162,170 and scripts/loaders/miami/climate_survey.ts:57-69 | No range check on item percentages in the parser and no guard in the loader. The 2024-25 student and staff workbooks contain publisher rows with agree share > 1 (students 0441 Q05: Agree_PCT 1.232, NA_PCT −0.303). | Parsed 2024-25: 94 unsuppressed student item rows > 100 at 34 schools (max 166.7), 165 staff rows at 18 schools (max 132.1). Served: 28 composite cells > 100 (survey_students_teaching 5091 = 119.30; survey_staff_leadership 3661 = 117.13) and every other composite for ~47 schools inflated below the cap. |
| C4 | climate_survey.py:55 (parents_2022-23 spec) | grade_ab=["A","B"] reads response counts (A+B+C+D+F = Total on 100% of rows), so grade_ab_pct is a count. The same file has recommend counts but the spec omits recommend, so survey_parents_recommend 2022-23 is never produced. | Served survey_parents_grade_ab 2022-23 mean 48.7 vs 86–88 every other year (0041 parsed 164 → clamped 100). survey_parents_recommend exists for 2021-22, 2023-24, 2024-25 only. Consequence for the survey→enrollment test: the pre-registered second cohort (2022-23) is "skipped: fewer than 60 schools" (n = 0), so the headline predictor rests on one cohort despite the docstring's "two survey years so the result is not one cohort's accident". |
| C5 | climate_survey.py:160-161,174-179 | In the 2018-19 parents/staff workbooks the Ncount sits only on the Q00 row, which is dropped; item rows get n = null and suppressed = false everywhere. The district zeroes withheld schools. | 8 parent schools (e.g. 0070, 7027) served survey_parents_{safety,teaching,leadership} 2018-19 = 0.0, suppressed = false; 60 staff schools with n < 10 served (0400, n = 9: 100 / 96.3); all 446 + 476 2018-19 composites have NULL denominator. The harness component for survey_parents_teaching 2018-19 PASSES because the reader reads n the same way (see M2). |
| C6 | climate_survey.ts:35,65-68 | nCore is the union of "core" canonicals across all years, but the crosswalk's core flag is not "asked every year". The vals.length*2 < nCore rule then suppresses whole domains, and where a composite does pass its item set changes by year, contradicting the registry text "using only the items asked in every survey year" (metric-definitions.ts:229, metric-notes.ts:131). | survey_staff_resources 467/467, 469/469, 459/459 suppressed in 2022-23..2024-25; survey_students_resources 482/482, 447/447, 488/488, 492/492 suppressed from 2021-22. 58 (respondent, year, domain) cells have a different item count than the union. |
| C7 | climate_survey.ts:81 | Math.min(100, v) clamps parser garbage to a valid-looking 100 instead of rejecting it. | survey_staff_response_rate 2022-23 342/342 = 100 (raw up to 7,400); survey_parents_response_rate 2020-21 66% = 100; survey_parents_grade_ab 2022-23 17% = 100 (raw up to 23,516); students response rate 2021-22 27 raw rows > 100. |
| C8 | scripts/loaders/miami/membership.ts:47,55 | Total-sheet lookup /total enrollment/i misses "Schl Total Membership" (2016-17) and "School Total Membership" (2017-18); the value-column regex /students/i would miss 2016-17's header too. With no total sheet the tokens-are-zero rule cannot run. | enrollment_oct 2017-18 stored suppressed for 43 schools (2–7 in other years); the ignored sheet publishes totals for 34 of them (0092 Sunny Isles K-8 = 2,174; 2191 Spanish Lake ES = 1,577; 0311 = 472 …). Packet: district_trend 2017-18 = 347,420 over 454 schools vs 357,288/486 (2016-17) and 350,442/500 (2018-19); "other" sector 6 schools/1,385 vs 30/3,088 and 35/3,406; enrollment_change_5yr_pct 2022-23 has 436 rows vs 465/474 adjacent. |
| C9 | pipeline_miami/analysis/residual_z.py:226 | activity_code == "A" & closed_at.isna() drops closed/inactive schools from every year's fit, including years when they were open. The baseline is survivors-only and no doc mentions it. | 36 closed schools carry 120 ELA cells with mean 47–51 vs 58–64 for kept schools in each year; they get no z and are absent from the reference distribution. This biases the fitted line upward for the schools most relevant to the closure analysis. |
| C10 | scripts/loaders/miami/race_membership.ts:397-413 | 2022-23 imputation: that year's file marks zeros with * (17,464 tokens vs ~5,900 in other years); the grade-total rule resolves only 1,482 as zeros, so 15,863 "hidden" cells share the unallocated count equally, inflating the small groups. | pct_amer_indian median 0.000 (2021-22) → 0.958 (2022-23) → 0.000 (2023-24), mean 0.111 → 1.673 → 0.099 (~15×); pct_hawaiian_pac_isl identical; pct_two_or_more mean 1.03 → 1.68 → 0.89; pct_white median 2.55 → 1.47 → 2.39. 10 school-years (all 2022-23, e.g. 7840, 7855) are served with exactly 14.29% for all seven races. The race cross-check passes them because its bound is looser than the loader (M7). |
| C11 | scripts/loaders/miami/directory.ts:223 vs fish_los.ts:64 | miami_schools.fish_capacity_permanent is order-dependent: directory.ts sets it from GIS CAPACITY in the update payload; fish_los.ts later overwrites with FISH. A documented-idempotent re-run of directory.ts would revert it. | 126 of 326 schools (39%) have GIS ≠ FISH (4491: FISH 1,723 vs GIS 689; 5101: 1,242 vs 403; 0231: 1,485 vs 2,099). Registry text "equals permanent FISH capacity for 7 of 9 checked schools" (metric-definitions.ts:131) vs 200/326 (61%) agreement in the loaded data. Current DB is correct only because fish_los ran last. |
Two more that change served numbers but are smaller in reach:
| # | Where | What | Evidence |
|---|---|---|---|
| C12 | scripts/loaders/miami/ell.ts:189-194 | Schools absent from the ELL file get no pct_ell rather than 0. Absence = zero LY students is the evident convention but is not verified against a codebook. | 11–19 enrolled schools per year (2024-25: the six School for Advanced Studies campuses, 3029, 6053, 8016). Removes them from peer-group features: 8 of the 45 ungrouped default-included schools. |
| C13 | scripts/loaders/miami/ccd_staff.ts:228 | TEACHERS = 0 kept as a value where CCD has no report; FTE = 1 yields absurd ratios. | 61 schools served "Teachers (FTE) 0" (0100 Mater Academy, 7042 …); students_per_teacher 999 (7016), 487 (6032), 322 (4010). |
1.2 Major: statistical method and harness
| # | Where | What | Evidence |
|---|---|---|---|
| M1 | pipeline_miami/verify/residual_checks.py:81 | R1 "independent refit" copies n (the WLS weight and the sampling N) and the eligible set straight from the engine's CSV (e["n"] = g.n.to_numpy()). Only the solver is independent; the docstring and METHODOLOGY.md:240 claim more. It cannot detect a wrong n_rule, grade sum, or eligibility rule. | Structural; C1 (doubled sesir scale) and M4 (science n) both pass R1 by construction. |
| M2 | pipeline_miami/verify/connectors/source_file.py:400-455 | The climate-survey "independent" reader imports the parser's SPECS, consumes the parser-built crosswalk, reuses the same median scaling and per-row n. It can only catch an error in the arithmetic mean. METHODOLOGY.md:241 describes it as "code that shares nothing with the normaliser except the committed crosswalk". | C3 and C5 are present in the raw files and both verified components PASS with 0 mismatches. |
| M3 | residual_z.py:182,200-202 | s is the weighted RMSE of residuals, which already contains the weighted-mean sampling variance, and it is added again in sqrt(s² + se_i²); no (n − k) correction. | Weighted-mean se² is 5.8% of s² at the median but 39–40% for learning_gains_*_l25 and attendance_rate (z understated up to ~18%, fewer flags); graduation (n = 83, k = 14) has s understated ≈10% (z overstated). Direction is conservative for most outcomes. Fix: τ² = max(0, s² − Σw·se²/Σw) with an (n − k) correction. |
| M4 | pipeline_miami/config/outcomes.yaml:32 with residual_z.py:143-150 | science_achievement: grades:5,8,9-12: Biology EOC is taken by roughly one high-school cohort, so summing 9–12 inflates high-school N (×1.89) and shrinks se_i. | Refit with 9-10 (same engine functions): 34 of 2,065 cells change flag (≈15% of flagged cells); high-school flags 19 → 17, elementary 77 → 60 (high-school weights move the whole fit). Add the n-rule choice to the ablation. |
| M5 | climate_survey.py:137-141,209-232,249-253 | Jaccard ≥ 0.6 on unstemmed words (stop list lacks feel/there/school/problem) splits the same item across wording changes, and core then excludes items that were asked every year. | Confirmed complementary splits: staff_21 "I feel there is a school violence problem." (2018-19→2023-24) vs staff_50 "School violence." (2024-25); staff_23/51 substance abuse; seven more staff pairs from the 2022-23 "I feel…" prefix drop; students_17/30 gangs; parents_06/38, 30/39, 36/44. Consequence: staff_safety and staff_teaching rest on 3 core items each, students_leadership on 3; the reverse-coded violence/substance items drop out of staff safety. Construct validity, not arithmetic. |
| M6 | climate_survey.py:47-51 | n_elig for students alternates between all-grade FTE (2020-21, 2022-23) and grades-4+ counts (2021-22, 2023-24, 2024-25), so survey_students_response_rate is one series with two denominators; nothing marks the break. | Served means by year: 24.1 / 57.6 / 29.8 / 52.2 / 56.5. |
| M7 | pipeline_miami/verify/cross_publisher_race.py:39,183-191 | hidden counts every * token, whereas the loader resolves tokens in fully-visible grades to zero, so the process bound is looser than the process it certifies and trivial where hidden cells are many. | 4,599 pairs: 13.5% have bound ≥ 10 pp (max 100.5). The 10 uniform-split school-years in C10 pass against a 100.5-pp bound. 36 school-years with minor-group shares summing ≥ 20 pp feed 20 residual-z eligible cells (11 flagged). The check has teeth for the 42.6% of pairs with no hidden cell (bound 0.5 pp, p90 |
| M8 | pipeline_miami/verify/cross_publisher_absenteeism.py:81 | (p["value"] or 0.0) puts schools whose PK cell is suppressed (1–9 PK students; 40 of 521 in 2022-23) into the "no PK" exact-comparison group. | The three largest "no-PK" deltas (5931 −8, 5241 −5, 9732 −3) are exactly such schools. Biases against passing and mislabels the population. |
| M9 | pipeline_miami/config/comparability.yaml:13-14 + residual_z.py:86-97,308-315 | The 2020-21 boundary is annotated "flag, do not drop" but flagged_years is read nowhere; every boundary starts a new persistence window (2018-19 and 2021-22, both FSA, can never share one; single-year windows make persistence impossible). Persistence counts consecutive fitted rows, not calendar years. | Today all 189 persistent cells span 2–3 calendar years, so no gap artefact yet; a school ineligible in a middle year would be bridged. |
| M10 | pipeline_miami/verify/reconcile.py:67 | status = PASS if matches and not mismatches: served-only and source-only cells never fail a component; only enrollment_oct has a completeness check. | Today absorbs only the 3 documented FISH served-only cells per year; a loader that fabricates or misattributes cells for any other metric would pass. |
| M11 | pipeline_miami/analysis/survey_enrollment.py:75,82 | The funnel's first row is counted after the inner join on 2025-26 enrollment and counts any metric row, not survey rows; 23 district-run schools with 2021-22 membership (5,545 students) drop invisibly. The survivors-only universe is pre-registered, but the funnel hides its size. | with_membership_both_years == district_run_active by construction. |
| M12 | scripts/loaders/miami/derive_peer_groups.ts:43,192,214 | Whole-school-magnet hard filter plus the <5-candidate floor leaves 22 fully-featured, active, default-included schools with no group and no peer percentile; separately, the subject filter on includeInDefaultComparisons gives the 80 non-regular schools no group at all (NYC and Philly filter candidates only). | Cells: K-8/district/magnet = 4 schools (0041 Air Base K-8, 3101, 3191, 5831); 6-12/district/magnet = 4; K-12/district = 2. School pages show "—". |
1.3 Published text that contradicts the data
Stories and notes are served to readers; each of these is a wrong claim on the page today.
| # | Where | Claim | Measured |
|---|---|---|---|
| S1 | data/cities/miami/stories/index.ts:79-98, recipes.ts:43-52 (small-school-cost-premium) | Dollar figures described as "school costs plus allocated district indirect costs"; recipe selects per_pupil_expenditure_total; evidence link goes to per-pupil-expenditure-total; cites "Table 7" and says the flagged-school table shows spending. | Every figure ($16,893 / $14,051 / $11,895 / $20.5M) is per_pupil_expenditure_school (packet ppe_vs_size, script line 198; DB median district ES <300 in 2024-25: _school 16,892.5, _total 17,545). The analysis page says "district indirect costs excluded". The by-size medians are Table 5; Table 7 has no spending column. |
| S2 | recipes.ts:27 (closure-lists-vs-utilization) | Step 2 flags school_year = '2026-27'; step 1 omits school_type = 'REGULAR'. | REASSESSED (12) and UNDER_REVIEW (15) are stored under 2027-28; 2026-27 holds BOUNDARY_CHANGE (16) and RECEIVED_CONSOLIDATION (4). The recipe as written returns 261 schools / 23 flagged vs the text's 260 / 35. ESE center 2531 (util 35%) is admitted. |
| S3 | recipes.ts:37 (charter-growth) | Sector SQL groups by is_charter only. | Returns 294,386 → 234,581, not the 291,298 → 231,607 quoted for district-run regular. |
| S4 | index.ts:155 (adjusted-scores-vs-learning-gains) | "economic disadvantage does most of it" (ablation). | ELA 2024-25: dropping pct_ell changes 20 flags, level 17, pct_econ_disadv 12, magnet 5; 2018-19: level 21, ELL 16, econ 13. ELL and level carry more in every year. |
| S5 | index.ts:195 (charter-participation) | "82 of 157 active charter schools … against 329 of 353 district-run". | 157 is active_schools.charter; 353 is cells (schools with any survey row). The climate page renders 329 of 385. Two denominators in one sentence. |
| S6 | index.ts:128, recipes.ts:67 (decline-since-pandemic) | Losses "concentrated … in the north of the county". Not in any packet. | District-run REGULAR, 2019-20 → 2025-26, by miami_schools.region: median change N −14.5% / C −18.5% / S −16.5% (mean N −12.2 / C −17.8 / S −14.1; share ≤ −25%: N 27% / C 29% / S 33%). North is the least affected. Recipe SQL returns 481 rows (all sectors, suppressed included) vs the text's 325. Confidence: plausible (depends on the GIS region field, but that is the site's own field). |
| S7 | index.ts:182 (federal-absenteeism-omits-prek) | "four-year-olds are chronically absent at two-thirds the rate of the school as a whole". | The slope 0.66 is chronically absent students per PK student, i.e. ~66% of PK students, which is what the same story's thesis (line 174) says. The two sentences contradict each other. |
| S8 | data/cities/miami/metric-notes.ts:85-88 | pct_* shares "from the NCES Common Core of Data … the 2023-24 vintage is the latest published". | Loaded from the FLDOE race file 2018-19 → 2025-26 with hidden-cell allocation (race_membership.ts); CCD is only the cross-check. Also :20 says blank School Grades components are "stored as suppressed" but school_grades.ts:346 writes no row for blanks. |
| S9 | Number slips | index.ts:65,132 k÷g5 "0.86–0.89" → packet 0.845–0.881; index.ts:62, recipes.ts:40 "95%" → exact_share 0.9833 on 1,018 school-years in two CCD years; index.ts:42,50 "31 unlisted … identifiable by name" → Table 7 lists only the 35 flagged, the histogram has no names; label "lift table, §4" → Table 4 in §3; recipes.ts:60, index.ts:105 2022-23 caveat blames missing letter grades (306 vs 305 schools) when the real reason is C4; "controls explain 11%" mixes controls_only.r2 (0.110, n 305) with the parent regression's r2_controls (0.105, n 256) and "adds another 11%" is partial R² (raw increment 9.5 points); index.ts:110 "utilization … a weak predictor" → coef 0.00194 (t ≈ 2.9), ≈ +4.6%/SD vs +6.9% for parents; metric-notes.ts:59 "6% larger" → measured +4.9%; index.ts:64 "high schools least (−9%)" → 6-12 (−7.4%) and K-12 (−6.3%) fell less; index.ts:178 "Both files are served here" → EDFacts is not served as a metric; enrollment/page.tsx:234 scholarship "roughly a quarter" is unsourced; residual-z/page.tsx:117 P3 "fails (small fits, low R²)" → also fails for science 2024-25 (n 426, R² 0.49). | |
| S10 | Stale / mislabelled UI text | app/miami/methodology/page.tsx:128-130 "What is not here yet: climate survey, residual-z, stories … in progress" (all shipped and linked above); app/miami/page.tsx:31 "Ratings … follow once test-score loads pass validation" (live); components/peer-pill.tsx:68 city tooltip says "~2,000 NYC public + charter schools" on every Miami page; [metric]/page.tsx:62,113 inputsOf renders "computed by sesir.ts from sesir_total_per_100" for derived sesir/survey metrics; climate-survey/page.tsx:40,71 "district-run" denominator 385 includes adult, technical, DJJ and OTHER (321 of 329 with-value schools are REGULAR); docs/cities/miami/sources.yaml:315 survey_student_* (singular) so the 2024-25 student workbook backs no served metric in metric_index.json; survey_items.csv canonical parents_37 with text "nan" (36 rows from a NaN item number). |
1.4 Latent and minor code robustness
scripts/loaders/miami/essa_ppe.ts:36district filter lacks.replace(/\.0$/, ""); the other eight loaders have it. A numeric13.0cell would silently drop every row.scripts/loaders/miami/compute_percentiles.ts:12-21dropped the reconnectingqueryloop that NYC/Philly keep for a stated reason (NYC header lines 13–15).scripts/loaders/miami/residual_z.ts:123-126,142-143reset covers only metric keys present in the CSV, so a metric dropped from the engine keeps stale residual columns;missingcounts z across all metrics.membership.ts:43header check accepts 13 of 14 grades; a missing grade column writes 0 (r[undefined]→ blank → 0). All ten files have 14 today.ccd_membership.ts:139,183-190+race_membership.ts:421: 6 school-years (42 cells) carry CCD-derivedpct_*with denominators 3–9 next to state-derived rows; race_membership skips those schools so they are never overwritten.nces_ccd_membership_2022-23is fetched (208 MB) but never loaded; noenrollment_oct_ccdfor 2022-23.school_events.ts:192-207deleteMany then per-row create outside a transaction._lib.ts:72-78readLatestSourcePathreturnsreaddirSync(...)[0]unsorted (every dated dir has exactly one file today).compute_percentiles.ts/derived.tsnever clear percentiles or derived rows for cells that later become null/suppressed (0 stale today).metric-definitions.ts:187registerscollege_career_accelerationbutschool_grades.tsCOMPONENTS never loads it (0 rows);miami_schools.latest_enrollmentpopulated for 0/756.lib/queries/miami-schools.ts:82DESCputs NULLs first for a hand-typedsort=zunder the peer method (1,067 fitted cells have no peer percentile) → page 1 all "—". Fixnulls: "last".app/miami/ratings/outliers/page.tsx:36-39,filters.tsx,movers/filters.tsx: undermethod=residualthe metric dropdown lists all 59 rated metrics but only 12 have fits, and the year list includes unfitted years; changing metric does not clearyear/from/to.lib/miami/evidence-version.ts:27-37does not register__outcomes_validation.jsonormetric_index.json, so those links carry no?v=and the 409 guard cannot catch a stale citation.components/analysis/miami-charts.tsx:141,154,186,miami-packet.ts:10hard-code "2024-25"/"October 2024"/MIAMI_YEARSand Table 7 relies onenrollment_2024_25field names; consistent today, stale on the next packet run.components/layout/school-search.tsx:51,60tworeact-hooks/set-state-in-effecteslint errors (pre-existing pattern; the Miami commit only added the bucket).docs/cities/miami/probe/race_imputation.py:174-186token_censuscounts blank cells as suppressed, contradicting its docstring (probe display only).
1.5 What was checked and holds
Loaders reconciled cell-by-cell against source files (3–6 cells each) and
against an independent re-read: absenteeism (all 9 files resolve the same
columns; fractions ×100; n/enr matches %), ESSA PPE (positional columns all
4 years; col4+col5 = col6, col6+col9 = col10 on every Dade row), ADA/ADM,
graduation (2018-19 race header variant handled; * → suppressed), school
grades (component prefixes in every year and all four 2022-23 sheets;
204+77+83+106 = 470, no cross-sheet duplicates), SRS (level-3 header unique;
2023 science layout; science_g8), membership apart from C8 (14 grade columns
in all 10 files; 2022-23 * → 0 rule; 2025-26 partial suppression),
race_membership apart from C10 (shares sum to exactly 100 in all 3,081
school-years), ELL apart from C12, derived (utilization, 5-yr change, K÷G5;
2025-26 fallbacks = 325 FISH carry-forward + 4), FISH/ABC (arithmetic
identities on 100% of rows), events (56 = 56), directory (dates 629/629 and
209/209; charter 232). Every metrics row is owned by its source's latest load.
Percentiles: direction correct, 0–1 scale matches PeerPill, no stale, no citywide for excluded schools. Peer groups: 417 groups per k, no hard-filter violations, no self-inclusion.
Residual-z: pivot_table(aggfunc="first") safe under the unique key; binomial
and Poisson SE algebra correct; LOWER_BETTER orientation correct; P1–P5
computed as documented (P3 is unweighted Spearman and does fail: science
2024-25, graduation, sesir); P2 fails in every ELA/math year and is reported as
a negative result on the page; ablation refits with the same weights; R2 and
R3 are real gates; CSV 18,297 = DB cells with z. Survey→enrollment
regression: design full rank, coefficient/SE/p and partial R² reproduced by an
independent QR solve (0.00549 ± 0.00101, p 1.5e-7, partial R² 0.1065).
Pivot-cache reader for 2021-22 verified field by field.
Enrollment packet: Fisher p-values equal scipy for all 7 cuts; lift cells 54/23 vs 206/12, universe 260/35, utilization n 316 median 66.7, excess cost $20,484,039 (26 small ES, reference median $14,051 over 47), 2025-26 sector totals all recomputed exactly in SQL. CCD vs state enrollment 2023-24: 518/518 within 5%.
App: tsc clean; 51/51 unit tests pass; no Miami page routes PERCENT through
the NYC fraction-rescaling formatter; every number on the enrollment,
residual-z and climate pages recomputes from the packets; robots gates read
the right verdicts; every registered artifact link is versioned; all 30
linked source-doc targets exist; generateStaticParams matches the registry
with 0 orphan keys.
1.6 Why the harness missed these
The harness is strong where it was designed to be strong (cell-level reconciliation of a loader against a re-read of the same file) and blind in four specific ways, each of which let a critical defect through:
- Shared inputs in "independent" checks. The survey reader shares SPECS, crosswalk, scale rule and n with the parser (M2); R1 shares n and eligibility with the engine (M1). C3, C5, M4 and the C1 scale all pass.
- No served-value invariants. Nothing asserts 0 ≤ PERCENT ≤ 100 (C3, C7), total ≥ max(category) and total ≈ Σ categories (C1), or response rate ≤ 100 (C2).
- No year-over-year distribution alarms. C8 (2017-18 school count drops by 32), C10 (a small-group median jumps from 0 to 0.96 for one year), M6 (response-rate means alternate 24/58/30/52/57) and C4 (a metric's mean halves for one year) are all visible as one-year shifts in a per-metric median/count series.
- Completeness is checked for one metric. Served-only and source-only cells never fail (M10).
Recommended additions, in the harness rather than in loaders: a
served_invariants.py step (range per unit; total-vs-components identities;
per-metric per-year count and median shift flagged at > 3 MAD from the
adjacent years, reported as a lift over the metric's own history); make R1
recompute n from enrollment_oct_g* under outcomes.yaml; make the survey
reader recompute suppression from Ncount and assert bounds; a per-component
served-only allowance in reconcile.
1.7 Which published conclusions survive
- Enrollment/closure packet (Tables 1–7, Fisher, lift, excess cost): survives. C8 dents the 2017-18 trend point and the 2022-23 five-year-change row count; the lift table uses 2024-25/2025-26 and is untouched.
- Survey → enrollment (parent recommend +6.9%/SD): the 2021-22 estimate reproduces independently and the 2021-22 file's recommend column is per-school constant (not affected by C2). But the pre-registered second cohort is silently absent (C4), so the result is one cohort's, and the story text must say so until 2022-23 is re-parsed.
- Residual-z (P1–P5, ablation, flags): every z will move after C9 (fit universe), C1 (sesir scale), M3 (variance formula) and M4 (science n). The P2 negative result (ρ(z, gains) < ρ(raw, gains)) is a statement about correlation ordering and is likely robust, but it must be re-run, not assumed. The ablation ordering (ELL, level, econ) is measured on the current fit and may change.
- Climate composites and ratings on
survey_*: not trustworthy until C2–C7 and M5 are fixed;survey_*_resourcesis empty; 2018-19 has served zeros; 2024-25 has values above 100. - SESIR total and its residual flags: wrong by 2× (C1); category rates are right.
- Race shares 2022-23 small groups: wrong (C10); 2018-19 and 2023-24 are within the CCD bound and the other years show no shift.
Part 2 — Duplication and shared code
Measured with two instruments: identical-line count (diff) and token
similarity (difflib ratio on comment-stripped, whitespace-collapsed text with
city names and id identifiers normalised). Token similarity is the honest
number where Miami's one-line style defeats line diffs. Git lineage confirms
copy-paste rather than convergence: Philly _lib.ts was created from NYC's,
Miami's from Philly's; two NYC percentile fixes were re-applied to Philly by
hand, and Miami's copy dropped the reconnect loop.
2.1 Evidence
| Area | Pair | Identical lines | Token sim | Note |
|---|---|---|---|---|
| Loader lib | NYC _lib.ts (86) vs Philly (217) | 74 | 0.41 | gitSha, batched, dotenv preamble byte-identical ×3 |
| Loader lib | Philly vs Miami _lib.ts (309) | 128 | 0.43 | startDataLoad/finishDataLoad differ only in delegate and sourceName/sourceId |
| Suppression tokens | 9 definitions, 4 token sets | NYC _fetch.ts:103; Philly 6 per-loader toNumber; Miami parseCell (only one distinguishing blank from token); Python ×4 | ||
| Percentiles | NYC vs Philly compute-percentiles.ts | 146/198 | 0.97 | rank() byte-identical ×3 plus a 4th copy in compute-proclivity.ts |
| Percentiles | Philly vs Miami | 0.78 | Miami dropped the reconnect loop | |
| Peer groups | NYC vs Philly | 79 | 0.54 | Same skeleton; three distance rules; Miami filters the subject on includeInDefaultComparisons, NYC/Philly the candidates |
| Python sidecars | Philly vs Miami discover.py | 117 | 0.61 | Miami is a strict superset |
| Python sidecars | load_env_local ×4, served_db_dsn ×5 | Three reconcile suppression policies (Philly served-side, Miami both-equal, NYC either) | ||
| Python sidecars | Philly vs Miami reconcile.py | 29 | 0.46 | Identical summary key set; Miami's reader: dispatch is already generic |
| Stories | Philly vs Miami [id]/page.tsx | 49/69 | 0.94 | Differ only in registry import, basePath, badge, Miami's evidence block |
| Stories | recipe/page.tsx | 35/48 | 0.99 | Story type declared 3× |
| Ratings | movers filters.tsx | 38 | 0.87 | Whitespace plus Philly's cut select |
| Ratings | outliers page.tsx | 60 | 0.67 | Same params/sort/pagination shell; different label columns and filters |
| Queries | getPhillyOutliers vs getMiamiOutliers | Same algorithm and memo wrapper; movers SQL identical modulo table | ||
| Metric registry | NYC / Philly / Miami | 0.20 | Same Def shape; Philly's cannot be imported by the app (lives in the seed script and binds Prisma) | |
| Methodology pages | Miami templated [metric] (254) vs Philly hand-written chronic-absenteeism (1,342) | Philly has one page for ~40 keys | ||
| Survey ingest | three architectures | 0.16–0.24 | Same output contract survey_{respondent}_{domain}; NYC has two parsers of the same files | |
| Residual-z | Miami engine vs scaffold engine.py | 0.13 | Re-implemented rather than imported; Philly has no engine | |
| Miami loaders | 9 per-outcome loaders pairwise | mean 0.45 | Skeleton (norm, discovery, header search, district filter, pad+universe, parseCell skip, load bookkeeping) copied 6–9×; norm has three charsets; essa_ppe is the divergent district filter |
Not meaningful duplicates (negative results): philly-packet.ts vs
miami-packet.ts (0.04), reconcile_computed.py pair (0.11), the two
qa-reports.ts (0.26), and the school pages (0.38) share only a shell; a full
merge of any of these would regress both cities.
2.2 Does the Prisma schema block sharing?
No. For everything written in raw SQL or Python (percentiles, movers, the
verify harness, the residual engine, the metric index) table and column names
are configuration. For Prisma-client code (loader upserts, peer groups,
lib/queries) the distinct delegates and unique-key names force a per-city
adapter of roughly 40 lines each, which removes the block without a
migration. What is preventing sharing today is copy-paste.
Schema options, assessed:
- A (recommended now): keep the three families; add the adapters.
- B (two-family): Philly and Miami are the same shape except Philly's
populationCutand Miami's residual columns. Addpopulation_cut TEXT NOT NULL DEFAULT 'n/a'to Miami and the residual columns to Philly (both additive, small). Do this when Philly gets the residual engine. - C (single
school_year_metrics(city, …)): requires a union subgroup enum with a rename map, resolving NYC's mixed 0–1/0–100 PERCENT storage (lib/format.ts:10), hand-written migrations (there is no versioned migrations directory to build on), acity =predicate on every query and index (a new bug class: "citywide" silently becoming "all cities"), and touching 10 NYC bulk-upsert sites, 8 Philly and 21 Miami loaders. Multi-week with high regression risk and little code-sharing gain over A. Only if a fourth city arrives, and then build that city on the unified shape rather than migrating NYC.
2.3 Decisions to record before refactoring (not inside it)
- Reconcile suppression policy. Three today: Philly counts a
served-suppressed cell as suppressed regardless of the source; Miami
requires both flags equal else mismatch; NYC treats either side suppressed
as suppressed. Miami's is the strictest and is the one that would have
caught C5 had the reader been independent. Proposal:
both_equalas the default, per-city override incomponents.yaml. - Peer-group distance and subject inclusion. NYC raw Euclidean + topic
Jaccard; Philly raw Euclidean on percent; Miami z-scored Euclidean with log
enrollment. Miami excludes non-default subjects from having a group; NYC
and Philly only exclude them as candidates. Unifying either changes
rankings in every city; keep
distanceconfigurable and makesubjectMustBeIncludedan explicit flag, then decide per city.
2.4 Ranked refactors
Payoff ≈ lines removable or capability gained; effort in days; ratio is payoff ÷ effort.
| # | Refactor | Payoff | Effort | Guard |
|---|---|---|---|---|
| 1 | Shared story routes: data/stories/types.ts + components/stories/{story-page,story-list,recipe-page}.tsx taking a registry; each city's three routes become 5–8-line wrappers | −320 lines at 0.94–0.99 sim; NYC can join | 0.5 | routes unit test + render smoke test with a two-story fake registry |
| 2 | compute-percentiles.ts --city with a PercentileCityConfig (tables, id column, extra key, combo filter) and _common/rank.ts used by compute-proclivity too | −300 lines; restores Miami's reconnect; 4 rank → 1 | 0.5 | COPY percentiles per city before/after, expect byte-equality; unit-test rank() on ties |
| 3 | scripts/loaders/_common/ (city config, gitSha/batched/provenance readers, one parseCell with per-city token additions, makeFactWriter) applied to Philly's 8 inline upsert loops and 6 toNumbers; fold sdp_pssa_keystone_2023-24.ts into a year argument | −430 lines; one suppression definition in TS | 1.5 | each city's reconcile --all stays PASS; vitest for parseCell (blank ≠ token is load-bearing for school_grades.ts:67) |
| 4 | pipeline_common/{env,sources,verify/served_db,verify/reconcile} with a frozen City dataclass; adopt Miami's reader: components schema; explicit suppression policy (decision 1) | −330 lines; 4 load_env_local + 5 served_db_dsn → 1; unified __validation.json so the TS qa-reports types merge | 2 | golden test of reconcile_one per policy; diff committed __validation.json per city |
| 5 | discover.py → pipeline_common/discover.py (Miami's superset + City.url_rewrite/prime_session) | −180 lines; Philly gains --force, pdf/json inspectors, header detection | 1 | re-run discover for both cities, diff findings |
| 6 | Miami TableSpec + dadeRows iterator: 5 of 9 loaders become specs (SRS, absenteeism, graduation, ESSA PPE, school-grades components); membership, race, ELL, SESIR keep custom code over the iterator | −150 lines; kills the essa_ppe filter-divergence bug class; next FLDOE source ≈ 20 lines | 2 | 25 harness components + fact-level parity dump per loader |
| 7 | Metric registry as pure data per city + shared components/methodology/metric-page.tsx (today's Miami page with the three Miami branches behind qa.crossPublisherFor) + export_metric_index --city | Philly: 1 → ~40 metric pages with QA banners; one seed script | 2 | vitest: every loaded key has a registry entry and vice versa; generateStaticParams count snapshot |
| 8 | Ratings pages + queries via CityRatingsAdapter / CityDb | −420 lines at 0.67–0.87 sim | 3–4 | snapshot getOutliers for a fixed (metric, year) per city |
| 9 | City-agnostic residual-z engine built as Philly's engine (FrameSource protocol; P-checks in outcomes.yaml); add residual columns to Philly (schema option B) | avoids a second 384-line engine; brings the scaffold's View-2 trends | 3 | R1/R2 generalised with City, run for both |
| 10 | Peer-group engine with PeerSpec (decision 2 first) | −275 lines | 2 + decision | dump (school, k, sorted peers) before/after; set-equality when distance and inclusion flags match |
Suggested order: 1 → 2 → 3 → 4 → 5 → 6 → 7 → 8 → 9 → 10. Items 3 and 4
produce the TS and Python city adapters that 6–9 consume; 1, 2 and 5 are
independent. Two deletions need no refactor: the NYC TS survey parser that
duplicates survey_to_csv.py, and sdp_pssa_keystone_2023-24.ts.
Survey ingest (three architectures) is deliberately not in the top ten as a
merge: the parsers are per-publisher and irreducible. What can be shared is
the intermediate (school_id, respondent, year, item_no, item_text, pct_positive, n_responses, suppressed), Miami's crosswalk builder (after M5
is fixed) as pipeline_common/survey/crosswalk.py, and the composite loader
with a FactWriter injected. NYC's regex MAPPINGS would become a committed,
reviewable survey_items.csv; some NYC values would move, which is a
documented diff, not a regression.
Caveat on the measurement: line counts proxy maintenance cost, they do not
measure it. The stronger argument in each area is the propagation record:
percentile fixes re-applied by hand, Miami's reconnect loop lost, the
essa_ppe filter diverging, and Miami's stricter suppression policy not
reaching the other two harnesses.
Part 3 — Proposed fix order (as executed; see 09_review_fixes_2026-09-16.md)
Nothing below has been done.
Batch A: served numbers (C1–C13). Fix loaders and parser; reload SESIR,
membership 2016-17/2017-18 + derived, race 2022-23, climate survey (all
years), ELL, CCD staff; remove fishCapacityPermanent from the directory
update payload. Then re-run the harness, the residual-z engine (with C9
removed), the survey→enrollment test (now with a real 2022-23 cohort), the
enrollment packet, peer groups and percentiles, and regenerate every packet
JSON. Estimated 1.5 days including re-validation. Every served number that
changes will be listed in the commit.
Batch B: method and harness (M1–M12 and §1.6). Independent n in R1;
independent survey reader with bounds and Ncount-based suppression; τ² with
(n − k) correction; science n-rule 9–10 with the choice added to the ablation;
crosswalk stemming/alias table and re-derived core; students response-rate
denominator break marked; race bound using the loader's hidden count and a
suppression threshold for high-U schools; PK-suppressed handling; regime
boundaries vs flagged years; served-only allowance; funnel counted before the
join; peer-group fallback for <6-candidate cells. Plus the new
served_invariants.py step. Estimated 2 days. Re-run everything from A.
Batch C: text and docs (S1–S10, §1.4 UI items). Rewrite the affected
story sentences and recipes from the re-run packets, fix the notes, the stale
page text, the tooltip, inputsOf, evidence-version registration, dropdown
filtering, NULLs-last sort, sources.yaml. Estimated 0.5 day, after A and B
so the numbers are final.
Batch D: refactors 1–5 (low risk, ~5.5 days), then 6–7, then decisions for 8–10.
Order rationale: A first because readers see those numbers today; B before C because the story text depends on re-run results; D after the numbers are stable so the parity guards have a correct baseline to compare against.