Method bake-off — design memo (Phase 3, written before computation)

Status: design lock, pre-computation. This memo fixes the candidates, covariates, universes, windows, and evaluation criteria for the chronic-absenteeism "vs-expectation" bake-off mandated by the deep-dive plan, Phase 3. It is written before any bake-off number exists so that results cannot quietly reshape the method choices. Every figure cited below comes from a committed artifact; three claims that do not yet have a committed citation are explicitly marked [unverified-in-repo] and must be verified before any public use. Findings ledger anchors (A.x) refer to 08_absenteeism_findings.md.

2026-06-11 · branch kx/bakeoff-prep · no analysis was run for this memo.

Decisions locked here

• D1. Four candidates enter; we exit with 1–2 recommended measures (status + trend), not four.
• D2. Candidate definitions are frozen as specified in §2; any deviation found necessary during computation is logged as an amendment to this memo before results are compared.
• D3. Covariate set for the regression candidates is frozen in §3, including the new ENI / top-code / STH-share fields; race composition is included in the bake-off but flagged as an open editorial decision (§3).
• D4. Peer/expectation vintage: year-specific expectations for status; trends of year-specific residuals for trend claims; fixed-vintage K-NN trends as a robustness check only (§4).
• D5. Per-year peer groups are computed offline; no schema or serving changes until a winner is chosen (productionization is a later PR, per the additive-columns pattern of 03_integration_plan.md C1).
• D6. Universes/windows per §5: adjusted candidates = district schools, post-COVID window; charters excluded from adjusted candidates v1; D75 separate; 2024-25 grades-1–8 carries A.11 source attribution.
• D7. Evaluation criteria and their order are pre-committed in §6; the stability baseline to beat and the orthogonality test are named before any candidate is scored.
• D8. Outlier gate carried from A.15: persistence in ≥2 of the 3 most recent post-COVID years (2022-23 → 2024-25) + size-aware uncertainty (spec §3.1's se_i down-weighting). This parameterizes the plan's Phase 4 "persist ≥2 years" gate; it is chosen now, not after seeing which schools clear it.
• D9. Named-school lists produced by the bake-off remain gated preliminary until the Chalkbeat checkpoint (§7).

1. Purpose & question

Chalkbeat's outlier question is directional in both senses: which schools have much higher — or much lower — chronic absenteeism than socioeconomically similar peers? (plan, Context). The raw rate cannot answer it: raw CA ranks schools substantially by poverty (r ≈ 0.57 with pct_econ_dis; 01_descriptives.md, SES section), the Detroit/Michigan-PEER caution the plan adopts (raw attendance vs disadvantage −0.56, value-added +0.42; plan, Phase 3). We need a "vs-expectation" measure for status (where a school sits relative to expectation this year) and for trend (whether it is moving relative to expectation), balancing the project's original parameter 3 — accessibility for a reporter vs statistical defensibility — the same tension the spec resolves per-surface (intuitive peer display, deterministic model flag; spec §3.1).

2. Candidates (4)

(a) K-NN peer percentile (live). The production method: K=40 nearest neighbors under hard filters (same grade band + admission bucket) with weighted Euclidean distance over 7 demographic shares + topic-Jaccard (×0.5), midrank percentile within the stored peer set, LOWER_BETTER inverted (01_inventory.md rows "Peer groups (KNN)" / "Outlier percentile"; scripts/loaders/nycenet-comparison-groups.ts, compute-percentiles.ts). Known weakness per 02_gap_analysis.md #3: flags depend directly on group-construction choices (K, distance weights, the ×30 Jaccard scale). Team-reported sensitivity: moving K to 20/80 shifts schools ~10–15 percentile points [unverified-in-repo — no committed artifact carries this number; reproduce it as part of the bake-off sensitivity panel].

(b) NYC-official comparison-group average (Snapshot cavg/pavg). Benchmark, not our method: per school-year cavg_student_chronic_absent (report years ≤2023 = SY ≤2022-23) and pavg_chronic_absent_* (2024+) from data/quality/snapshot-values.json. Caveats, all committed: Snapshot values are grade-band rates, not school-wide (A.6) — compare like bands only; the feed has zero non-null cavg in SY 2020-21 and pavg is unvalidated and on the complement scale (CENSUS.md §4–5), so within the scored window the validated benchmark exists only for 2022-23 (2021-22 is robustness-only after the D6 amendment); we observe the average, not the group membership (CENSUS.md §4 makes no claim about how NYC constructs the groups). NYC reportedly changed the comparison-group methodology in the 2023 SQRs from student-matching to a Blueprint-Labs regression approach [unverified-in-repo — cite the SQR Educator Guide before using across the 2022-23/2023-24 boundary].

(c) WLS residual-z (scaffold engine). Spec §3.1 as implemented in reference-scaffold/pipeline/analysis/engine.py (described, not run): design = intercept + numeric covariates + drop-first dummies; WLS via sqrt(N)-scaled rows (weights = denominator N); z = resid / sqrt(s² + se_i²) with s = weighted residual SD and se_i = proportion sampling SE; direction-aware flags at |z| ≥ 2 (desirable_direction: low for CA). The descriptives preview shows the approach is viable: R² 0.43–0.50/yr, residual YoY r 0.884 (01_descriptives.md; A.16).

(d) Lagged-DV residual (school-level PEER analogue). Candidate (c) plus prior-year school CA as a regressor alongside composition — the spec's optional prior_achievement covariate that "converts the residual to a growth-like measure" (spec §3.1 step 2), and the school-level analogue of Michigan PEER's attendance value-added named in the plan, Phase 3 item 4. Interpretation differs from (c) by construction: it asks "did this school beat its own track record, given composition," not "does it beat similar schools."

3. Covariates (candidates c/d)

Frozen set, all district-school rows, demographics from school_year_demographics as repaired on this branch (A.4):

Race-covariate editorial question (open, named now): the preview's race coefficients are large conditional on poverty and collinear with it; 01_descriptives.md caveats states that including race in a public "expected CA" model is an editorial decision, not a statistical one. The bake-off runs candidates (c)/(d) with and without race composition and reports both; the editorial call is made at the Chalkbeat checkpoint, not by this memo. Descriptive use only either way (spec §3.4: associational, not causal).

ENI vs pct_econ_dis redundancy is expected; the bake-off reports the model with both and with each alone (sensitivity row), rather than choosing blind.

4. Peer/expectation vintage — the trend-methodology axis

Three ways to anchor "expectation" over time, with different failure modes:

1. Year-specific groups/expectations — peers/model refit each year on that year's composition. Right for status ("vs similar schools now"); for trends it conflates a school's own movement with peer churn (your percentile can move because your peer set changed).
2. Fixed-vintage peers — one peer set (e.g., 2021-22 composition) followed across years. Clean trajectories (same comparison set every year) but increasingly stale composition, and NYC enrollment composition is moving (STH share 10.68% → 15.35% over the window; A.4).
3. Year-specific residuals, trend on the residuals — spec §3.2: fit the slope of within-year z across comparable years (trends.py: OLS slope, t-test with n−2 df, slope + p thresholds, leave-one-year-out sensitivity). Composition change is netted out by construction each year, so the trend reads "gaining/losing vs demographically similar schools."

Decision (D4): status candidates are evaluated under (i); trend claims under (iii); (ii) is computed only as a robustness check on K-NN trends (does a fixed-vintage peer trajectory tell the same story as the residual slope?). Note the live K-NN loader is none of these cleanly — it builds one group per school from its latest demographics year (nycenet-comparison-groups.ts, latest-year selection), so the bake-off computes per-year K-NN groups offline (D5); no schema changes until a winner is chosen.

5. Universes & windows

• Adjusted candidates (a, c, d): district schools (include_in_default_comparisons), the universe of 01_descriptives.md (~1,450–1,470 schools/yr). Demographics exist only from 2020-21 (01_descriptives.md caveats), so adjusted measures are computable 2020-21 onward; per the comparability conventions (07_absenteeism_metric_alignment.md §5) 2020-21 is regime-excluded. Scored window: 2022-23 → 2024-25 (amended 2026-06-11 per D2, on review: 2021-22 — the pandemic-peak year — is the noisiest in the post-COVID set (its YoY correlation into 2022-23 is 0.892 vs 0.928/0.936 for later pairs, A.22) and its Omicron-winter absences are least attributable to schools; dropping it also aligns the scored window with the D8 gate years). 2021-22 stays as a clearly labeled robustness year: residuals are computed but not scored, and any school whose outlier status flips when 2021-22 is included is reported in the flip-diff tables. The 2018-19 baseline has no same-year demographics; it enters only as a labeled robustness variant using nearest-year covariates, never in headline tables. Consequence for trends: three scored points cannot support slope significance tests — trend claims are framed as direction + magnitude across 2022-23 → 2024-25 (endpoint change in within-year z), not fitted-slope p-values.
• Charters: excluded from adjusted candidates v1. No charter demographics are ingested (the DOE Demographic Snapshot is district-only; charter demographics from NYSED is a named Phase 1 follow-up — plan, Phase 1). This is a coverage gap, not a judgment: charters appear in unadjusted descriptive context only (with the A.13 Success Academy non-reporting hole and A.20/A.21 caveats attached).
• D75 separate, as always; D79 has no school-level data from either publisher (A.12).
• 2024-25 grades 1–8 is publisher-dependent (A.11): any candidate score touching 2024-25 EM-grade rates carries source attribution (DOE InfoHub basis stated).
• Benchmark (b) is additionally limited per §2: band-basis comparisons only; within the scored window the validated cavg covers 2022-23 only (plus 2021-22 in the robustness set) — criterion 4 therefore judges a single scored year and is weighted accordingly.

6. Evaluation criteria (pre-committed)

Scored in this order; a candidate that fails 1–2 is not rescued by 3–5.

1. Year-over-year stability. Baseline to beat: raw school-level CA levels at r ≈ 0.92–0.94 on matched school sets (A.22: 0.917/0.930/0.943; 01_descriptives.md Table 1: 0.892–0.936). Adjustment usually costs stability (preview: residual r 0.884 vs raw 0.937 on the same schools); the question is how much, reported per candidate.
2. SES-orthogonality. corr(measure, ENI/poverty) ≈ 0 is by construction for (c)/(d) — not evidence. The real test is correlation with omitted factors: STH share where not in the model, borough, school size — the PEER diagnostic extended to variables the model never saw (01_descriptives.md, SES finding 4).
3. Cross-method agreement. Top/bottom-decile list overlap between candidates, REL-Midwest-style measure-comparison framing [unverified-in-repo — external literature reference; add the citation when the comparison table is built]. Expectation set now: ~40–60% overlap; that is the finding, not a failure — it quantifies how much the method choice itself drives who gets named.
4. Agreement with the NYC-official benchmark. Does our peer expectation track NYC's published comparison-group average (band-matched, validated years per §2b)? Large divergence = investigate, per the spec's convergence check (spec §3.5).
5. Interpretability (qualitative row). Can a reporter state the measure in one sentence without a footnote? Scored qualitatively per candidate; this is parameter 3 made explicit, and it is allowed to break ties — but only ties.

Gate carried into all named-outlier output (D8): multi-year persistence (≥2 of the 3 most recent post-COVID years) + size-aware uncertainty (small-N schools need larger raw gaps; spec §3.1 / A.15).

7. Deliverables

1. Comparison-stats table: criteria 1–5 × candidates (a)–(d), one row per criterion, numbers + the qualitative row.
2. Flip-diff tables (pattern from 03_integration_plan.md C1): for each candidate pair, the schools one flags that the other does not, with N, proclivity decile, school type — so the recommendation is decided from data, not taste.
3. A recommendation (1–2 measures: which for status, which for trend), then the Chalkbeat checkpoint. Named-school lists produced along the way remain gated preliminary (ledger status conventions; A.21/A.22 precedent) until that meeting.

8. Out of scope

• Causal claims — residuals are associational; omitted variables can drive them (spec §3.4).
• Strategy attribution — PEER links value-added to school-practice surveys; we have no practice data and make no "why" claims.
• Student-level value-added — no student-level data; all candidates are school-level. The pseudo-cohort machinery (A.22) is descriptive context, not a VA estimator.

9. Amendments (2026-06-11, per D2): recovery metrics + targeted outlier qualifiers

D10 — Recovery metrics (2018-19-anchored), a third measure family. Descriptive companions to the adjusted candidates — raw values, no covariates required, so the 2018-19 anchor is usable directly. Primary: recovery gap = CA(endpoint) − CA(2018-19), endpoint = the mean of 2023-24 and 2024-25 (damps single-year endpoint noise; single-year variants reported as robustness). Secondary: recovery fraction = (peak − endpoint)/(peak − 2018-19), reported only where the surge (peak − baseline) ≥ 5pp. Caveats carried on every output: composition change since 2018-19 is only partially observable (decomposable within 2020-21→endpoint; the 2018-19→2020-21 leg is unobservable and stated as such); the 2024-25 endpoint carries A.11 attribution for grades-1–8 schools (2023-24-only endpoint variant reported alongside). Recovery outliers must hold under both endpoint choices (the D8 spirit).

D11 — Composition-stability screen. Every outlier table gains a composition-stable flag: over 2020-21→endpoint, max drift below pre-committed thresholds across ENI, pct_econ_dis, pct_swd, pct_ell, race shares (each < 5pp absolute), and |Δ log enrollment| < 0.2 — thresholds fixed here, before any school lists are computed; the empirical pass rate is reported, not tuned. Narrative priority goes to flagged schools: where composition held still, the residual change is most plausibly school-attributable. Limitation stated: drift before 2020-21 is unobservable (no demographics).

D12 — Cohort-coherence score. Using the per-grade pseudo-cohort machinery (A.22): for each school × adjacent-year pair, each observable transition g→g+1 gets a cohort change net of the citywide median for that transition/year (removing U-shape maturation); the school's coherence = mean sign-agreement across its transitions (≥4 transitions required, cohort N ≥ 30, churn > 20% flagged). High-coherence synchronized shifts serve as corroborating evidence on recovery/status outliers and as an independent discovery list. Pseudo-cohort caveats apply throughout (school-level, not linked students).