# Method bake-off — design memo (Phase 3, written before computation)

> **Status: design lock, pre-computation.** This memo fixes the candidates, covariates, universes, windows, and evaluation criteria for the chronic-absenteeism "vs-expectation" bake-off mandated by [the deep-dive plan, Phase 3](../../design/06_absenteeism_deep_dive_plan.md). It is written **before** any bake-off number exists so that results cannot quietly reshape the method choices. Every figure cited below comes from a committed artifact; three claims that do **not** yet have a committed citation are explicitly marked `[unverified-in-repo]` and must be verified before any public use. Findings ledger anchors (`A.x`) refer to [08_absenteeism_findings.md](../../design/08_absenteeism_findings.md).

*2026-06-11 · branch `kx/bakeoff-prep` · no analysis was run for this memo.*

## Decisions locked here

- **D1.** Four candidates enter; we exit with **1–2 recommended measures** (status + trend), not four.
- **D2.** Candidate definitions are frozen as specified in §2; any deviation found necessary during computation is logged as an amendment to this memo *before* results are compared.
- **D3.** Covariate set for the regression candidates is frozen in §3, including the new ENI / top-code / STH-share fields; race composition is included in the bake-off but flagged as an open editorial decision (§3).
- **D4.** Peer/expectation vintage: **year-specific** expectations for status; **trends of year-specific residuals** for trend claims; fixed-vintage K-NN trends as a robustness check only (§4).
- **D5.** Per-year peer groups are computed **offline**; no schema or serving changes until a winner is chosen (productionization is a later PR, per the additive-columns pattern of [03_integration_plan.md C1](../../design/03_integration_plan.md)).
- **D6.** Universes/windows per §5: adjusted candidates = district schools, post-COVID window; charters excluded from adjusted candidates v1; D75 separate; 2024-25 grades-1–8 carries A.11 source attribution.
- **D7.** Evaluation criteria and their order are pre-committed in §6; the stability baseline to beat and the orthogonality test are named before any candidate is scored.
- **D8.** Outlier gate carried from A.15: **persistence in ≥2 of the 3 most recent post-COVID years (2022-23 → 2024-25) + size-aware uncertainty** (spec §3.1's `se_i` down-weighting). This parameterizes the plan's Phase 4 "persist ≥2 years" gate; it is chosen now, not after seeing which schools clear it.
- **D9.** Named-school lists produced by the bake-off remain **gated preliminary** until the Chalkbeat checkpoint (§7).

## 1. Purpose & question

Chalkbeat's outlier question is directional in both senses: *which schools have much higher — or much lower — chronic absenteeism than socioeconomically similar peers?* ([plan, Context](../../design/06_absenteeism_deep_dive_plan.md)). The raw rate cannot answer it: raw CA ranks schools substantially by poverty (r ≈ 0.57 with `pct_econ_dis`; [01_descriptives.md, SES section](01_descriptives.md)), the Detroit/Michigan-PEER caution the plan adopts (raw attendance vs disadvantage −0.56, value-added +0.42; [plan, Phase 3](../../design/06_absenteeism_deep_dive_plan.md)). We need a "vs-expectation" measure for **status** (where a school sits relative to expectation this year) and for **trend** (whether it is moving relative to expectation), balancing the project's original parameter 3 — **accessibility for a reporter vs statistical defensibility** — the same tension the spec resolves per-surface (intuitive peer display, deterministic model flag; [spec §3.1](../../design/nyc-school-outcomes-spec.md)).

## 2. Candidates (4)

**(a) K-NN peer percentile (live).** The production method: K=40 nearest neighbors under hard filters (same grade band + admission bucket) with weighted Euclidean distance over 7 demographic shares + topic-Jaccard (×0.5), midrank percentile within the stored peer set, `LOWER_BETTER` inverted ([01_inventory.md rows "Peer groups (KNN)" / "Outlier percentile"](../../design/01_inventory.md); `scripts/loaders/nycenet-comparison-groups.ts`, `compute-percentiles.ts`). Known weakness per [02_gap_analysis.md #3](../../design/02_gap_analysis.md): flags depend directly on group-construction choices (K, distance weights, the ×30 Jaccard scale). Team-reported sensitivity: moving K to 20/80 shifts schools ~10–15 percentile points `[unverified-in-repo — no committed artifact carries this number; reproduce it as part of the bake-off sensitivity panel]`.

**(b) NYC-official comparison-group average (Snapshot `cavg`/`pavg`).** Benchmark, not our method: per school-year `cavg_student_chronic_absent` (report years ≤2023 = SY ≤2022-23) and `pavg_chronic_absent_*` (2024+) from `data/quality/snapshot-values.json`. Caveats, all committed: Snapshot values are **grade-band** rates, not school-wide (A.6) — compare like bands only; the feed has **zero non-null cavg in SY 2020-21** and `pavg` is **unvalidated and on the complement scale** ([CENSUS.md §4–5](../../qa_reports/snapshot/CENSUS.md)), so within the scored window the validated benchmark exists only for **2022-23** (2021-22 is robustness-only after the D6 amendment); we observe the **average, not the group membership** (CENSUS.md §4 makes no claim about how NYC constructs the groups). NYC reportedly changed the comparison-group methodology in the 2023 SQRs from student-matching to a Blueprint-Labs regression approach `[unverified-in-repo — cite the SQR Educator Guide before using across the 2022-23/2023-24 boundary]`.

**(c) WLS residual-z (scaffold engine).** [Spec §3.1](../../design/nyc-school-outcomes-spec.md) as implemented in [`reference-scaffold/pipeline/analysis/engine.py`](../../design/reference-scaffold/pipeline/analysis/engine.py) (described, not run): design = intercept + numeric covariates + drop-first dummies; WLS via sqrt(N)-scaled rows (weights = denominator N); `z = resid / sqrt(s² + se_i²)` with `s` = weighted residual SD and `se_i` = proportion sampling SE; direction-aware flags at `|z| ≥ 2` (`desirable_direction: low` for CA). The descriptives preview shows the approach is viable: R² 0.43–0.50/yr, residual YoY r 0.884 ([01_descriptives.md](01_descriptives.md); A.16).

**(d) Lagged-DV residual (school-level PEER analogue).** Candidate (c) plus **prior-year school CA** as a regressor alongside composition — the spec's optional `prior_achievement` covariate that "converts the residual to a growth-like measure" ([spec §3.1 step 2](../../design/nyc-school-outcomes-spec.md)), and the school-level analogue of Michigan PEER's attendance value-added named in [the plan, Phase 3 item 4](../../design/06_absenteeism_deep_dive_plan.md). Interpretation differs from (c) by construction: it asks "did this school beat its own track record, given composition," not "does it beat similar schools."

## 3. Covariates (candidates c/d)

Frozen set, all district-school rows, demographics from `school_year_demographics` as repaired on this branch (A.4):

| Covariate | Notes / provenance |
|---|---|
| `eni` (Economic Need Index) | **New** (A.4 fix): populated on every row; top-coded `Above 95%` → 0.95 |
| `pct_econ_dis` + `pct_econ_dis_topcoded` | A.4 fix: top-code stored as 0.95 + flag (was NULL for 8–18%/yr, biased high-CA schools out); the flag enters as a dummy |
| `pct_temp_housing` | Derived STH share from the attendance STH sheet, `STH_N/(STH_N+NotSTH_N)`, attendance-contributing basis — **different basis from the demographics file**; full series at `data/analysis/sth-shares.json` (A.4) |
| `pct_ell`, `pct_swd` | as in the descriptives preview |
| Race composition (`pct_black`, `pct_hispanic`, `pct_asian`, `pct_white` drop-one) | see editorial note below |
| `log(total_enrollment)` | enrollment is negatively correlated with CA (r ≈ −0.3; [01_descriptives.md](01_descriptives.md)) |
| Grade-band + admission-category terms | **Required**, not optional: the worse-than-predicted tail of the preview model concentrates in small D2 high schools — a school-type confound (A.16); the K-NN hard filters handle this by construction, the regression must handle it by covariates ([01_descriptives.md, implications](01_descriptives.md)) |

**Race-covariate editorial question (open, named now):** the preview's race coefficients are large conditional on poverty and collinear with it; [01_descriptives.md caveats](01_descriptives.md) states that including race in a public "expected CA" model **is an editorial decision, not a statistical one**. The bake-off runs candidates (c)/(d) **with and without** race composition and reports both; the editorial call is made at the Chalkbeat checkpoint, not by this memo. Descriptive use only either way (spec §3.4: associational, not causal).

ENI vs `pct_econ_dis` redundancy is expected; the bake-off reports the model with both and with each alone (sensitivity row), rather than choosing blind.

## 4. Peer/expectation vintage — the trend-methodology axis

Three ways to anchor "expectation" over time, with different failure modes:

1. **Year-specific groups/expectations** — peers/model refit each year on that year's composition. Right for **status** ("vs similar schools *now*"); for trends it conflates a school's own movement with **peer churn** (your percentile can move because your peer set changed).
2. **Fixed-vintage peers** — one peer set (e.g., 2021-22 composition) followed across years. Clean trajectories (same comparison set every year) but increasingly **stale composition**, and NYC enrollment composition is moving (STH share 10.68% → 15.35% over the window; A.4).
3. **Year-specific residuals, trend on the residuals** — [spec §3.2](../../design/nyc-school-outcomes-spec.md): fit the slope of within-year `z` across comparable years ([`trends.py`](../../design/reference-scaffold/pipeline/analysis/trends.py): OLS slope, t-test with n−2 df, slope + p thresholds, leave-one-year-out sensitivity). Composition change is netted out **by construction** each year, so the trend reads "gaining/losing vs demographically similar schools."

**Decision (D4):** status candidates are evaluated under (i); trend claims under (iii); (ii) is computed **only** as a robustness check on K-NN trends (does a fixed-vintage peer trajectory tell the same story as the residual slope?). Note the live K-NN loader is none of these cleanly — it builds one group per school from its **latest** demographics year (`nycenet-comparison-groups.ts`, latest-year selection), so the bake-off computes **per-year K-NN groups offline** (D5); no schema changes until a winner is chosen.

## 5. Universes & windows

- **Adjusted candidates (a, c, d): district schools** (`include_in_default_comparisons`), the universe of [01_descriptives.md](01_descriptives.md) (~1,450–1,470 schools/yr). Demographics exist only from **2020-21** ([01_descriptives.md caveats](01_descriptives.md)), so adjusted measures are computable 2020-21 onward; per the comparability conventions ([07_absenteeism_metric_alignment.md §5](../../design/07_absenteeism_metric_alignment.md)) 2020-21 is regime-excluded. **Scored window: 2022-23 → 2024-25** *(amended 2026-06-11 per D2, on review: 2021-22 — the pandemic-peak year — is the noisiest in the post-COVID set (its YoY correlation into 2022-23 is 0.892 vs 0.928/0.936 for later pairs, [A.22](../../design/08_absenteeism_findings.md)) and its Omicron-winter absences are least attributable to schools; dropping it also aligns the scored window with the D8 gate years)*. **2021-22** stays as a clearly **labeled robustness year**: residuals are computed but not scored, and any school whose outlier status flips when 2021-22 is included is reported in the flip-diff tables. The **2018-19 baseline** has no same-year demographics; it enters only as a labeled robustness variant using nearest-year covariates, never in headline tables. Consequence for trends: three scored points cannot support slope significance tests — trend claims are framed as **direction + magnitude across 2022-23 → 2024-25** (endpoint change in within-year z), not fitted-slope p-values.
- **Charters: excluded from adjusted candidates v1.** No charter demographics are ingested (the DOE Demographic Snapshot is district-only; charter demographics from NYSED is a named Phase 1 follow-up — [plan, Phase 1](../../design/06_absenteeism_deep_dive_plan.md)). This is a coverage gap, not a judgment: charters appear in **unadjusted descriptive context** only (with the A.13 Success Academy non-reporting hole and A.20/A.21 caveats attached).
- **D75 separate, as always**; D79 has no school-level data from either publisher (A.12).
- **2024-25 grades 1–8 is publisher-dependent (A.11):** any candidate score touching 2024-25 EM-grade rates carries source attribution (DOE InfoHub basis stated).
- Benchmark (b) is additionally limited per §2: band-basis comparisons only; within the scored window the validated `cavg` covers **2022-23 only** (plus 2021-22 in the robustness set) — criterion 4 therefore judges a single scored year and is weighted accordingly.

## 6. Evaluation criteria (pre-committed)

Scored in this order; a candidate that fails 1–2 is not rescued by 3–5.

1. **Year-over-year stability.** Baseline to beat: raw school-level CA levels at r ≈ 0.92–0.94 on matched school sets (A.22: 0.917/0.930/0.943; [01_descriptives.md Table 1](01_descriptives.md): 0.892–0.936). Adjustment usually costs stability (preview: residual r 0.884 vs raw 0.937 on the same schools); the question is **how much**, reported per candidate.
2. **SES-orthogonality.** corr(measure, ENI/poverty) ≈ 0 is **by construction** for (c)/(d) — not evidence. The real test is correlation with **omitted** factors: STH share where not in the model, borough, school size — the PEER diagnostic extended to variables the model never saw ([01_descriptives.md, SES finding 4](01_descriptives.md)).
3. **Cross-method agreement.** Top/bottom-decile list overlap between candidates, REL-Midwest-style measure-comparison framing `[unverified-in-repo — external literature reference; add the citation when the comparison table is built]`. Expectation set now: **~40–60% overlap; that is the finding, not a failure** — it quantifies how much the method choice itself drives who gets named.
4. **Agreement with the NYC-official benchmark.** Does our peer expectation track NYC's published comparison-group average (band-matched, validated years per §2b)? Large divergence = investigate, per the spec's convergence check ([spec §3.5](../../design/nyc-school-outcomes-spec.md)).
5. **Interpretability (qualitative row).** Can a reporter state the measure in one sentence without a footnote? Scored qualitatively per candidate; this is parameter 3 made explicit, and it is allowed to break ties — but only ties.

**Gate carried into all named-outlier output (D8):** multi-year persistence (≥2 of the 3 most recent post-COVID years) + size-aware uncertainty (small-N schools need larger raw gaps; spec §3.1 / A.15).

## 7. Deliverables

1. **Comparison-stats table**: criteria 1–5 × candidates (a)–(d), one row per criterion, numbers + the qualitative row.
2. **Flip-diff tables** (pattern from [03_integration_plan.md C1](../../design/03_integration_plan.md)): for each candidate pair, the schools one flags that the other does not, with N, proclivity decile, school type — so the recommendation is decided from data, not taste.
3. **A recommendation** (1–2 measures: which for status, which for trend), then the **Chalkbeat checkpoint**. Named-school lists produced along the way remain **gated preliminary** (ledger status conventions; A.21/A.22 precedent) until that meeting.

## 8. Out of scope

- **Causal claims** — residuals are associational; omitted variables can drive them ([spec §3.4](../../design/nyc-school-outcomes-spec.md)).
- **Strategy attribution** — PEER links value-added to school-practice surveys; we have no practice data and make no "why" claims.
- **Student-level value-added** — no student-level data; all candidates are school-level. The pseudo-cohort machinery (A.22) is descriptive context, not a VA estimator.

## 9. Amendments (2026-06-11, per D2): recovery metrics + targeted outlier qualifiers

**D10 — Recovery metrics (2018-19-anchored), a third measure family.** Descriptive companions to the adjusted candidates — raw values, no covariates required, so the 2018-19 anchor is usable directly. Primary: **recovery gap** = CA(endpoint) − CA(2018-19), endpoint = the **mean of 2023-24 and 2024-25** (damps single-year endpoint noise; single-year variants reported as robustness). Secondary: **recovery fraction** = (peak − endpoint)/(peak − 2018-19), reported only where the surge (peak − baseline) ≥ 5pp. Caveats carried on every output: composition change since 2018-19 is only partially observable (decomposable within 2020-21→endpoint; the 2018-19→2020-21 leg is unobservable and stated as such); the 2024-25 endpoint carries [A.11](../../design/08_absenteeism_findings.md) attribution for grades-1–8 schools (2023-24-only endpoint variant reported alongside). Recovery outliers must hold under both endpoint choices (the D8 spirit).

**D11 — Composition-stability screen.** Every outlier table gains a **composition-stable** flag: over 2020-21→endpoint, max drift below pre-committed thresholds across ENI, pct_econ_dis, pct_swd, pct_ell, race shares (each < 5pp absolute), and |Δ log enrollment| < 0.2 — thresholds fixed here, before any school lists are computed; the empirical pass rate is reported, not tuned. Narrative priority goes to flagged schools: where composition held still, the residual change is most plausibly school-attributable. Limitation stated: drift before 2020-21 is unobservable (no demographics).

**D12 — Cohort-coherence score.** Using the per-grade pseudo-cohort machinery ([A.22](../../design/08_absenteeism_findings.md)): for each school × adjacent-year pair, each observable transition g→g+1 gets a cohort change net of the citywide median for that transition/year (removing U-shape maturation); the school's **coherence** = mean sign-agreement across its transitions (≥4 transitions required, cohort N ≥ 30, churn > 20% flagged). High-coherence synchronized shifts serve as corroborating evidence on recovery/status outliers and as an independent discovery list. Pseudo-cohort caveats apply throughout (school-level, not linked students).