# Research notes — methods & citations for the clustering/decile bake-off

Working notes feeding `methodology/clustering-decile-bakeoff.md`. Not the final report.

## What NYC actually does (the "proprietary"/official baselines)

### Comparison Groups (peer "horizon") — STUDENT level, not school level
Source: NYC DOE "Data Explained: Comparison Groups" (tools.nycenet.edu/resources/comp-group.html);
School Quality Reports Educator Guides (infohub.nyced.org).
- For **each student**, DOE finds the **50 most-similar students citywide**.
- **Step 1 — exact match** on: grade level, ELL category, IEP/SWD category, economic-need status
  (temp housing / public-assistance eligible).
- **Step 2 — nearest** within that exact-match pool on **prior test scores (weighted heavily)** +
  school-composition factors (IEP %, ENI, ELL %, % overage).
- A school's Comparison Group = union of its students' 50-peer sets → a "hypothetical school of
  similar students." A 100-student school → 5,000 comparison students.
- **Takeaway for us:** the authoritative method is (a) student-level, (b) exact-match on need/ELL/SWD,
  and (c) dominated by **prior achievement**. The repo's k-NN uses demographics only, school level,
  and **no prior achievement** — the single biggest divergence to test.

### Economic Need Index (ENI) — the official need score
Source: SQR Educator Guides; CCC-NY "Student Economic Need Index".
- Per student need value: 1.0 if HRA-eligible OR in temporary housing; 1.0 if HS student with home
  language ≠ English and entered DOE within last 4 yrs; else = census-tract child-poverty rate
  (ACS 5-yr) for the student's tract.
- School ENI = average of student need values. Higher = higher need.
- **Peer Index** (UFT explainer): schools ranked by a need composite to define "highest/lowest need."

## Open-standard methods literature

### SES index construction: weighting barely matters
- Jewsbury et al. (NAEP 2017 G8 math): sum-score vs PCA vs clustered-PCA SES indices are **very highly
  correlated**; item *quality* matters more than weighting. (NAGB "Measuring SES", 2023.)
- Broer, Bai & Fonseca (TIMSS): sum-score SES index ≈ first-PC index.
- Vyas & Kumaranayake (Health Policy Plan 2006), "Constructing SES indices: how to use PCA."
- **Implication:** current mean-percentile-rank proclivity is likely already near-optimal among
  *index* methods; PCA unlikely to beat it much. Test empirically.

### Value-added / regression adjustment (gold standard for "expected outcome")
Sources: Leckie & Goldstein; "A comparison of value-added models for school accountability"
(School Effectiveness & School Improvement, 2022); RAND VAM reprints.
- Fit current outcome on **prior achievement (+ student/school demographics)**; school effect = mean
  residual. Prior achievement adjustment is fundamental — unadjusted means mostly reflect intake.
- This is the natural "punching above expectation" estimator: residual = actual − expected.

### Propensity scores are for *balancing*, not pure prediction
- Adding propensity scores to pure prediction models does **not** improve predictive performance
  beyond multivariable adjustment (Wyss et al., AJE 2014). So a propensity-score peer group is
  defensible for *fair comparison* but won't win a *predictiveness* contest vs. direct regression.

### Clustering for typologies
- k-means / GMM (latent profile) / Ward agglomerative are the standard school-typology tools.
- Method choice changes group composition and the strength of group→outcome association
  (peer-group detection sensitivity literature) — so worth comparing head-to-head.

## The 10 approaches tested (final list)

CLUSTER / peer-group family (assign ~40 peers, or a group, per school within grade band):
1. **k-NN demographics (current production)** — Euclidean over 7 demog features; hard grade-band +
   admission bucket; K=40. Baseline.
2. **k-means** on standardized demographics (k chosen so mean group ≈ 40).
3. **Gaussian Mixture Model** (soft latent profiles) on demographics.
4. **Ward agglomerative hierarchical** clustering on demographics.
5. **k-NN + prior achievement (NYCDOE-style)** — demographics + prior-year school proficiency in the
   distance, exact-match-ish on grade band. Tests the DOE's key ingredient.
6. **Need-decile × grade-band strata (similar-schools / coarsened exact matching)** — transparent
   stratification by need-index decile.

DECILE / need-index family (1–10 ranking + expected-outcome):
7. **Mean percentile rank (current proclivity)** — mean pct-rank of econ_dis, ELL, SWD → ntile(10).
8. **PCA first component** SES index → deciles.
9. **ENI-style weighted composite** (official-style: econ need primary + ELL + temp housing if avail) → deciles.
10. **Regression expected-outcome index (value-added style)** — predict each outcome from the full
    demographic vector (intake-only) and from intake + prior-year outcome; predicted = expected-outcome
    decile, residual = punching-above. Gold standard for predictiveness.

## Evaluation (predictive validity), per outcome, per grade band
Outcomes: ELA & Math proficiency (ES/MS/K8), 4-yr grad rate (HS), chronic absenteeism (all).
1. **Variance explained** — R² (continuous indices) / η² between-group (clusters) of outcome.
2. **Within-group homogeneity** — mean within-group outcome SD & demographic SD (clusters; lower better).
3. **Out-of-sample (temporal)** — fit on year T, apply to T+1, measure R²/η². Rewards generalization,
   penalizes overfit.
4. **Residual stability** — corr(residual_T, residual_{T+1}); high = real school signal vs. noise.

Data notes: drop pct_temp_housing (100% null) & college_career_readiness (no rows). econ_dis ~17% null →
grade-band-median impute (flagged). Proficiency on 0–100 scale; econ_dis a 0–1 fraction; z-score per
feature per year. Absenteeism is lower-better (sign-flip when interpreting "good").