Find What Works: Schools
← All source documentsRaw file
Source document

scripts/analysis/RESEARCH_NOTES.md

Served verbatim from the project repository. Internal working document conventions apply: documents may reference file paths, branch names, and findings-ledger anchors from the repo.

Research notes — methods & citations for the clustering/decile bake-off

Working notes feeding methodology/clustering-decile-bakeoff.md. Not the final report.

What NYC actually does (the "proprietary"/official baselines)

Comparison Groups (peer "horizon") — STUDENT level, not school level

Source: NYC DOE "Data Explained: Comparison Groups" (tools.nycenet.edu/resources/comp-group.html); School Quality Reports Educator Guides (infohub.nyced.org).

  • For each student, DOE finds the 50 most-similar students citywide.
  • Step 1 — exact match on: grade level, ELL category, IEP/SWD category, economic-need status (temp housing / public-assistance eligible).
  • Step 2 — nearest within that exact-match pool on prior test scores (weighted heavily) + school-composition factors (IEP %, ENI, ELL %, % overage).
  • A school's Comparison Group = union of its students' 50-peer sets → a "hypothetical school of similar students." A 100-student school → 5,000 comparison students.
  • Takeaway for us: the authoritative method is (a) student-level, (b) exact-match on need/ELL/SWD, and (c) dominated by prior achievement. The repo's k-NN uses demographics only, school level, and no prior achievement — the single biggest divergence to test.

Economic Need Index (ENI) — the official need score

Source: SQR Educator Guides; CCC-NY "Student Economic Need Index".

  • Per student need value: 1.0 if HRA-eligible OR in temporary housing; 1.0 if HS student with home language ≠ English and entered DOE within last 4 yrs; else = census-tract child-poverty rate (ACS 5-yr) for the student's tract.
  • School ENI = average of student need values. Higher = higher need.
  • Peer Index (UFT explainer): schools ranked by a need composite to define "highest/lowest need."

Open-standard methods literature

SES index construction: weighting barely matters

  • Jewsbury et al. (NAEP 2017 G8 math): sum-score vs PCA vs clustered-PCA SES indices are very highly correlated; item quality matters more than weighting. (NAGB "Measuring SES", 2023.)
  • Broer, Bai & Fonseca (TIMSS): sum-score SES index ≈ first-PC index.
  • Vyas & Kumaranayake (Health Policy Plan 2006), "Constructing SES indices: how to use PCA."
  • Implication: current mean-percentile-rank proclivity is likely already near-optimal among index methods; PCA unlikely to beat it much. Test empirically.

Value-added / regression adjustment (gold standard for "expected outcome")

Sources: Leckie & Goldstein; "A comparison of value-added models for school accountability" (School Effectiveness & School Improvement, 2022); RAND VAM reprints.

  • Fit current outcome on prior achievement (+ student/school demographics); school effect = mean residual. Prior achievement adjustment is fundamental — unadjusted means mostly reflect intake.
  • This is the natural "punching above expectation" estimator: residual = actual − expected.

Propensity scores are for balancing, not pure prediction

  • Adding propensity scores to pure prediction models does not improve predictive performance beyond multivariable adjustment (Wyss et al., AJE 2014). So a propensity-score peer group is defensible for fair comparison but won't win a predictiveness contest vs. direct regression.

Clustering for typologies

  • k-means / GMM (latent profile) / Ward agglomerative are the standard school-typology tools.
  • Method choice changes group composition and the strength of group→outcome association (peer-group detection sensitivity literature) — so worth comparing head-to-head.

The 10 approaches tested (final list)

CLUSTER / peer-group family (assign ~40 peers, or a group, per school within grade band):

  1. k-NN demographics (current production) — Euclidean over 7 demog features; hard grade-band + admission bucket; K=40. Baseline.
  2. k-means on standardized demographics (k chosen so mean group ≈ 40).
  3. Gaussian Mixture Model (soft latent profiles) on demographics.
  4. Ward agglomerative hierarchical clustering on demographics.
  5. k-NN + prior achievement (NYCDOE-style) — demographics + prior-year school proficiency in the distance, exact-match-ish on grade band. Tests the DOE's key ingredient.
  6. Need-decile × grade-band strata (similar-schools / coarsened exact matching) — transparent stratification by need-index decile.

DECILE / need-index family (1–10 ranking + expected-outcome): 7. Mean percentile rank (current proclivity) — mean pct-rank of econ_dis, ELL, SWD → ntile(10). 8. PCA first component SES index → deciles. 9. ENI-style weighted composite (official-style: econ need primary + ELL + temp housing if avail) → deciles. 10. Regression expected-outcome index (value-added style) — predict each outcome from the full demographic vector (intake-only) and from intake + prior-year outcome; predicted = expected-outcome decile, residual = punching-above. Gold standard for predictiveness.

Evaluation (predictive validity), per outcome, per grade band

Outcomes: ELA & Math proficiency (ES/MS/K8), 4-yr grad rate (HS), chronic absenteeism (all).

  1. Variance explained — R² (continuous indices) / η² between-group (clusters) of outcome.
  2. Within-group homogeneity — mean within-group outcome SD & demographic SD (clusters; lower better).
  3. Out-of-sample (temporal) — fit on year T, apply to T+1, measure R²/η². Rewards generalization, penalizes overfit.
  4. Residual stability — corr(residual_T, residual_{T+1}); high = real school signal vs. noise.

Data notes: drop pct_temp_housing (100% null) & college_career_readiness (no rows). econ_dis ~17% null → grade-band-median impute (flagged). Proficiency on 0–100 scale; econ_dis a 0–1 fraction; z-score per feature per year. Absenteeism is lower-better (sign-flip when interpreting "good").