# NYC School Outcomes — Outlier & Trend Analysis Platform ## First-Principles Design Spec & Implementation Plan **Status:** Draft v1 for review and Claude Code handoff **Scope:** A reproducible pipeline + static website that, for each of 8 NYC school outcomes, surfaces (View 1) peer-adjusted positive/negative outliers and (View 2) demographically-adjusted trends over time — using a single shared, deterministic, outcome-configurable analysis engine. > How to read this doc. Sections 1–6 are the first-principles spec (what this should be, independent of any existing code). Section 7 evaluates the existing prototype against that spec and proposes an integrated workflow. Section 8 is the build sequence. Section 9 lists the decisions I need from you before Claude Code starts. Appendices give concrete schemas Claude Code can implement directly. --- ## 1. Product definition (first principles) ### 1.1 The two questions, in plain language Every outcome gets two analysis views, and only these two: - **View 1 — Outliers vs. similar peers.** "Among schools serving demographically similar students under a similar admissions structure, which schools score notably *higher* or *lower* than that profile would predict?" - **View 2 — Trends not explained by demographics.** "Which schools are *improving* or *declining* over time in a way that is **not** explained by changes in their student population?" ### 1.2 What these views are NOT (framing guardrails — load-bearing) This is a civic-transparency tool, and mislabeling a school as a "negative outlier" has real reputational cost. The framing must stay descriptive: - An outlier flag means **"performs above/below what its student composition and admission type would statistically predict."** It is **not** a causal claim that the school is good/bad, and **not** a ranking of school quality. - Demographic variables are **statistical controls to enable fair comparison**, never judgments about students. - The platform shows **data with uncertainty**, not verdicts. Small-N schools, suppressed cells, and noisy single-year blips must be visibly down-weighted, not silently flagged. - Echoing the Civic Intelligence brief: **publish methodology and error/coverage rates openly**, maintain editorial neutrality ("data, not analysis"), and keep a "Not affiliated with NYCDOE/NYSED" disclaimer site-wide. ### 1.3 The 8 outcomes, by grade band | # | Outcome | Band | Native metric (proposed) | Source family | |---|---|---|---|---| | 1 | 3rd-grade ELA + Math | ES | Composite: mean of % Level 3+4 ELA and Math | Test results | | 2 | 7th-grade ELA + Math | MS | Composite: mean of % Level 3+4 ELA and Math | Test results | | 3 | Graduation + College/Career Readiness | HS | 4-yr cohort grad rate; CCR rate (reported separately, optional composite) | Grad results; School Quality | | 4 | Bullying | ES/MS/HS | Survey: % reporting bullying (+ cross-check: NYSED SSEC incident rate) | NYC School Survey; NYSED SSEC | | 5 | Chronic absenteeism | All | % chronically absent (≥10% of days) | Attendance/CA data | | 6 | Teacher/Principal trust | All | Survey: % favorable on trust items | NYC School Survey | | 7 | Teachers would recommend school | All | Survey: % favorable on the recommend item | NYC School Survey | | 8 | Course access | HS | Index of advanced-course availability (see §9, decision 7) | HS Directory (xlsx) | Two outcomes need an explicit operational definition before build: **#8 Course access** (which courses count, and how to index them) and **#3 CCR** (which published CCR metric, and which graduation cohort). See §9. ### 1.4 Unit of analysis - Grain: **school × year**. Primary key: **DBN** (District-Borough-Number) + canonical year string `YYYY-YY`. - A **DBN crosswalk** tracks identity changes (opens/closes/merges/renumbering). Trend analysis runs only on DBNs with enough comparable years (§3.2). --- ## 2. Data model & canonical entities ### 2.1 Covariates (the "controls") — and a gap to close The outcome source list you provided does **not** include a covariate source. We need two: 1. **Sociodemographic covariates** — from the **NYC School Demographic Snapshot** (InfoHub → School Quality → Demographic Data). Fields: % economically disadvantaged / Economic Need Index (ENI), % ELL/MLL, % students with disabilities (SWD), race/ethnicity composition (% Black, Hispanic, Asian, White), % students in temporary housing, total enrollment. *(Note: NYC's poverty proxy is ENI + % economic need, not California-style FRPM; pick one poverty measure as canonical — see §9 decision 9.)* 2. **Admission / structure metadata** — admission method comes from the **HS Directory** (`method1`) and **MS Directory** (`admissionsmethod_prog1`) directory fields; elementary defaults to *Zoned* unless the school is G&T. Charter status and school-type flags (D75 special-ed, D79/transfer) from the school master list. *(Source #9 — the HS Directory xlsx — therefore does double duty: it is both the admission-method source for HS and the source for outcome #8 course access.)* ### 2.2 Covariate sets are partly band-specific A **base sociodemographic set** is shared across all outcomes (ENI/poverty, %ELL, %SWD, race composition, enrollment). On top of that, **structural covariates differ by band**: - ES: G&T flag (no admissions screening otherwise). - MS: admission method (`admissionsmethod_prog1`), where applicable. - HS: admission method (`method1`) — the dominant predictor; buckets like Specialized/exam, Screened, Ed-Opt, Zoned, Unscreened/Lottery, Audition. Define these in `config/covariates.yaml` keyed by band so the engine stays shared (§Appendix B). ### 2.3 Outcome catalog with known comparability breaks | Outcome | Years (typical) | Comparability hazards to handle | |---|---|---| | Tests (g3/g7 ELA+Math) | ~2013–2019, 2022, 2023→ | **2023 NextGen standards reset: proficiency NOT comparable pre/post 2023.** 2020 cancelled; 2021 ~21% participation (unusable). 2022 = old standards, recovery year. | | Graduation | multi-year | Cohort definition (4-yr Aug vs 6-yr); transfer/D79 schools serve over-age/under-credited students — must be separated, not compared to comprehensive HS. | | CCR | 2023-24+ for some indices | Metric availability varies by year; confirm which index. | | Bullying (survey) | survey years | Survey item wording/scale changes across years; verify a stable window. | | Bullying (SSEC incidents) | state years | Incident counts are low-N and reporting-practice-sensitive; use as cross-check, not primary. | | Chronic absenteeism | multi-year | **COVID structural break:** absenteeism spiked citywide 2021-22+; 2020 partial. Pre-2020 vs 2021/22+ are different regimes. | | Trust / Recommend (survey) | survey years | Same survey-instrument-continuity caveat as bullying. | | Course access | directory snapshots | Snapshot, not a time series in the same sense; trend may be N/A or limited. | These hazards drive `config/comparability.yaml` (§3.2, Appendix C). They are the single biggest reason a naive "year-over-year change" view is wrong. --- ## 3. Statistical methodology — the shared engine (the crux) Goal: **one** fitting routine and **one** flagging routine, parameterised per outcome via config. No bespoke per-outcome statistics in v1. ### 3.1 View 1 — peer-adjusted outlier detection (model-residual approach) **Why regression residuals rather than peer-group deciles.** Defining ~40-school peer groups and flagging top/bottom decile (the prototype's v1 approach) is intuitive but: (a) the flag depends on arbitrary group construction (K, distance weights); (b) it yields a rank, not a continuous effect size with uncertainty; and (c) it doesn't down-weight schools whose extreme position is just small-N noise. A regression that conditions on the same demographic + admission variables does the same job *continuously and deterministically*, and the same model serves every outcome by swapping the response variable. **Recommendation: model residual is primary; keep peer-group context as a complementary display + robustness cross-check (§7).** For each `(outcome, year)`: 1. **Eligibility filter.** Drop suppressed cells; drop school types flagged non-comparable for this outcome (config, e.g., D75/D79/transfer for graduation); require `N ≥ min_n` (config). 2. **Fit** weighted least squares of the outcome `y` on the band's covariate set, weights = `N` (tested/respondents/cohort size), within that single year (cross-section): - Admission handling (config): `covariate` (categorical term — default) or `stratify` (separate model per admission bucket when a bucket is large enough). With demographic + admission terms, the regression already encodes "similar peers." - Functional form (config): `identity` (OLS, default) or `empirical_logit` for bounded proportions near 0/1. OLS-on-rate with robust SEs is the transparent default; logit is offered for robustness and is usually immaterial unless rates approach the bounds. - Optional `prior_achievement` covariate (config) converts the residual to a growth-like measure — see §9 decision 3. 3. **Residual + uncertainty.** Raw residual `r_i = y_i − ŷ_i`. Standardize accounting for both model misfit and the school's own sampling error: ``` z_i = r_i / sqrt( s^2 + se_i^2 ) s = model RMSE (residual SD, the unexplained between-school spread) se_i = school's sampling SE (for a proportion: sqrt( p_i (1 - p_i) / N_i )) ``` This is the key fairness move: a small-N school needs a *much* larger raw gap to clear the threshold, so we don't flag noise. 4. **Flag (direction-aware).** Positive outlier if `z_i ≥ z_threshold`, negative if `z_i ≤ −z_threshold` (config, e.g., 2.0). Every outcome carries `desirable_direction` (high/low) so that for chronic absenteeism and bullying — where high = bad — a "positive outlier" still means "doing well." **This single field prevents the most common interpretation bug.** 5. **Stability.** Report per-year flags plus a persistence summary ("flagged in ≥k of last m years") so single-year blips are visibly distinguished from durable outliers. 6. **Outputs per school-year:** `ŷ`, `r`, `z`, percentile-within-peers (for display), `flag`, `N`, suppression flag, pointer to model diagnostics. ### 3.2 View 2 — trend not explained by demographics (residual-trend, reusing §3.1) **Core move:** run the trend on the **demographic-adjusted residual** from §3.1, expressed in **within-year standardized units (`z`)**. This simultaneously (a) nets out demographic change — a rising residual means the school is gaining *relative to demographically similar schools*, not because its intake changed; and (b) sidesteps the 2023 test-scale break and citywide COVID shifts, because `z` measures relative standing each year rather than absolute proficiency/rate. For each `(outcome, school)`: 1. **Build the residual series** `{(year, z_iy)}` across **comparable years only**, per `config/comparability.yaml`. Trends are computed **within a regime**, never across a break: - Tests: Regime A ≈ 2013–2019, **gap 2020–2021 excluded**, Regime B ≈ 2023→. 2022 handled per config. - Chronic absenteeism: pre-2020 vs 2021/22→ regimes. - Survey outcomes: instrument-stable window(s) defined after verification. 2. **Fit trend.** OLS of `z_iy` on `year` within a regime → slope `β` (z-units/yr) with SE; require `min_years_within_regime` (config, e.g., ≥4) and `N ≥ min_n` each year. 3. **Flag improving/declining.** `|β| ≥ slope_threshold` AND statistically distinguishable from 0 (slope-SE t-test or robust criterion) AND not driven by a single endpoint (report a leave-one-year-out sensitivity). Direction-aware via `desirable_direction`. 4. **Outputs per school-outcome:** slope (z/yr; and native-unit slope *within regime* for readability), SE, p, `n_years`, regime label, flag, and the sparkline series. **Documented alternative (v2 rigor option):** a school fixed-effects panel model on the raw outcome with time-varying demographics and a school×time slope. More rigorous, less transparent; the residual-slope default is the shared, explainable v1. Note it in `METHODOLOGY.md`. ### 3.3 Why this is genuinely "shared + configurable + deterministic" - Both views call **the same `fit_year()` routine**; the trend view just iterates it over years and fits a slope on the residuals. - Everything that differs by outcome lives in `config/outcomes.yaml` (response variable, direction, form, covariate set, thresholds, comparability regime, excluded school types). Adding an outcome = adding a config block + a parser, not new statistics. - Determinism: OLS/WLS is closed-form (no seeds needed); any randomized step (e.g., a KNN peer display) is seeded; the analysis stage makes **no network calls** and records input content-hashes in its outputs; golden/snapshot tests assert byte-identical re-runs (§5). ### 3.4 Limitations to publish (overclaim guardrails) State these plainly on the methodology page: - Residuals are **associational, not causal**; omitted variables (e.g., incoming achievement unless modeled, neighborhood, peer effects, selection) can drive them. - Status models without prior-achievement controls partly reward favorable intake even after demographic adjustment; growth-style controls (if enabled) change interpretation. - Survey outcomes carry **response-rate bias** (low-response schools) on top of sampling error — surface response rate alongside survey residuals. - Cross-regime comparisons are reported only in standardized-position terms, never as raw proficiency/rate deltas. ### 3.5 Validation strategy (per outcome, before it ships) - **Face/known-case checks:** a curated list of schools whose standing is independently knowable as a sanity test (not as ground truth). - **Diagnostics:** model R², residual distribution/heteroskedasticity note, leverage/influence check. - **Convergence check:** does the model-residual outlier set overlap sensibly with (i) the peer-group-decile method and (ii) NYCENET comparison-group standing where available? Large divergence = investigate. - **Sensitivity:** vary `z_threshold`, weighting on/off, admission covariate-vs-stratify, and confirm flags are not knife-edge. --- ## 4. Pipeline architecture (deterministic, staged, verifiable) **Governing principle (your standing rule):** *source/research files are read-only; all transformation is via reproducible, documented pipeline scripts that read source files and generate the website JSON; never edit source data from the website side.* The stages below enforce that. ``` Stage 0 Ingest download → /data/raw (read-only) + provenance manifest + checksums Stage 1 Parse raw → /data/interim tidy tables (parquet), one parser per source Stage 2 Assemble interim → /data/processed/analysis_table.parquet (join covariates, apply DBN crosswalk) Stage 3 Analyze analysis_table + config → outlier & trend results (shared engines) Stage 4 Verify (QA) automated checks + per-outcome human sign-off (outcome cards) Stage 5 Export results → /data/outputs/website/*.json (versioned) Stage 6 Site Next.js static site reads the JSON (no computation in the front-end) ``` Each stage is an independently runnable CLI step (`make ingest`, `make analyze OUTCOME=…`, etc.), emits a log + a small report artifact, and has explicit **acceptance criteria** (§5.2) that gate the next stage. ### 4.1 Determinism & reproducibility requirements - Pin all dependencies with a lockfile (`uv.lock`/`requirements.txt` with hashes); record tool versions in each run's report. - Analysis stage: **no network**, fixed seeds for any stochastic display step, input content-hashes written into outputs. - **Golden tests:** committed expected outputs for the pilot outcome; CI fails on any drift. Re-running Stage 3–5 from the same raw inputs must reproduce identical JSON. - Provenance: every raw file carries `{source_url, retrieved_at, http_status, sha256, bytes, notes}` in `/data/raw///provenance.json`. ### 4.2 Repo layout ``` /data /raw///... # read-only; provenance.json + .sha256 /interim//*.parquet # tidy parsed tables /processed/analysis_table.parquet # canonical school-year analysis dataset /outputs/website//*.json # exported, versioned site data /pipeline /ingest/_download.py # one per source → raw + manifest /parse/_parse.py # raw → interim /assemble/build_analysis_table.py # join covariates + crosswalk → processed /analysis/engine.py # SHARED fit_year(), standardize(), flag() /analysis/outliers.py # View 1 orchestration /analysis/trends.py # View 2 orchestration /analysis/run_outcome.py # one outcome end-to-end /verify/checks.py # automated stage checks /export/to_website_json.py /config outcomes.yaml covariates.yaml comparability.yaml thresholds.yaml /crosswalk/dbn_crosswalk.csv /docs METHODOLOGY.md DATA_DICTIONARY.md PROVENANCE.md /outcome_cards/.md # filled per outcome (definition of done) /qa_reports/__.md /site # Next.js static (or evolve the prototype) /tests # golden/snapshot + unit Makefile (or justfile) ``` --- ## 5. Verification & documentation system "Every step verifiable; methodology fully documented for replication" is the headline requirement — so verification is a first-class subsystem, not an afterthought. ### 5.1 Artifacts that must exist - `PROVENANCE.md` + per-file manifests (URL, date, checksum) — anyone can confirm what was downloaded and when. - `METHODOLOGY.md` — human-readable version of §3, plus the published limitations (§3.4). - `DATA_DICTIONARY.md` — every field in `analysis_table` and every JSON field the site consumes. - **Outcome cards** (`/docs/outcome_cards/.md`) — per outcome: exact response definition, covariate set, admission handling, functional form, thresholds, comparability regime, excluded school types, data vintage, diagnostics summary, known-case check results, sign-off. **An outcome is not "done" until its card is filled and checks pass.** - **QA reports** — auto-generated per run: coverage matrix (schools × years), % suppressed, join-match rate (outcome rows matched to a demographic row), distribution snapshots, model R², flag counts. - `CHANGELOG.md` + data versioning so a published figure can always be traced to a specific data vintage + code commit. ### 5.2 Stage acceptance criteria (gates) - **Stage 0:** all expected source files present; checksums recorded; manifest complete. - **Stage 1:** schema validates; row counts within expected band; no duplicate primary keys; year strings normalized. - **Stage 2:** join-match rate ≥ threshold (and unmatched rows enumerated, not dropped silently); crosswalk applied; coverage matrix produced. - **Stage 3:** model converges; diagnostics within sane ranges; flag counts plausible; standardized residuals finite. - **Stage 4:** automated checks green **and** human sign-off on the outcome card. - **Stage 5:** JSON validates against the front-end schema; golden test passes. --- ## 6. Website spec (front-end) The front-end **only renders pre-computed JSON** — no analysis in the browser (keeps it deterministic and the methodology auditable). ### 6.1 Information architecture ``` Home → Outcome index → Outcome page (tabs: Outliers | Trends) → School profile → Methodology / Transparency → Data downloads ``` ### 6.2 Outlier view (per outcome) - Scatter and/or table of schools; flagged positive/negative outliers highlighted, **colored by `desirable_direction`** (good = one color regardless of metric polarity). - For each flagged school: residual in **native units + standardized `z`**, peer context (peer distribution 10/50/90 + drill-down to the peer set), `N`, suppression badge, **"persistent" badge** for multi-year flags. - **Uncertainty visible:** confidence band on the residual or an explicit "low-N — interpret with caution" flag for schools near the `min_n` floor. - Filters: grade band, admission bucket, borough; opt-in toggles for suppressed rows and D75/D79/transfer. ### 6.3 Trend view (per outcome) - Per-school sparkline of the **demographic-adjusted standardized residual** over time; slope + significance; **regime boundaries drawn as vertical markers** (e.g., the 2020–2022 test break, the COVID absenteeism break) so users never read a spurious jump across a break. - Improving/declining flags (direction-aware); secondary toggle to view the raw outcome with a demographic overlay for context. ### 6.4 School profile One school across all 8 outcomes: each outcome's residual + trend mini-view, the peer set used, the school's demographic snapshot, and the relevant caveats inline. ### 6.5 Transparency, downloads, tech - Methodology pages (human-readable + linked machine-readable config + outcome cards + the published error/coverage report). - **Bulk + per-outcome downloads** (CSV/Parquet) — serves the "researchers" use case and the Civic Intelligence "infrastructure others build on" goal. - Tech: **Next.js static export reading JSON** (matches the prototype and your prior static-site pattern). Charts via the prototype's existing library (Recharts) for consistency. Accessibility and neutral presentation throughout. - Optional (borrowed from the adjacent project, see §7): expose a small read-only **API/MCP endpoint** over the same JSON for programmatic access. --- ## 7. Evaluation of the existing prototype + integrated workflow Based on the live methodology page at `nyc-edu-data.vercel.app/methodology` (the repo `mwg1378/nyc-edu-data` is private — a **code-level audit is a defined step once you have access**, §8 Phase 0). For contrast, the adjacent public project `jkinberg/nyc-school-explorer` is a *conversational* AI tool (chat-over-data with guardrails, LLM-as-judge scoring, an MCP endpoint, 328 tests) — a different paradigm, but a good source of patterns for verification, responsible framing, and programmatic access. ### 7.1 What the prototype already does well — keep it - **Peer-context as the core lens** ("vs. peers, not citywide averages") — exactly View 1's intent. - Hard filters (grade band, admission bucket) + demographic similarity — the same covariate spirit as §3. - **Proclivity decile** = a working version of the "punching above its profile" narrative. - **Suppression handling** (low-N shown, not silently dropped; hidden from outlier lists by default with a toggle) — matches §3.1/§6.2. - **School-universe tagging** (charters in; D75/D79/alt tagged out of defaults, opt-in) — matches §6.2. - **Survey rollups** (DOE-published + own from raw distributions) and **Story ideas** (great for the "discoverability" gap the Civic brief flags). - A machine-readable `SPECS.md`, year normalization (`YYYY-YY`), and a sources list that already includes the **Demographic Snapshot** and **NYCENET comparison groups**. ### 7.2 Gaps vs. this spec — enhance 1. **Outlier statistic is rank-based, not model-based.** The prototype flags top/bottom **decile of the peer-group distribution** and explicitly defers proportion/low-count handling to "v2." It has no reliability weighting, so small-N schools can land in extreme deciles on noise. → Add the **§3.1 regression-residual engine with reliability-weighted standardized residuals**; keep the peer-decile as a complementary display and a cross-validation check (§3.5). 2. **"Expected outcome" uses only 3 covariates.** Proclivity decile = mean percentile rank across % econ-disadvantaged, ELL, SWD — ignoring race/ethnicity composition, size, and admission beyond the hard filter, and it's a separate construct from the outlier computation. → **Unify:** the regression's covariate set subsumes proclivity; "expected" becomes the model's continuous prediction. 3. **Trend view is the biggest gap.** "Year-over-year movers" appear to be raw movers; the methodology page describes no demographic-adjustment-over-time and no comparability-regime handling — so it does not yet satisfy View 2, and it is exposed to the **2023 NextGen test break** and the **COVID absenteeism break**. → Add the **§3.2 residual-trend engine + `comparability.yaml`**, expressed in within-year standardized units. 4. **Determinism/verification unverified.** KNN (K=40) + `DERIVED` admission/tags is reasonable but introduces tuning sensitivity and a derived-source dependency; I can't yet see golden tests, provenance manifests, per-outcome verification cards, or a published error/coverage report. → Add the **§5 verification scaffolding** and reconcile `DERIVED` admission/tags against NYCENET comp-groups and the directory `method1`/`admissionsmethod_prog1` fields they already cite. 5. **Composites should be built on standardized residuals**, not raw rates, for cross-metric comparability and direction-awareness. ### 7.3 Integrated workflow (evolve, don't rebuild) 1. **Keep** the prototype's Next.js front-end, IA (Ratings / Story ideas / Methodology), peer-context UX, suppression + universe tagging, and `SPECS.md` discipline. 2. **Insert a Python analysis layer** (the §3 shared engines + `/config`) as the single source of truth that **emits the site's JSON**, augmenting/replacing the current rating computation. The front-end gains new fields (`residual`, `z`, `slope`, `regime`, flags) alongside the existing peer-decile fields, so the two methods can be shown side by side during transition. 3. **Add** `comparability.yaml` + regime-aware trends, and the verification scaffolding wired into CI. 4. **Reconcile** admission/tags with NYCENET + directory fields. 5. **Sequence:** run **one pilot outcome** through the new layer, validate against the prototype's existing ratings for sanity (§3.5 convergence check), then migrate the rest. --- ## 8. Build sequence / milestones **Phase 0 — Foundation (after repo access).** Code-level audit of the private prototype; stand up `/config` schema, repo layout, provenance manifests, and CI with golden tests; lock the §9 decisions. **Phase 1 — Vertical slice (one outcome, end to end).** Take a single pilot outcome through Stages 0→6, proving the whole spine before scaling. **Recommended pilot: 3rd-grade ELA + Math (ES)** — it is the flagship academic metric, has clean structured InfoHub files, and forces us to nail the two hardest features early: the **composite** and the **2023 comparability regime**. (Alternative: Chronic absenteeism, which instead exercises the COVID break and a single non-composite rate — a good *second* outcome.) **Phase 2 — Validate the pilot.** Diagnostics, known-case checks, convergence vs. the prototype + NYCENET, fill the outcome card, publish the methodology page + first error/coverage report. **Phase 3 — Templatize and roll out the rest, one at a time** (each gated by its outcome card + checks), grouped by source affinity to share parsers: - Tests g7 (reuses the g3 loader/engine). - The three **survey** outcomes — bullying, teacher/principal trust, would-recommend — share one NYC School Survey loader (verify instrument-stable windows here). - HS: graduation + CCR (shared HS sources; resolve cohort + CCR-metric decisions). - Chronic absenteeism (COVID regime). - **Course access (HS) last** — needs the operational definition (§9 decision 7) and non-trivial HS Directory parsing; may have limited/No trend. - Bullying SSEC incident data added as a cross-check layer. **Phase 4 — Site polish + public beta.** School profiles, bulk downloads, optional API/MCP, full transparency pages, public error/coverage report. --- ## 9. Decisions I need from you (before Claude Code starts) 1. **Outlier statistic:** model residual primary + peer-decile as cross-check (my recommendation) — confirm? 2. **Admission handling:** covariate (default) vs. stratify for large screened/specialized HS sets? 3. **Prior-achievement controls:** status-only for v1 (keeps methodology shared across outcomes; recommended) vs. add growth-style controls where available (grad/CCR/g7)? (Changes interpretation — would be flagged as such.) 4. **Thresholds:** `z_threshold` (default 2.0?), persistence window (e.g., ≥2 of last 3 years?), and `min_n` per outcome family. 5. **Comparability regimes:** confirm tests = {2013–2019} / exclude {2020,2021} / {2023→}, and how to treat 2022. Survey instrument-stable windows to be set during verification — OK? 6. **School universe:** include charters (prototype does)? Keep D75/D79/transfer tagged-out of defaults with opt-in (recommended)? 7. **Course access (HS) definition:** propose an **advanced-course-availability index** from the HS Directory — components e.g. {# AP courses, calculus offered, advanced science, # world languages, dual-enrollment available}. Confirm/adjust the components and whether it's an index or a set of separate availability flags. 8. **CCR metric + graduation cohort:** which published CCR index, and 4-year (August) vs 6-year grad rate as primary? 9. **Covariate source + poverty measure:** confirm Demographic Snapshot as canonical; choose ENI vs. % economically disadvantaged (+ % temporary housing) as the poverty control. 10. **Integration target:** evolve `mwg1378/nyc-edu-data` (recommended) vs. a fresh repo? License for the public-data outputs? 11. **Site tech:** stay Next.js static + JSON (recommended)? 12. **Programmatic access:** add a read-only API/MCP export over the JSON (borrowing the adjacent project's pattern)? --- ## Appendix A — `config/outcomes.yaml` (example block) ```yaml - id: ela_math_g3 label: "3rd Grade ELA + Math Proficiency" grade_band: ES unit: school_year sources: [test_results] value: type: composite # mean of components components: [g3_ela_pct_l34, g3_math_pct_l34] denominator: g3_tested_n desirable_direction: high # high = good (use "low" for absenteeism/bullying) functional_form: identity # or empirical_logit covariates: es_base # -> covariates.yaml admission_handling: covariate # covariate | stratify | none prior_achievement: null min_n: 15 outlier: z_threshold: 2.0 persistence: { window: 3, min_flags: 2 } comparability: ela_math_regimes # -> comparability.yaml trend: min_years_within_regime: 4 slope_threshold_z_per_yr: 0.15 non_comparable_school_types: [D75, D79, transfer] notes: "2023 NextGen standards reset; trend within-regime only." ``` ## Appendix B — `config/covariates.yaml` (example) ```yaml base_demographic: &base - pct_econ_disadvantaged # or eni (pick one canonical, see decision 9) - pct_ell - pct_swd - pct_black - pct_hispanic - pct_asian - enrollment_total # size es_base: covariates: [*base, gt_flag] ms_base: covariates: [*base] admission_field: admissionsmethod_prog1 hs_base: covariates: [*base] admission_field: method1 ``` ## Appendix C — `config/comparability.yaml` (example) ```yaml ela_math_regimes: regimes: - { id: A_commoncore, years: [2013, 2014, 2015, 2016, 2017, 2018, 2019] } - { id: B_nextgen, years: [2023, 2024, 2025] } excluded_years: [2020, 2021] # 2020 cancelled; 2021 ~21% participation ambiguous_years: [2022] # old standards, recovery year — handle per policy cross_regime_reporting: standardized_only # never raw proficiency deltas across regimes chronic_absenteeism_regimes: regimes: - { id: pre_covid, years: [2016, 2017, 2018, 2019] } - { id: post_covid, years: [2022, 2023, 2024, 2025] } excluded_years: [2020, 2021] ``` ## Appendix D — Outcome card template (`/docs/outcome_cards/.md`) ```markdown # Outcome card: - Response definition: - Covariate set: | Admission handling: - Functional form: | Prior achievement: - Thresholds: z=<>, persistence=<>, min_n=<> - Comparability regime: - Excluded school types: - Data vintage: - Diagnostics: R²=<>, residual notes=<>, flag counts (pos/neg)=<> - Known-case checks: - Convergence vs peer-decile / NYCENET: - Sensitivity: - Sign-off: ```