NYC School Outcomes — analysis scaffold

A runnable starter for the NYC school-data analysis platform. It implements one shared, deterministic analysis engine that produces two views for any outcome:

• View 1 — Outliers: schools doing better/worse than demographically- and admission-similar peers, in a given year.
• View 2 — Trends: schools improving/declining over time, net of demographic change and immune to scale breaks.

The pilot outcome is 3rd-grade ELA + Math proficiency. Adding an outcome is a config block plus a parser — no engine changes.

Companion design doc: nyc-school-outcomes-spec.md (first-principles spec). This repo is the spec's §8 pilot, built end-to-end.

⚠️ Synthetic vs. real data

NYC/NYSED domains are not reachable from the build/CI sandbox, so this scaffold runs end-to-end on a small synthetic fixture (pipeline/ingest/_fixtures.py) that mimics the real file shapes and writes provenance manifests. The fixture plants known outliers and trends so the engine's output can be checked against ground truth (see tests/test_recovery.py).

To run on real data, replace the fixture step with the real downloaders (pipeline/ingest/test_results_download.py is a template) and rerun. Everything downstream — parse, assemble, analyze, export, verify — is identical.

Quickstart

pip install -r requirements.txt          # numpy, pandas, pyarrow, pyyaml, scipy, pytest
make demo                                 # full pipeline on synthetic data
make test                                 # unit + recovery + golden determinism tests
make analyze OUTCOME=ela_math_g3          # one outcome -> website JSON
make reconcile OUTCOME=ela_math_g3        # correctness check (add --strict in CI)
make help                                 # list targets

make demo chains: fixtures → parse → assemble → analyze → demo-reference → verify → reconcile. Outputs land in data/outputs/website/<outcome>/ and QA reports in docs/qa_reports/.

Repo layout

config/                 outcomes.yaml, covariates.yaml, comparability.yaml  (all behavior is here)
pipeline/
  common.py             paths, config loading, provenance hashing, year normalization
  ingest/               _fixtures.py (synthetic) + *_download.py (real, templated)
  parse/                raw files -> tidy interim (one parser per source)
  assemble/             interim + demographics + directory -> canonical analysis_table
  analysis/
    engine.py           SHARED core: fit_year (WLS), standardize, flag        <-- read this
    outliers.py         View 1 driver (+ persistence)
    trend_runner.py     View 2 driver (regime-aware)
    trends.py           trend math (slope + t-test p-value)
    run_outcome.py      orchestrate one outcome -> processed + website JSON
  export/               to_website_json.py (deterministic JSON the site renders)
  verify/
    checks.py           internal stage gates (coverage, finite z, flag sanity)
    reconcile.py        CORRECTNESS vs source/website/reference                <-- and this
    references/         reference adapters: manual_csv, socrata, cached_file
docs/
  outcome_cards/        per-outcome definition-of-done
  qa_reports/           generated check + reconcile reports
tests/                  engine unit tests, planted-recovery tests, golden snapshots
data/                   generated (gitignored)

Two layers of verification (they answer different questions)

A pipeline can be perfectly deterministic and still wrong. Reconciliation is what catches wrong. It compares, per sampled school-year:

source_raw  ──parse?──>  analysis_table  ──export?──>  website_json  ──truth?──>  reference
            parse fidelity              export fidelity              external truth (headline)

• parse fidelity — analysis value vs an independent re-read of the raw file (catches column/unit/suppression bugs).
• export fidelity — the value on the site vs the analysis value.
• external truth — the site value vs an independent authoritative source.

Tolerances are per-outcome and nonzero (publishers round differently). Under --strict, any external mismatch or internal-fidelity break exits nonzero, so it can gate a data release in CI.

Reference adapters (pipeline/verify/references/)

• manual_csv — values a human transcribes from official per-school pages (NYC School Quality Snapshot, NYSED report card). Highest trust; best for survey outcomes NYSED doesn't publish. This directly implements "compare what we visualize to the NYC official per-school reporting."
• socrata — NYC Open Data JSON API. DBN-native, fully automatable. Republished through a different pipeline than InfoHub, so agreement is a real cross-check.
• cached_file — an independent bulk file (e.g. a NYSED assessment/graduation database export) committed offline. Deterministic + different publisher. Gotcha: NYSED keys schools by 12-digit BEDS code, NYC by DBN — supply a DBN↔BEDS crosswalk (the adapter takes one).

The demo fabricates a manual_csv reference from our own data and injects a known mismatch/suppression/missing value onto flagged cells, so you can watch reconcile catch them (STATUS: FAIL). In real use, a human fills that CSV — never auto-generate it.

Adding a new outcome

1. Add a block to config/outcomes.yaml (value definition, covariate set, thresholds, comparability regime, reconciliation settings).
2. Write a parser in pipeline/parse/ that emits a tidy interim table, and join it in assemble/build_analysis_table.py.
3. make analyze OUTCOME=<id> → make verify → make reconcile.
4. Write an outcome card in docs/outcome_cards/.

No changes to engine.py.

Known limitations / recommended next steps

• Multiple-testing correction for trends: a per-school p < 0.05 gate implies ~5% false positives across all schools. Add Benjamini–Hochberg FDR (or a stricter α) before publishing trend claims.
• Composite sampling SE is approximated as a single proportion on tested N; a two-proportion / design-based SE would be more exact.
• empirical_logit functional form is configurable but raises NotImplementedError until implemented (with a delta-method SE).
• Status vs growth controls, 2022 treatment, and survey comparability regimes are open decisions (see the spec §9 and comparability.yaml).
• The front-end (Next.js static site reading these JSON files) is not in this repo; this is the data/analysis/verification spine it consumes.

Architecture principle (enforced)

Source/research files under data/raw/ are read-only. All transformation flows through documented, reproducible pipeline scripts that generate website JSON. The website never edits source data.