# NYC School Outcomes — analysis scaffold

A runnable starter for the NYC school-data analysis platform. It implements **one
shared, deterministic analysis engine** that produces two views for any outcome:

- **View 1 — Outliers:** schools doing better/worse than demographically- and
  admission-similar peers, in a given year.
- **View 2 — Trends:** schools improving/declining over time, net of demographic
  change and immune to scale breaks.

The pilot outcome is **3rd-grade ELA + Math proficiency**. Adding an outcome is a
config block plus a parser — no engine changes.

> Companion design doc: `nyc-school-outcomes-spec.md` (first-principles spec).
> This repo is the spec's §8 pilot, built end-to-end.

---

## ⚠️ Synthetic vs. real data

NYC/NYSED domains are not reachable from the build/CI sandbox, so this scaffold runs
end-to-end on a **small synthetic fixture** (`pipeline/ingest/_fixtures.py`) that
mimics the real file shapes and writes provenance manifests. The fixture **plants
known outliers and trends** so the engine's output can be checked against ground
truth (see `tests/test_recovery.py`).

To run on **real data**, replace the fixture step with the real downloaders
(`pipeline/ingest/test_results_download.py` is a template) and rerun. Everything
downstream — parse, assemble, analyze, export, verify — is identical.

---

## Quickstart

```bash
pip install -r requirements.txt          # numpy, pandas, pyarrow, pyyaml, scipy, pytest
make demo                                 # full pipeline on synthetic data
make test                                 # unit + recovery + golden determinism tests
make analyze OUTCOME=ela_math_g3          # one outcome -> website JSON
make reconcile OUTCOME=ela_math_g3        # correctness check (add --strict in CI)
make help                                 # list targets
```

`make demo` chains: `fixtures → parse → assemble → analyze → demo-reference →
verify → reconcile`. Outputs land in `data/outputs/website/<outcome>/` and QA
reports in `docs/qa_reports/`.

---

## Repo layout

```
config/                 outcomes.yaml, covariates.yaml, comparability.yaml  (all behavior is here)
pipeline/
  common.py             paths, config loading, provenance hashing, year normalization
  ingest/               _fixtures.py (synthetic) + *_download.py (real, templated)
  parse/                raw files -> tidy interim (one parser per source)
  assemble/             interim + demographics + directory -> canonical analysis_table
  analysis/
    engine.py           SHARED core: fit_year (WLS), standardize, flag        <-- read this
    outliers.py         View 1 driver (+ persistence)
    trend_runner.py     View 2 driver (regime-aware)
    trends.py           trend math (slope + t-test p-value)
    run_outcome.py      orchestrate one outcome -> processed + website JSON
  export/               to_website_json.py (deterministic JSON the site renders)
  verify/
    checks.py           internal stage gates (coverage, finite z, flag sanity)
    reconcile.py        CORRECTNESS vs source/website/reference                <-- and this
    references/         reference adapters: manual_csv, socrata, cached_file
docs/
  outcome_cards/        per-outcome definition-of-done
  qa_reports/           generated check + reconcile reports
tests/                  engine unit tests, planted-recovery tests, golden snapshots
data/                   generated (gitignored)
```

---

## Two layers of verification (they answer different questions)

| Layer | Question | Mechanism |
|---|---|---|
| **Golden / snapshot** (`tests/test_golden.py`) | "Does the pipeline reproduce *itself* exactly?" → determinism | Re-run, byte-compare exported JSON to committed goldens. Regenerate with `make golden`. |
| **Reconciliation** (`pipeline/verify/reconcile.py`) | "Do the numbers we publish match *reality*?" → correctness | For a sample (always incl. every flagged outlier), compare across 4 layers and against an independent source. |

A pipeline can be perfectly deterministic and still wrong. Reconciliation is what
catches *wrong*. It compares, per sampled school-year:

```
source_raw  ──parse?──>  analysis_table  ──export?──>  website_json  ──truth?──>  reference
            parse fidelity              export fidelity              external truth (headline)
```

- **parse fidelity** — analysis value vs an independent re-read of the raw file
  (catches column/unit/suppression bugs).
- **export fidelity** — the value on the site vs the analysis value.
- **external truth** — the site value vs an **independent authoritative source**.

Tolerances are per-outcome and nonzero (publishers round differently). Under
`--strict`, any external mismatch or internal-fidelity break exits nonzero, so it can
gate a data release in CI.

### Reference adapters (`pipeline/verify/references/`)
- **`manual_csv`** — values a human transcribes from official **per-school pages**
  (NYC School Quality Snapshot, NYSED report card). Highest trust; best for survey
  outcomes NYSED doesn't publish. This directly implements "compare what we visualize
  to the NYC official per-school reporting."
- **`socrata`** — NYC Open Data JSON API. DBN-native, fully automatable. Republished
  through a different pipeline than InfoHub, so agreement is a real cross-check.
- **`cached_file`** — an independent bulk file (e.g. a NYSED assessment/graduation
  database export) committed offline. Deterministic + different publisher.
  **Gotcha:** NYSED keys schools by 12-digit **BEDS** code, NYC by **DBN** — supply a
  DBN↔BEDS crosswalk (the adapter takes one).

> The demo fabricates a `manual_csv` reference from our own data and injects a known
> mismatch/suppression/missing value onto flagged cells, so you can watch reconcile
> catch them (`STATUS: FAIL`). In real use, a human fills that CSV — never auto-generate it.

---

## Adding a new outcome
1. Add a block to `config/outcomes.yaml` (value definition, covariate set, thresholds,
   comparability regime, reconciliation settings).
2. Write a parser in `pipeline/parse/` that emits a tidy interim table, and join it in
   `assemble/build_analysis_table.py`.
3. `make analyze OUTCOME=<id>` → `make verify` → `make reconcile`.
4. Write an outcome card in `docs/outcome_cards/`.

No changes to `engine.py`.

---

## Known limitations / recommended next steps
- **Multiple-testing correction** for trends: a per-school p < 0.05 gate implies ~5%
  false positives across all schools. Add Benjamini–Hochberg FDR (or a stricter α)
  before publishing trend claims.
- **Composite sampling SE** is approximated as a single proportion on tested N; a
  two-proportion / design-based SE would be more exact.
- **`empirical_logit`** functional form is configurable but raises `NotImplementedError`
  until implemented (with a delta-method SE).
- **Status vs growth controls**, **2022 treatment**, and **survey comparability
  regimes** are open decisions (see the spec §9 and `comparability.yaml`).
- The front-end (Next.js static site reading these JSON files) is **not** in this repo;
  this is the data/analysis/verification spine it consumes.

## Architecture principle (enforced)
Source/research files under `data/raw/` are **read-only**. All transformation flows
through documented, reproducible pipeline scripts that generate website JSON. The
website never edits source data.