# 02 — Gap Analysis: Prototype vs. Design Spec

> Each dimension is marked **Aligned /
> Diverges / Missing** with a one-line "why it matters" and `file:line` citations.
> Spec = `nyc-school-outcomes-spec.md`. Scaffold (reference Python
> implementation) = `reference-scaffold/`. Prototype = this repository (the repo root).

**Legend:** 🟢 Aligned · 🟡 Diverges · 🔴 Missing

| # | Dimension | Verdict |
|---|---|---|
| 1 | Peer groups — match vs. adjust | 🟡 Diverges (matches, doesn't regress out) |
| 2 | Outlier definition — decile vs. residual-z | 🟡 Diverges (core gap) |
| 3 | Proclivity decile — control vs. display | 🟡 Diverges (display/filter only) |
| 4 | Trends / time — regime-aware z-slope | 🔴 Missing (raw deltas) |
| 5 | Comparability years (2020/21/22, NextGen) | 🔴 Missing in code (editorial only) |
| 6 | Suppression / small-N | 🟢 Aligned (suppression) / 🟡 (no reliability weighting) |
| 7 | Survey rollups — cross-year continuity | 🟡 Diverges (no continuity check) |
| 8 | Provenance & determinism | 🟡 Diverges (provenance partial; no golden/checksum) |
| 9 | **Correctness vs reality** (official per-school reconcile) | 🔴 **Missing — the largest gap** |

---

## 1. Peer groups — **🟡 Diverges**

*Why it matters:* matching on demographics and regressing them out answer subtly
different questions; the spec wants the outcome *adjusted*, not just the comparison
set *constrained*.

- Prototype: KNN(K=40) with hard filters (grade band + admission bucket) + weighted Euclidean over 7 demographic shares (1.0) + topic Jaccard (0.5). `scripts/loaders/nycenet-comparison-groups.ts:24-95`.
- The peer group only **constrains who you're ranked against**; the outcome value itself is never adjusted for covariates — the percentile is a raw midrank within the matched set. `scripts/loaders/compute-percentiles.ts:141-156`.
- **Key question (answered):** the prototype **matches on** covariates; it does **not regress them out**. The spec's method fits `outcome ~ covariates (+ admission)` per year and standardizes the residual — covariates enter the *model*, not a *filter*. Spec §3.1 (`nyc-school-outcomes-spec.md:91-113`); scaffold `pipeline/analysis/engine.py:36-53` (`fit_year`).
- Consequence: peer-group construction sensitivity (K, distance weights, the ×30 Jaccard scale at `nycenet-comparison-groups.ts:88`) directly determines flags; the regression makes that a continuous, weight-free prediction. Spec §3.1 (`nyc-school-outcomes-spec.md:93`) and the team's own note (`docs/agents/quantitative-knowledge.md:38-47`).
- *Verdict nuance:* the spec explicitly says **keep** peer groups as complementary display + a convergence cross-check (`nyc-school-outcomes-spec.md:93,150,287`). So this is "diverges in role," not "throw away."

## 2. Outlier definition — **🟡 Diverges (the core methodological gap)**

*Why it matters:* a fixed-10%-per-tail rule flags schools whether or not the gap is
meaningful or just small-N noise, and produces a rank rather than an effect size
with uncertainty.

- Prototype: top/bottom **decile of the peer-group percentile** — `comparison_group_percentile ≥ 0.9` / `≤ 0.1`. `lib/queries/schools.ts:166-172`. The compute step has **no reliability weighting**; a 30-tested-student school and a 300-tested-student school are ranked identically (`compute-percentiles.ts:141-156`). Min peer count is the only guard (≥5, :148).
- Prototype's own spec defers proportion/low-count handling to "v2": `SPECS.md:300`, `methodology/README.md:7`.
- Spec: **covariate-adjusted standardized residual** `z = r / √(s² + se_i²)`, flag `|z| ≥ z_threshold` (default 2.0), direction-aware. `nyc-school-outcomes-spec.md:102-111`; scaffold `engine.py:72-95` (`standardize`, `flag_outliers`) + `outliers.py:40-56`.
- **Practical difference:**
  - *Decile* always flags ~10% of each tail per metric, regardless of whether the spread is real → over-flags in tight peer groups, under-flags in dispersed ones; flags noisy small-N schools at the extremes (exactly what `quantitative-knowledge.md:101-109` warns about — "0% on n=3").
  - *Residual-z* flags **only** schools whose gap is large **relative to its own uncertainty**: `se_i = √(p(1−p)/N)` means a small-N school needs a much larger raw gap to clear `|z|≥2` (`engine.py:56-79`). It yields a continuous effect size, a fitted "expected" value, and a count that can be zero or many — not a forced 10%.
- *Decision is deliberately deferred* (per KX decisions in the runbook): run residual-z **alongside** decile, diff which schools flip, decide replace-vs-supplement after. This is sequenced first in `03_integration_plan.md`.

## 3. Proclivity decile — **🟡 Diverges (display/control hybrid, not a statistical control)**

*Why it matters:* if "expected outcome" is a separate decile badge rather than the
model's prediction, the outlier flag and the "punching above its profile" narrative
are computed from **different** notions of "expected."

- Prototype: `proclivity_decile` = `ntile(10)` of the mean percentile rank across **3** challenge features (`pctEconDis`, `pctEll`, `pctSwd`). `scripts/loaders/compute-proclivity.ts:17,92-115`.
- It is **not** a statistical control on the outcome: it is stored on the school and used only as a **filter/sort** in the outlier query (`lib/queries/schools.ts:176-181,195-196`) and a display badge (`components/proclivity-badge.tsx`). The percentile/outlier computation never references it.
- Reconcile with spec covariates: the spec's covariate set is **broader** — base demographic set = econ-disadvantaged/ENI, ELL, SWD, **race composition (Black/Hispanic/Asian/White), enrollment size**, plus **band-specific admission** (G&T for ES, `admissionsmethod_prog1` for MS, `method1` for HS). `nyc-school-outcomes-spec.md:54-68,373-393` (`covariates.yaml`). The spec explicitly says: **unify** — the regression's covariate set subsumes proclivity; "expected" becomes the model's continuous prediction (`nyc-school-outcomes-spec.md:288`). So proclivity's 3 features are a strict subset of the model covariates.
- *Verdict:* keep proclivity as a display grouping (it's a good plain-language hook — `nyc-school-outcomes-spec.md:279`), but it is not the spec's control and shouldn't be conflated with it.

## 4. Trends / time — **🔴 Missing**

*Why it matters:* a raw year-over-year delta on proficiency is exactly the view the
spec calls "the single biggest reason a naive year-over-year change is wrong."

- Prototype: `getYearOverYearMovers` = raw `to_value − from_value` between two chosen years. `lib/queries/schools.ts:272-318`; rendered as a raw delta at `app/ratings/movers/page.tsx:45-51,110-113`.
- There is **no time-dimension analysis** beyond this: no within-year z, no slope, no regime, no persistence. The trend is on **absolute proficiency**, so it crosses scale breaks freely.
- Spec View 2: slope of the **within-year standardized residual (z)** fit **within a comparability regime**, with a t-test gate and leave-one-year-out sensitivity. `nyc-school-outcomes-spec.md:115-129`; scaffold `pipeline/analysis/trends.py:18-60` (`fit_trend`, `flag_trend`) + `trend_runner.py`.
- The 2023 NextGen break: because the prototype trends on raw proficiency, a school can show a large "+delta" that is entirely the 2023→ standards reset or the 2024-25 cutscore move. The prototype's **stories** explicitly know this (`data/stories/answers.ts:1000,1920,2128`), but the **movers query/UI does not enforce it** — the awareness is editorial, not systematic.

## 5. Comparability years — **🔴 Missing in code (handled editorially only)**

*Why it matters:* mixing pre/post-NextGen test years or pre/post-COVID absenteeism
years compares non-comparable regimes.

- Prototype loaders ingest **all** years 2018–2025 with **no comparability flagging**: `scripts/loaders/test-results.ts:12-23` (fixed 2018–2025 files), no regime/exclusion logic anywhere in `lib/` or `scripts/` (grep: regime/comparability hits only `docs/` + `data/stories/`).
- 2020/2021 are **not excluded** in code; 2022 is **not** treated as ambiguous; the movers query will happily diff 2019→2024 across the NextGen break.
- The data needed to *detect* the break exists — `Mean Scale Score` is ingested alongside proficiency (`test-results.ts:167-182`) — but is not used for any comparability gate.
- Spec: `config/comparability.yaml` with explicit regimes, `excluded_years: [2020,2021]`, `ambiguous_years: [2022]`, `cross_regime_reporting: standardized_only`. `nyc-school-outcomes-spec.md:70-83,395-411`; scaffold `config/comparability.yaml:5-26`.
- The editorial layer *does* the right thing manually (e.g. `data/stories/answers.ts:1770` filters movers to schools with both endpoints comparable), confirming the team agrees with the spec — it just isn't encoded.

## 6. Suppression / small-N — **🟢 Aligned (suppression) · 🟡 (no reliability weighting)**

*Why it matters:* showing suppression builds trust; but suppression ≠ down-weighting
noisy-but-published small-N cells.

- Suppression handling is good and matches the spec: low-N cells detected (`s`/`*`/`n<5`) at `_fetch.ts:111-115`, stored as `suppressed=true`, **hidden from outlier lists by default with an opt-in toggle** (`lib/queries/schools.ts:207-214`, `includeSuppressed`), shown not dropped per `SPECS.md:301,264`. Aligns with `nyc-school-outcomes-spec.md:26,250-251`.
- **But** published-yet-small-N cells are **not down-weighted** in the outlier computation — only a min-peer-count guard exists (`compute-percentiles.ts:148`). The spec's `se_i` term and `min_n` eligibility floor (`outliers.py:46`, `outcomes.yaml:23`) are the missing reliability mechanism. This is the same root as gap #2.

## 7. Survey rollups — **🟡 Diverges**

*Why it matters:* if item wording/scale shifts across years, a cross-year survey
trend (or even a level comparison) is measuring the instrument, not the school.

- Prototype rolls up surveys by **regex-matching question text independently within each year** (questions are year-versioned) and averaging `positive_pct`. `scripts/loaders/compute-survey-metrics.ts:25-99,113-128`. It stores both NYCDOE-published and derived domain scores per `SPECS.md:165-173,302` — that transparency design is **aligned**.
- **But** there is **no verification that a matched item's wording/scale is continuous across years** before the values are compared over time. The keyword lists are static; a re-worded item in a new year either silently drops out or silently joins the average.
- Spec requires an explicit instrument-stable window, set **only after** verifying continuity (`nyc-school-outcomes-spec.md:77,80,124,316`); scaffold leaves `survey_regimes` as an intentional placeholder pending that verification (`config/comparability.yaml:27-35`).

## 8. Provenance & determinism — **🟡 Diverges**

*Why it matters:* "every figure traceable to a data vintage + code commit" and
"byte-identical re-runs" are headline requirements; partial provenance can't gate a
release.

- **Provenance — partial-aligned.** `DataLoad` records `source_name`, `source_url`, `script_path`, **`git_sha`**, row counts, notes per run (`_lib.ts:30-62`, `prisma/schema.prisma:399-419`). Good foundation.
- **Gaps vs spec §4.1/§5.1 (`nyc-school-outcomes-spec.md:171-176,214-221`):**
  - No **source-file content hash** (`sha256`/`bytes`/`retrieved_at`/`http_status`) — cache is keyed by URL only (`_fetch.ts:9-29`); spec wants a per-file `provenance.json` manifest.
  - No **golden/determinism tests** — `tests/unit/` is only `format`/`normalize`/`routes`; nothing asserts a re-run reproduces identical outputs. Scaffold has `make golden` + `tests/test_golden.py` + committed fixtures (`Makefile:43-51`, `tests/fixtures/golden/ela_math_g3/`).
  - Determinism is also harder to guarantee because the pipeline mutates a **live DB** in place (e.g. `compute-proclivity.ts:124` resets all deciles each run; `nycenet-comparison-groups.ts:211-217` deletes+recreates all groups) rather than producing a versioned, hashable artifact. Spec wants staged immutable artifacts → versioned JSON (`nyc-school-outcomes-spec.md:159-167`) with `analysis_table_sha256` in every output (`to_website_json.py:54-73`).

## 9. Correctness vs reality — **🔴 Missing — the single largest gap**

*Why it matters:* nothing currently confirms that a number the site shows for a
specific school matches what NYC/NYSED officially report for that school. For a
civic-transparency tool that may flag a named school as a "negative outlier," this
is the highest-stakes check.

- **No reconciliation / official-data correctness check exists** anywhere in the prototype. A full-repo grep for `reconcile|verify|official|residual|ground.?truth|golden` finds matches **only in editorial docs** (`docs/agents/*`, `docs/editorial-process.md`) describing the *story-writing* review process — **not** a data-verification harness. There are no reference adapters, no per-school cross-check, no QA reports.
- Internal "verify" is also absent: there are no automated stage gates analogous to the scaffold's `checks.py` (coverage matrix, finite-z, plausible flag counts).
- Spec: a first-class verification subsystem — `verify/checks.py` (internal gates) **and** `verify/reconcile.py` + `references/` adapters comparing the published value against an **independent authoritative source**, with QA reports and a `--strict` CI gate. `nyc-school-outcomes-spec.md:160-176,210-231`; scaffold `pipeline/verify/reconcile.py:1-233` (four-layer check: `source_raw` / `analysis_table` / `website_json` / `reference`, headline = website-vs-reference), `pipeline/verify/checks.py:1-78`, `pipeline/verify/references/{base,manual_csv,socrata,cached_file}.py`. The `manual_csv` adapter (`references/manual_csv.py`) is the lowest-friction starting point: a reviewer records official per-school figures in a CSV and `reconcile.py` diffs against them.
- **Status — addressed in PR B (`verify/`).** A read-only validation harness now closes the *correctness* portion of this gap: **(ii) base case** (DB vs the exact ingested file — 0 mismatches across ~29k metric cells + 826,354 survey cells) and **(iii) spot check** (DB vs the NYC School Snapshot). Scope so far = the directly-ingested components, `ALL` subgroup; **(iv)** computed-metric checks and a different-publisher (NYSED) cross-check remain pending. See `verify/METHODOLOGY.md`.

---

## Summary read

The prototype is a **well-built data warehouse + multi-persona site** whose
*infrastructure* (peer groups, suppression, universe tagging, survey dual-scores,
provenance rows, long-format schema) closely matches the spec's intent and should be
kept. The divergences are concentrated in the **statistical core and verification
spine**:

1. **Outlier statistic** is rank-based (decile), not model-based (residual-z) — no reliability weighting (#2, #6).
2. **Trend/time analysis** is a raw cross-year delta with **no regime handling** (#4, #5).
3. **No correctness check against official per-school data** at all (#9).

Notably, the prototype's **own editorial standards already prefer** the continuous
regression baseline and already reason in residual / comparability-regime terms
(`docs/agents/quantitative-knowledge.md:38-47,18-36`; `data/stories/answers.ts`),
so the spec's method formalizes a direction the team has independently arrived at —
which de-risks adoption.