02 — Gap Analysis: Prototype vs. Design Spec

Each dimension is marked Aligned / Diverges / Missing with a one-line "why it matters" and file:line citations. Spec = nyc-school-outcomes-spec.md. Scaffold (reference Python implementation) = reference-scaffold/. Prototype = this repository (the repo root).

Legend: 🟢 Aligned · 🟡 Diverges · 🔴 Missing

1. Peer groups — 🟡 Diverges

Why it matters: matching on demographics and regressing them out answer subtly different questions; the spec wants the outcome adjusted, not just the comparison set constrained.

• Prototype: KNN(K=40) with hard filters (grade band + admission bucket) + weighted Euclidean over 7 demographic shares (1.0) + topic Jaccard (0.5). scripts/loaders/nycenet-comparison-groups.ts:24-95.
• The peer group only constrains who you're ranked against; the outcome value itself is never adjusted for covariates — the percentile is a raw midrank within the matched set. scripts/loaders/compute-percentiles.ts:141-156.
• Key question (answered): the prototype matches on covariates; it does not regress them out. The spec's method fits outcome ~ covariates (+ admission) per year and standardizes the residual — covariates enter the model, not a filter. Spec §3.1 (nyc-school-outcomes-spec.md:91-113); scaffold pipeline/analysis/engine.py:36-53 (fit_year).
• Consequence: peer-group construction sensitivity (K, distance weights, the ×30 Jaccard scale at nycenet-comparison-groups.ts:88) directly determines flags; the regression makes that a continuous, weight-free prediction. Spec §3.1 (nyc-school-outcomes-spec.md:93) and the team's own note (docs/agents/quantitative-knowledge.md:38-47).
• Verdict nuance: the spec explicitly says keep peer groups as complementary display + a convergence cross-check (nyc-school-outcomes-spec.md:93,150,287). So this is "diverges in role," not "throw away."

2. Outlier definition — 🟡 Diverges (the core methodological gap)

Why it matters: a fixed-10%-per-tail rule flags schools whether or not the gap is meaningful or just small-N noise, and produces a rank rather than an effect size with uncertainty.

• Prototype: top/bottom decile of the peer-group percentile — comparison_group_percentile ≥ 0.9 / ≤ 0.1. lib/queries/schools.ts:166-172. The compute step has no reliability weighting; a 30-tested-student school and a 300-tested-student school are ranked identically (compute-percentiles.ts:141-156). Min peer count is the only guard (≥5, :148).
• Prototype's own spec defers proportion/low-count handling to "v2": SPECS.md:300, methodology/README.md:7.
• Spec: covariate-adjusted standardized residual z = r / √(s² + se_i²), flag |z| ≥ z_threshold (default 2.0), direction-aware. nyc-school-outcomes-spec.md:102-111; scaffold engine.py:72-95 (standardize, flag_outliers) + outliers.py:40-56.
• Practical difference:
  • Decile always flags ~10% of each tail per metric, regardless of whether the spread is real → over-flags in tight peer groups, under-flags in dispersed ones; flags noisy small-N schools at the extremes (exactly what quantitative-knowledge.md:101-109 warns about — "0% on n=3").
  • Residual-z flags only schools whose gap is large relative to its own uncertainty: se_i = √(p(1−p)/N) means a small-N school needs a much larger raw gap to clear |z|≥2 (engine.py:56-79). It yields a continuous effect size, a fitted "expected" value, and a count that can be zero or many — not a forced 10%.
• Decision is deliberately deferred (per KX decisions in the runbook): run residual-z alongside decile, diff which schools flip, decide replace-vs-supplement after. This is sequenced first in 03_integration_plan.md.

3. Proclivity decile — 🟡 Diverges (display/control hybrid, not a statistical control)

Why it matters: if "expected outcome" is a separate decile badge rather than the model's prediction, the outlier flag and the "punching above its profile" narrative are computed from different notions of "expected."

• Prototype: proclivity_decile = ntile(10) of the mean percentile rank across 3 challenge features (pctEconDis, pctEll, pctSwd). scripts/loaders/compute-proclivity.ts:17,92-115.
• It is not a statistical control on the outcome: it is stored on the school and used only as a filter/sort in the outlier query (lib/queries/schools.ts:176-181,195-196) and a display badge (components/proclivity-badge.tsx). The percentile/outlier computation never references it.
• Reconcile with spec covariates: the spec's covariate set is broader — base demographic set = econ-disadvantaged/ENI, ELL, SWD, race composition (Black/Hispanic/Asian/White), enrollment size, plus band-specific admission (G&T for ES, admissionsmethod_prog1 for MS, method1 for HS). nyc-school-outcomes-spec.md:54-68,373-393 (covariates.yaml). The spec explicitly says: unify — the regression's covariate set subsumes proclivity; "expected" becomes the model's continuous prediction (nyc-school-outcomes-spec.md:288). So proclivity's 3 features are a strict subset of the model covariates.
• Verdict: keep proclivity as a display grouping (it's a good plain-language hook — nyc-school-outcomes-spec.md:279), but it is not the spec's control and shouldn't be conflated with it.

4. Trends / time — 🔴 Missing

Why it matters: a raw year-over-year delta on proficiency is exactly the view the spec calls "the single biggest reason a naive year-over-year change is wrong."

• Prototype: getYearOverYearMovers = raw to_value − from_value between two chosen years. lib/queries/schools.ts:272-318; rendered as a raw delta at app/ratings/movers/page.tsx:45-51,110-113.
• There is no time-dimension analysis beyond this: no within-year z, no slope, no regime, no persistence. The trend is on absolute proficiency, so it crosses scale breaks freely.
• Spec View 2: slope of the within-year standardized residual (z) fit within a comparability regime, with a t-test gate and leave-one-year-out sensitivity. nyc-school-outcomes-spec.md:115-129; scaffold pipeline/analysis/trends.py:18-60 (fit_trend, flag_trend) + trend_runner.py.
• The 2023 NextGen break: because the prototype trends on raw proficiency, a school can show a large "+delta" that is entirely the 2023→ standards reset or the 2024-25 cutscore move. The prototype's stories explicitly know this (data/stories/answers.ts:1000,1920,2128), but the movers query/UI does not enforce it — the awareness is editorial, not systematic.

5. Comparability years — 🔴 Missing in code (handled editorially only)

Why it matters: mixing pre/post-NextGen test years or pre/post-COVID absenteeism years compares non-comparable regimes.

• Prototype loaders ingest all years 2018–2025 with no comparability flagging: scripts/loaders/test-results.ts:12-23 (fixed 2018–2025 files), no regime/exclusion logic anywhere in lib/ or scripts/ (grep: regime/comparability hits only docs/ + data/stories/).
• 2020/2021 are not excluded in code; 2022 is not treated as ambiguous; the movers query will happily diff 2019→2024 across the NextGen break.
• The data needed to detect the break exists — Mean Scale Score is ingested alongside proficiency (test-results.ts:167-182) — but is not used for any comparability gate.
• Spec: config/comparability.yaml with explicit regimes, excluded_years: [2020,2021], ambiguous_years: [2022], cross_regime_reporting: standardized_only. nyc-school-outcomes-spec.md:70-83,395-411; scaffold config/comparability.yaml:5-26.
• The editorial layer does the right thing manually (e.g. data/stories/answers.ts:1770 filters movers to schools with both endpoints comparable), confirming the team agrees with the spec — it just isn't encoded.

6. Suppression / small-N — 🟢 Aligned (suppression) · 🟡 (no reliability weighting)

Why it matters: showing suppression builds trust; but suppression ≠ down-weighting noisy-but-published small-N cells.

• Suppression handling is good and matches the spec: low-N cells detected (s/*/n<5) at _fetch.ts:111-115, stored as suppressed=true, hidden from outlier lists by default with an opt-in toggle (lib/queries/schools.ts:207-214, includeSuppressed), shown not dropped per SPECS.md:301,264. Aligns with nyc-school-outcomes-spec.md:26,250-251.
• But published-yet-small-N cells are not down-weighted in the outlier computation — only a min-peer-count guard exists (compute-percentiles.ts:148). The spec's se_i term and min_n eligibility floor (outliers.py:46, outcomes.yaml:23) are the missing reliability mechanism. This is the same root as gap #2.

7. Survey rollups — 🟡 Diverges

Why it matters: if item wording/scale shifts across years, a cross-year survey trend (or even a level comparison) is measuring the instrument, not the school.

• Prototype rolls up surveys by regex-matching question text independently within each year (questions are year-versioned) and averaging positive_pct. scripts/loaders/compute-survey-metrics.ts:25-99,113-128. It stores both NYCDOE-published and derived domain scores per SPECS.md:165-173,302 — that transparency design is aligned.
• But there is no verification that a matched item's wording/scale is continuous across years before the values are compared over time. The keyword lists are static; a re-worded item in a new year either silently drops out or silently joins the average.
• Spec requires an explicit instrument-stable window, set only after verifying continuity (nyc-school-outcomes-spec.md:77,80,124,316); scaffold leaves survey_regimes as an intentional placeholder pending that verification (config/comparability.yaml:27-35).

8. Provenance & determinism — 🟡 Diverges

Why it matters: "every figure traceable to a data vintage + code commit" and "byte-identical re-runs" are headline requirements; partial provenance can't gate a release.

• Provenance — partial-aligned. DataLoad records source_name, source_url, script_path, git_sha, row counts, notes per run (_lib.ts:30-62, prisma/schema.prisma:399-419). Good foundation.
• Gaps vs spec §4.1/§5.1 (nyc-school-outcomes-spec.md:171-176,214-221):
  • No source-file content hash (sha256/bytes/retrieved_at/http_status) — cache is keyed by URL only (_fetch.ts:9-29); spec wants a per-file provenance.json manifest.
  • No golden/determinism tests — tests/unit/ is only format/normalize/routes; nothing asserts a re-run reproduces identical outputs. Scaffold has make golden + tests/test_golden.py + committed fixtures (Makefile:43-51, tests/fixtures/golden/ela_math_g3/).
  • Determinism is also harder to guarantee because the pipeline mutates a live DB in place (e.g. compute-proclivity.ts:124 resets all deciles each run; nycenet-comparison-groups.ts:211-217 deletes+recreates all groups) rather than producing a versioned, hashable artifact. Spec wants staged immutable artifacts → versioned JSON (nyc-school-outcomes-spec.md:159-167) with analysis_table_sha256 in every output (to_website_json.py:54-73).

9. Correctness vs reality — 🔴 Missing — the single largest gap

Why it matters: nothing currently confirms that a number the site shows for a specific school matches what NYC/NYSED officially report for that school. For a civic-transparency tool that may flag a named school as a "negative outlier," this is the highest-stakes check.

• No reconciliation / official-data correctness check exists anywhere in the prototype. A full-repo grep for reconcile|verify|official|residual|ground.?truth|golden finds matches only in editorial docs (docs/agents/*, docs/editorial-process.md) describing the story-writing review process — not a data-verification harness. There are no reference adapters, no per-school cross-check, no QA reports.
• Internal "verify" is also absent: there are no automated stage gates analogous to the scaffold's checks.py (coverage matrix, finite-z, plausible flag counts).
• Spec: a first-class verification subsystem — verify/checks.py (internal gates) and verify/reconcile.py + references/ adapters comparing the published value against an independent authoritative source, with QA reports and a --strict CI gate. nyc-school-outcomes-spec.md:160-176,210-231; scaffold pipeline/verify/reconcile.py:1-233 (four-layer check: source_raw / analysis_table / website_json / reference, headline = website-vs-reference), pipeline/verify/checks.py:1-78, pipeline/verify/references/{base,manual_csv,socrata,cached_file}.py. The manual_csv adapter (references/manual_csv.py) is the lowest-friction starting point: a reviewer records official per-school figures in a CSV and reconcile.py diffs against them.
• Status — addressed in PR B (verify/). A read-only validation harness now closes the correctness portion of this gap: (ii) base case (DB vs the exact ingested file — 0 mismatches across ~29k metric cells + 826,354 survey cells) and (iii) spot check (DB vs the NYC School Snapshot). Scope so far = the directly-ingested components, ALL subgroup; (iv) computed-metric checks and a different-publisher (NYSED) cross-check remain pending. See verify/METHODOLOGY.md.

Summary read

The prototype is a well-built data warehouse + multi-persona site whose infrastructure (peer groups, suppression, universe tagging, survey dual-scores, provenance rows, long-format schema) closely matches the spec's intent and should be kept. The divergences are concentrated in the statistical core and verification spine:

1. Outlier statistic is rank-based (decile), not model-based (residual-z) — no reliability weighting (#2, #6).
2. Trend/time analysis is a raw cross-year delta with no regime handling (#4, #5).
3. No correctness check against official per-school data at all (#9).

Notably, the prototype's own editorial standards already prefer the continuous regression baseline and already reason in residual / comparability-regime terms (docs/agents/quantitative-knowledge.md:38-47,18-36; data/stories/answers.ts), so the spec's method formalizes a direction the team has independently arrived at — which de-risks adoption.