# 01 — Prototype Inventory (`mwg1378/nyc-edu-data`)

> Prototype inventory — read-only audit; no prototype code was modified.
> Clone audited at `main` @ `a4baa1a` ("Editorial sweep: finish the remaining ~30 stories"),
> 157 tracked files, single branch, pushed 2026-05-31.
>
> This is one of the design/review docs in `docs/design/`. The reference Python
> scaffold these docs cite lives in `docs/design/reference-scaffold/` (scaffold paths
> like `pipeline/...` are relative to it); `nyc-school-outcomes-spec.md` is in this
> folder. See `docs/design/README.md` for the map.

---

## 1. Stack summary

| Layer | Technology | Evidence |
|---|---|---|
| Language | **TypeScript** (primary), some Python helpers | repo language = TypeScript; `package.json`; `scripts/loaders/*.py` |
| Framework | **Next.js 16.2.6** (App Router, React 19, Turbopack) | `package.json:24,27`; `app/` tree; `next.config.ts` |
| DB / ORM | **Supabase (Postgres) via Prisma 7.8** (`@prisma/adapter-pg`) | `package.json:15-17`; `prisma/schema.prisma`; `lib/supabase/*` |
| Data access | Server components query Prisma directly; **no JSON data files read by the site** | `lib/queries/schools.ts:2`; JSON-read grep found none in `app/`,`lib/` |
| UI | shadcn/ui + Tailwind v4 + `@base-ui/react` + lucide + sonner | `package.json:18,22-23,29-33`; `components/ui/` |
| Auth | Supabase SSR (Google OAuth); most pages public | `SPECS.md:230-239`; `lib/supabase/`, `app/auth/` |
| Ingest libs | `xlsx`, `csv-parse`, `pg`, `zod`, `tsx`, `dotenv` | `package.json:19,21,26,34,43,48` |
| AI | `@anthropic-ai/sdk` (editorial tooling; no runtime AI features in v1) | `package.json:14`; `SPECS.md:246-252` |
| Tests | **vitest** (3 unit specs only) | `vitest.config.ts`; `tests/unit/` |
| Lint | **eslint-config-next** flat config (core-web-vitals + typescript) | `eslint.config.mjs` |
| TS | strict, ES2017, bundler resolution, `@/*` alias | `tsconfig.json` |
| Deploy | **Vercel** | `vercel.json`, `next.config.ts` |
| Hooks/CI | local `.githooks/pre-push` runs unit tests only; **no `.github/` → no CI** | `.githooks/pre-push`; `find .github` → absent |

**Entry points / scripts** (`package.json:5-12`): `dev` (`next dev -p 3008`),
`build` (`prisma generate && next build`), `start`, `lint` (`eslint`),
`test` (`vitest`), `prepare` (wires `.githooks`).
Data pipeline is **not** in `package.json`; loaders run ad-hoc via `tsx`
(e.g. `tsx scripts/loaders/test-results.ts`), gated by `.env.local` (`DIRECT_URL`).

> ⚠️ **Stack mismatch with our scaffold.** The prototype's analysis layer is
> **TypeScript + a live Postgres DB**. Our reference scaffold
> (`reference-scaffold/`) is **Python** (`numpy/pandas/scipy/pyarrow`,
> `requirements.txt`) emitting **static JSON files** the site reads. This is the
> single largest "does it integrate cleanly" question — carried into `03_integration_plan.md` §Stack-fit.

---

## 2. Data flow (raw → site)

The prototype is **DB-centric**, not file-centric. There is no `analysis_table`
artifact and no JSON export stage; computed columns are written back into
Postgres and the site reads them live.

```
NYCDOE InfoHub xlsx / NYC Open Data CSV / Socrata / NYSED
        │  (fetchAndCache → data-sources/_cache, gitignored)         scripts/loaders/_fetch.ts:14
        ▼
[ingest+parse, one loader per source]  test-results.ts, demographics.ts, graduation-results.ts,
        │   normalizeDbn / normalizeYearLoose, isSuppressed                school-quality.ts, attendance.ts,
        ▼                                                                  nyc-school-survey.ts, nysed-ssec.ts, hs-directory.ts,
   Postgres (Prisma)  ── school_year_metrics (long format)                 nyc-od-schools.ts, nyc-od-charters.ts, dbn-beds-crosswalk.ts
        │              ── schools, school_year_demographics, survey_*
        │              ── data_loads (provenance row per run)         scripts/loaders/_lib.ts:30-62
        ▼
[derive / compute, in-DB]
   derive-admission-topic.ts   → schools.admission_category, topic_tags
   nycenet-comparison-groups.ts→ comparison_groups (DERIVED, K=40) + schools.comparison_group_id
   compute-proclivity.ts       → schools.proclivity_decile / proclivity_score
   compute-percentiles.ts      → school_year_metrics.comparison_group_percentile / citywide_percentile
   compute-composites.ts       → composite_* metric rows
   compute-survey-metrics.ts   → survey_* metric rows
        ▼
[site reads DB live via Prisma]   lib/queries/schools.ts
   app/ratings/outliers  (decile filter on comparison_group_percentile)
   app/ratings/movers    (raw cross-year value delta)
   app/schools/[dbn], /parents/*, /researchers/*, /stories/*
```

Sources, vintages & provenance:
- Source catalog: `SPECS.md:321-339`; per-source notes stub at `data-sources/README.md`.
- Test results pull years **2018–2025** from two fixed InfoHub xlsx URLs (`scripts/loaders/test-results.ts:12-23`); ingests grades 3–8 + "All Grades", both **% Level 3+4** and **Mean Scale Score** (`test-results.ts:154-182`).
- Provenance row per load: `source_name`, `source_url`, `script_path`, **`git_sha`**, row counts, notes (`_lib.ts:30-62`, `prisma/schema.prisma:399-419`). **No source-file content checksum/sha256** is recorded; the cache is keyed by URL only (`_fetch.ts:9-29`).
- Raw downloads are gitignored (`.gitignore:32-46`) — not committed.

---

## 3. Analysis-relevant module map

| Concern | File:line | What it does |
|---|---|---|
| **Peer groups (KNN)** | `scripts/loaders/nycenet-comparison-groups.ts:24-95,180-207` | K=40 KNN. **Hard filters**: same grade band + admission bucket (`ADMISSION_BUCKETS` :39-47). **Weighted distance**: Euclidean over 7 demographic shares (weight 1.0, :66-79) + topic **Jaccard** distance ×30 (weight 0.5, :81-95). Peers stored as JSON list in `comparison_groups.methodology_notes` (:193-204). All groups are `DERIVED`; NYCENET portal is *not* scraped (no bulk endpoint — :1-3). |
| **Outlier percentile** | `scripts/loaders/compute-percentiles.ts:60-78,141-156` | Midrank percentile (0..1) of a school's value **within its stored peer set** (needs ≥5 peers present, :148) and within all default-included schools (citywide). `LOWER_BETTER` inverted via `sign` (:133-134). Pure rank — **no covariate adjustment of the outcome**. |
| **Outlier flagging (decile)** | `lib/queries/schools.ts:166-172` | "Outlier" = top decile (`comparison_group_percentile ≥ 0.9`) or bottom decile (`≤ 0.1`). Threshold lives in the **query layer**, not the compute step. |
| **Proclivity decile** | `scripts/loaders/compute-proclivity.ts:17-35,92-115` | `ntile(10)` of the mean percentile rank across 3 "challenge" features (`pctEconDis`, `pctEll`, `pctSwd`). Stored on `schools.proclivity_decile/score`. Used only as a **filter/sort** in the outlier query (`schools.ts:176-181,195-196`) — never feeds the percentile/outcome computation. |
| **Composites** | `scripts/loaders/compute-composites.ts:20-59,126-149` | Simple mean of constituent raw metrics (ES/MS academic, HS outcomes, teacher-voice); requires all inputs present. Built on **raw rates**, not residuals. |
| **Survey rollups** | `scripts/loaders/compute-survey-metrics.ts:25-99,113-185` | Regex keyword-matches question text per (metric, respondent) **independently each year** (questions are year-versioned), then averages `positive_pct`. **No cross-year wording/scale-continuity verification** before rollup. Subgroup always `ALL` (:135-136). |
| **Trends / movers** | `lib/queries/schools.ts:272-318` + `app/ratings/movers/page.tsx:45-51,110-113` | `getYearOverYearMovers` = raw `to_value − from_value` between two chosen years; requires both non-null + not suppressed. **No within-year z, no comparability regime, no NextGen-break awareness.** UI renders the raw delta. |
| **Site read layer** | `lib/queries/schools.ts` | `getOutliers` (decile filter + sort + 1h memo, :147-270), `getPeerGroupContext` (:26-48), `getYearOverYearMovers` (:272-318). All Prisma/Postgres. |
| **Admission/topic derivation** | `scripts/loaders/derive-admission-topic.ts` | Derives `admission_category` + `topic_tags` (inputs to peer matching). |
| **Ingest helpers** | `scripts/loaders/_fetch.ts`, `_lib.ts`, `_bulk.ts`, `normalize.ts` | fetch+cache, DataLoad provenance + `gitSha()`, bulk upsert, year/DBN normalization. |
| **Schema** | `prisma/schema.prisma` | `School.proclivityDecile/Score` (:173-174), `admissionCategory`/`topicTags` (:176-178), `SchoolYearMetric.comparisonGroupPercentile/citywidePercentile` (:289-290, indexed :301), `ComparisonGroup` (:221), `DataLoad` w/ `gitSha` (:399-419). Long-format facts per `SPECS.md:106-122`. |

**Methodology surface to users:** `app/methodology/page.tsx`; the decile rule is the
documented v1 method (`SPECS.md:300`, `methodology/README.md:7`). Planned methodology
write-ups (`methodology/README.md:16-24`) are not yet authored.

**Notable signal — the editorial side already prefers regression residuals.** The
team's own quant playbook says decile means are a piecewise-constant baseline that
distorts the extremes and that "a continuous regression baseline … is the more
defensible methodology" (`docs/agents/quantitative-knowledge.md:38-47`), warns that
top-N residual lists are selected-on-outcome (:18-26), and recommends a 3-year
rolling residual for stability (:28-36). The published stories speak in
"proclivity-adjusted residual" / "gap above expectation" terms
(`docs/agents/plain-language.md:60,67`; `data/stories/answers.ts:1216`) even though
the **code** computes deciles. The spec's residual-z method formalizes language the
project already uses editorially.

---

## 4. Contribution conventions (for Phase 5)

| Item | Finding | Source |
|---|---|---|
| Default branch | `main` (single branch; `origin/HEAD → main`) | `git branch -a` |
| **Branch protection** | **None** — `branches/main/protection` returns 404. `main` is technically pushable directly; the *plan* mandates branch+PR anyway. | `gh api …/branches/main/protection` |
| Open PRs / Issues | **None** (clean slate; no in-flight work to collide with) | `gh pr list`, `gh issue list` |
| Visibility / license | **private**, **no license file** (license decision deferred, spec §9.10) | `gh api repos/…`; `nyc-school-outcomes-spec.md:338` |
| Lint/format | eslint flat config (`eslint-config-next`: core-web-vitals + typescript). No Prettier config; no separate formatter. | `eslint.config.mjs` |
| Type checking | strict TS; not run in pre-push hook | `tsconfig.json:7` |
| Test runner | vitest; specs in `tests/**/*.test.ts`, `@`→repo root alias; node env | `vitest.config.ts`; existing: `tests/unit/{format,normalize,routes}.test.ts` |
| Pre-push gate | `.githooks/pre-push` runs `vitest run tests/unit/` only (no lint/typecheck/build). Wired via `prepare` script. | `.githooks/pre-push`; `package.json:11` |
| CI | **None** (`.github/` absent). All gating is local. | `find .github` |
| CODEOWNERS | **None** | `find` (absent) |
| Commit style | Sentence-case, scope-prefixed imperative summaries; heavy "Editorial sweep: …" series; no Conventional-Commits prefixes (`feat:`/`fix:`). | `git log --oneline -8` |
| Agent config | `.claude/` and `.codex/` hooks present (editorial agent tooling) | repo root |
| Data hygiene | raw xlsx/csv, `_cache`, `.env*`, scrape caches all gitignored | `.gitignore:9-46` |
| Test fixtures | sample CSVs allowed (`!data-sources/**/sample/*.csv`) | `.gitignore:38` |

**House-style implications:** new code matching the prototype is **TypeScript** under
`scripts/loaders/` (or `lib/`), with **vitest** tests under `tests/`. There is no CI
to extend — only the local pre-push hook — so any new gate (golden/determinism,
reconcile) must be added as a vitest spec (TS) or a new hook/CI workflow. A Python
pipeline (the scaffold) would introduce a **second toolchain** with no existing
runner — see `03_integration_plan.md`.

---

## 5. README staleness flag

`README.md:5` still says **"Status: Spec phase. No code yet."** — stale; the repo is a
fully built Next.js app with a complete loader pipeline and ~30 published stories.
Worth correcting in an early housekeeping PR.