# 03 — Integration Plan (P0–P5 + C0–C3)

Sequenced capabilities, one branch + one focused PR each
(`04_contribution_plan.md`). Effort = S (<1 day) / M (1–3 days) / L (>3
days) / XL (>1 week). Risk = Low / Med / High.

**Sequencing principle:** the closure track (C0–C3) is independent of the
outcomes machinery and is the timely deliverable, so it runs in parallel
with P0–P1 and ships first. Everything else follows Philly's order.

```
P0 ──► P1 ──► P2 ──► P3 ──► P4 ──► P5
 └───► C0 ──► C1 ──► C2 ──► C3   (ships before P3)
```

---

> **Status 2026-09-15:** P0–P5 and C0–C3 shipped, plus the residual-z engine (`pipeline_miami/analysis/residual_z.py`) and the climate-survey track. What remains is listed under *Not built* in `README.md`. Runbook: `07_runbook.md`.

## P0. Repo setup + scaffolding — **S · Low**

- `prisma/schema.prisma`: `MiamiSchool`, `MiamiMetricDefinition`, `MiamiSchoolYearMetric`, `MiamiComparisonGroup`, `MiamiDataLoad`, `MiamiSchoolEvent`; enums `MiamiSchoolType`, `MiamiAdmissionCategory`, `MiamiSubgroup`, `MiamiEventAction`; `MetricUnit` gains `USD`. Migration is additive.
- `pipeline_miami/` (copy of `pipeline_philly/` skeleton): `discover.py` with (a) `www.fldoe.org → origin.fldoe.org` rewrite recorded as `fetched_from`, (b) an MSID fetch that primes the session cookie, (c) inspectors for `.xls` (xlrd), PDF (pdfplumber page text + table extraction), ArcGIS JSON.
- `scripts/loaders/miami/_lib.ts`: `readLatestProvenance`, `startDataLoad`, `padSchoolNumber`, `suppressionToken` census helper, `schoolYearFromSuffix`.
- `pipeline_miami/config/{outcomes,covariates,comparability}.yaml` populated from `00_overview.md` M.5/M.9.
- `.gitignore` rules already added for `data/cities/miami/sources/**`.

*Gate:* `python -m pipeline_miami.discover` lists every `sources.yaml`
entry; `--source fldoe_school_grades_2024-25` re-fetches and reproduces the
committed sha256.

## C0. Closure-track sources — **M · Med**

- Fetch the full FLDOE membership series (2016-17 → 2025-26, by grade and by grade × race), FISH LOS Dade (2022-23 → 2024-25 and any earlier editions found under `/core/fileparse.php/20080/`), ESSA PPE (2018-19 → 2024-25), DEFP 2024 + 2025, COFTE, the ABC books 2019-20 → 2026-27, Statistical Highlights 2013-14 → 2025-26; CCD membership 2014-15 → 2022-23 (the two years already fetched bracket them).
- PDF-table parsers for FISH LOS (row = facility; multi-line names) and ABC capacity blocks; each parser's output is checked against a second extraction (camelot or per-page regex) and the ABC numbers already read by hand for the nine closed schools.
- Findings markdown per source under `pipeline_miami/findings/`.

*Gate:* every FISH facility row parses to 10 numeric fields; ABC blocks for
all targeted schools parse; both parsers agree cell-for-cell.

## P1. Source ingest — directory, accountability, tests, graduation, absenteeism, safety — **L · Med**

Loaders under `scripts/loaders/miami/`:

1. `msid_directory.ts` + `ccd_directory.ts` → `miami_schools` (three-way charter check; level from grade flags; region/board district from GIS; lat/lng from MSID with GIS fallback).
2. `school_grades.ts` → `school_grade` per year (1999 →), components as metrics, `title_i`, `pct_econ_disadv` (covariate), roster history.
3. `srs_assessments.ts` — walks each results year page, maps file prefixes to (subject, grade, administration, regime), loads proficiency + mean scale score + level distribution per school × grade.
4. `learning_gains.ts`.
5. `graduation.ts` — race and category files; cohort-year indexing documented.
6. `absenteeism.ts` — both definitions; ADA/ADM; the 2016-17 inaccuracy flag stored as a load note.
7. `sesir.ts` → incident counts and per-100 rates by category (mapping documented in methodology).
8. `metric_definitions.ts`.

*Gate:* row counts per (source, year) match the findings; smoke query
returns FAST ELA-3 proficiency for ≥ 290 Dade schools in 2024-25.

## C1. Closure-track loaders — **M · Low-Med**

- `membership.ts` (FLDOE by grade; by grade × race), `ccd_membership.ts` (cross-publisher copy), `fish_los.ts`, `abc_book.ts` (capacity blocks + projections), `essa_ppe.ts`, `ccd_staff.ts`, `school_events.ts` (hand-curated CSV → `MiamiSchoolEvent`, one row per board action with the PDF URL and page).
- Derived metrics: `enrollment_change_5yr_pct`, `k_to_g12_ratio`, `utilization_permanent_pct`, `utilization_school_capacity_pct` (COFTE-based, FISH LOS), `students_per_teacher`.

*Gate:* every district-run E/M/S school has enrollment for every year
2016-17 → 2025-26 and capacity for 2022-23 → 2024-25; the nine closed
schools' stored values equal the ABC book.

## P2. Climate-survey ingest — **M · Low**

- `climate_survey.ts`: students / parents / staff workbooks 2018-19 → 2024-25 → `miami_survey_responses` (school × year × item × respondent: agree/disagree/DK %, N, suppressed).
- Rollups: `survey_student_safety`, `survey_student_teaching`, `survey_staff_leadership`, `survey_parent_recommend`, `survey_student_grade_a_b` etc. — first-class set decided after item-text continuity check across years.

*Gate:* rollups present for ≥ 450 schools in 2024-25; item wording diffed
across years and recorded in a survey-regime config.

## C2. Closure-track validation — **M · Low**

- `pipeline_miami/verify/`: (i) completeness vs MSID; (ii) base case vs FLDOE xlsx / FISH parse / ESSA xlsx; (iii-a) ABC book + dashboards; (iii-b) CCD membership and staff, GIS capacity; (iv) packet recomputation. Full-QA JSON committed under `docs/qa_reports/miami/`.

*Gate:* PASS on all sections; QA report drives the page's noindex flag.

## C3. Closure-track analysis + page — **L · Med**

- Packet: `scripts/analysis/miami-enrollment-descriptives.ts` → `data/analysis/miami-enrollment.json` (trend by sector, grade pipeline, school-level change distribution, utilization distribution, lift table vs the 2026 lists, PPE vs size, decomposition, ablations).
- Page `app/miami/methodology/analysis/enrollment/page.tsx` in the working-analysis idiom; figures per `06_…md §6`; map component (attendance boundaries + points).
- Proof trees for every figure and stat (`components/proof/*` reused; `lib/miami/lineage.ts`).
- Methodology page `app/miami/methodology/metrics/enrollment-capacity/page.tsx` with the three utilization definitions.

*Gate:* full site walkthrough; editorial passes per `docs/editorial-process.md`.

## P3. Peer groups + percentiles — **S · Low**

`derive_peer_groups.ts` and `compute_percentiles.ts` copied from Philly
with Miami features (M.9); flip-diff at k = 20/30/40.

*Gate:* Outliers renders; magnet schools do not dominate the top decile
once choice status is a hard filter.

## P4. Validation harness (outcomes) — **M · Low-Med**

`components.yaml` for every P1 metric; (iii-a) School Grades components at
±2 pp; (iii-b) EDFacts absenteeism at ±1.5 pp on the 10% metric, CRDC 15+
days as a correlation check only (different definition).

## P5. Site + stories — **XL · Med**

Routes under `app/miami/*` mirroring `app/philly/*`; navbar tab; search
bucket; ~20 story stubs; methodology hub; `tests/unit/routes.test.ts`
additions.

---

## Effort

| Item | Effort |
|---|---|
| P0 | S |
| C0 + C1 + C2 + C3 | M + M + M + L ≈ 3–4 weeks |
| P1 | L |
| P2 | M |
| P3 | S |
| P4 | M |
| P5 | XL |

Total ≈ 9–12 weeks for one engineer who knows the Philly build; the
closure page can be live in the first month. Risk concentrates in C0 (PDF
parsing) and P1 (yearly assessment file drift).