# 02 — Gap Analysis (Miami source layer vs. NYC and Philly)

Same format as `docs/cities/philly/02_gap_analysis.md`. "Same" = lift
verbatim; "Adapt" = config or schema tweak; "New" = code with no analogue.

## A. Source ingest

| # | Dimension | NYC | Philly | Miami | Change |
|---|---|---|---|---|---|
| A.1 | Connector kind | xlsx via InfoHub CDN | Socrata + PDE xlsx + SDP CDN | FLDOE xlsx/xls (two URL forms, yearly name drift), MSID cookie-gated text, M-DCPS CMS xlsx/pdf, county ArcGIS REST, federal csv zips, **PDF tables** | **Adapt + New.** xlsx/xls connector lifts; add an ArcGIS REST reader, a per-year assessment-page scraper, and a PDF-table connector with two-parser agreement. |
| A.2 | Join key | DBN | ULCS (+ AUN/PA code) | **4-digit school number, zero-padded** (+ NCES id, virtual provider number) | **Adapt.** Padding normalizer at the boundary. |
| A.3 | Year encoding | `YYYY-YY` | `YYYY-YY` / end-year int | file suffix `2425`; assessment `Spring25`; School Grades `25`; graduation `2425` = cohort entering 2021-22; FISH report date | **Adapt.** Add `suffix_yyyy` and `spring_yy` year formats to `sources.yaml`. |
| A.4 | Suppression | n<5 / n<10 | n<11 | **n<10**: `*`, `**`, blank, `Less_10`; none in SESIR/ADA; perturbation in EDFacts | **Adapt.** Token census per file. |
| A.5 | Provenance | sha256 + bytes + retrieved_at | same | same, plus `fetched_from` (origin host) | **Same.** |
| A.6 | URL stability | stable | stable | FLDOE stable per file but blocked from cloud on `www`; assessment names drift | **Adapt.** Origin rewrite; scrape year pages. |
| A.7 | Publisher coverage of charters | charters via separate directory | SDP files district-only; state files all | **all FLDOE files carry charters** as district-13 rows; M-DCPS survey and board files are district-run-centric | **Simpler than Philly.** No per-sector source routing for state metrics. |

## B. Schema (Postgres, additive `miami_*` tables)

| # | Dimension | Philly | Miami | Change |
|---|---|---|---|---|
| B.1 | `schools` PK | `ulcs_code` | `school_number` (text, 4-digit) with `nces_id`, `msid_type`, `virtual_provider_number` | **Adapt.** |
| B.2 | `school_type` enum | regular/charter/compctc/cyber/specialed/contracted | `REGULAR, CHARTER, ALTERNATIVE, ESE_CENTER, TECHNICAL_COLLEGE, VIRTUAL, DJJ, ADULT` + `is_charter`, `is_schools_of_hope`, `charter_open_year` | **Adapt.** |
| B.3 | admission category | catchment / lottery / special admission / … | `NEIGHBORHOOD, MAGNET_SCHOOL, MAGNET_PROGRAM_HOST, CHARTER_LOTTERY, CHOICE, ALTERNATIVE` from MSID `MAGNET_STATUS` (Z/P/S) + `CS_SCHOOL_CHOICE` | **Adapt.** Smaller enum; magnet-within-school is a flag, not a type. |
| B.4 | subgroup enum | PA set incl. NON_BINARY | Florida set: `ALL, WHITE, BLACK, HISPANIC, ASIAN, AMER_INDIAN_AK_NATIVE, HAWAIIAN_PAC_ISL, TWO_OR_MORE_RACES, ECON_DISADV, ELL, SWD, MIGRANT, AT_RISK, MALE, FEMALE`; CRDC adds `IDEA, SECTION_504` | **Adapt.** No NON_BINARY, no temp-housing (EDFacts has HOM as a subgroup — carry `HOMELESS` for that source). |
| B.5 | long-format metrics | same | same; `population_cut` stays `n/a` (Florida publishes one cut) | **Same.** |
| B.5a | first-class designations | `essa_designation` | `school_grade` (A–F, per year, as a metric *and* the latest on `schools`), `title_i`, `sir_rating`, `plp_flag`, `school_of_excellence` | **New columns / metrics.** |
| B.5b | growth indicator | PVAAS | FLDOE learning gains (all + lowest 25%) | **Adapt.** `learning_gains_*` keys. |
| B.5c | **facilities** | — | `fish_capacity_permanent`, `fish_capacity_school`, `utilization_*`, `cofte` as metrics; `MiamiSchoolEvent` table | **New.** `06_…md §3`. |
| B.6 | `metric_definitions` | ~90 keys | ~110 keys (FAST × grade × subject, EOCs, gains, grad × subgroup, two absenteeism, SESIR rates, survey rollups, enrollment/capacity/PPE) + `MetricUnit.USD` | **Same structure.** |
| B.7 | peer groups | KNN k=40 | KNN k=40 (flip-diff 20/30), re-tuned features (M.9) | **Same code**, new config. |
| B.8 | survey raw table | PSES shape | climate workbook shape (school × item × respondent: agree/disagree/DK %, N) | **Same.** |
| B.9 | geography | learning network, council district | `region` (N/C/S), `board_district`, `municipality`, `zip`, attendance-boundary polygon id | **Adapt.** |

## C. Analysis pipeline

| # | Capability | Change |
|---|---|---|
| C.1–C.2 | engine, outliers | **Same.** |
| C.3 | comparability regimes | **Adapt.** FCAT 2.0 / FSA / FAST; EOC NGSSS→FSA→B.E.S.T.; grad rate unchanged; absenteeism 2017-18 definition start, 2020-21 COVID; School Grades 2023 baseline + 2026 scale change. |
| C.4 | assemble | **Adapt** to `miami_*`. |
| C.5 | outcomes config | **New file.** ~16 outcomes. |
| C.6 | covariates | **Adapt.** `pct_black, pct_hispanic, pct_ell, pct_econ_disadv, cep_flag, log_enrollment, magnet_flag, sector`. |
| C.7 | **closure-track analysis** (lift table, decomposition, cost curve, ablations) | **New.** `06_…md §4`. |

## D. Validation harness

| # | Check | Miami posture |
|---|---|---|
| D.1 | (i) completeness | source-only school numbers vs MSID active list; per-year rosters vs School Grades history. |
| D.2 | (ii) base case | re-read of the exact FLDOE/M-DCPS file; PDF tables via two parsers. |
| D.3 | (iii-a) presentation | School Grades components; Report Cards spot sample; ABC book capacity blocks; Statistical Highlights totals. |
| D.4 | (iii-b) independent | CCD (enrollment, staff), EDFacts (absenteeism), CRDC (absenteeism, discipline), county GIS (capacity). |
| D.5 | (iv) computed | utilization, change rates, survey rollups, SESIR rates — packet recomputation. |

## E. Front-end

| # | Surface | Change |
|---|---|---|
| E.1 | navbar city tabs | **Adapt.** `CITIES` gains `{ key: "miami", label: "Miami", basePath: "/miami" }`; search API returns a `miami` bucket. |
| E.2 | Outliers / Movers / school page / methodology / proof | **Same** components under `app/miami/*`, Miami labels (Region badge instead of Learning Network; School Grade badge instead of ESSA designation). |
| E.3 | stories | **All new** (~20 stubs; `00_overview.md §4`). |
| E.4 | **enrollment/capacity working analysis + map** | **New page** (`06_…md §6`). Needs one new chart type (utilization histogram with marked schools) and a map component (attendance boundaries + points) — the first map in the product. |
| E.5 | routes test | add `app/miami/*` routes to `tests/unit/routes.test.ts` when they land. |

## F. Bottom line

- **Source layer:** mostly new connectors, but simpler universe handling than Philly (one publisher covers everyone).
- **Schema:** same shape; new facilities metrics and an events table.
- **Analysis / verification / front-end:** lifted; one new analytical track and one new map surface.
- **Effort:** roughly **50% of the NYC build** — more schools and more files than Philly, PDF parsing, and a new page, offset by the solved methodology and reusable components. See `03_integration_plan.md` for the per-item estimate.