02 — Gap Analysis (Miami source layer vs. NYC and Philly)
Same format as docs/cities/philly/02_gap_analysis.md. "Same" = lift
verbatim; "Adapt" = config or schema tweak; "New" = code with no analogue.
A. Source ingest
| # | Dimension | NYC | Philly | Miami | Change |
|---|---|---|---|---|---|
| A.1 | Connector kind | xlsx via InfoHub CDN | Socrata + PDE xlsx + SDP CDN | FLDOE xlsx/xls (two URL forms, yearly name drift), MSID cookie-gated text, M-DCPS CMS xlsx/pdf, county ArcGIS REST, federal csv zips, PDF tables | Adapt + New. xlsx/xls connector lifts; add an ArcGIS REST reader, a per-year assessment-page scraper, and a PDF-table connector with two-parser agreement. |
| A.2 | Join key | DBN | ULCS (+ AUN/PA code) | 4-digit school number, zero-padded (+ NCES id, virtual provider number) | Adapt. Padding normalizer at the boundary. |
| A.3 | Year encoding | YYYY-YY | YYYY-YY / end-year int | file suffix 2425; assessment Spring25; School Grades 25; graduation 2425 = cohort entering 2021-22; FISH report date | Adapt. Add suffix_yyyy and spring_yy year formats to sources.yaml. |
| A.4 | Suppression | n<5 / n<10 | n<11 | n<10: *, **, blank, Less_10; none in SESIR/ADA; perturbation in EDFacts | Adapt. Token census per file. |
| A.5 | Provenance | sha256 + bytes + retrieved_at | same | same, plus fetched_from (origin host) | Same. |
| A.6 | URL stability | stable | stable | FLDOE stable per file but blocked from cloud on www; assessment names drift | Adapt. Origin rewrite; scrape year pages. |
| A.7 | Publisher coverage of charters | charters via separate directory | SDP files district-only; state files all | all FLDOE files carry charters as district-13 rows; M-DCPS survey and board files are district-run-centric | Simpler than Philly. No per-sector source routing for state metrics. |
B. Schema (Postgres, additive miami_* tables)
| # | Dimension | Philly | Miami | Change |
|---|---|---|---|---|
| B.1 | schools PK | ulcs_code | school_number (text, 4-digit) with nces_id, msid_type, virtual_provider_number | Adapt. |
| B.2 | school_type enum | regular/charter/compctc/cyber/specialed/contracted | REGULAR, CHARTER, ALTERNATIVE, ESE_CENTER, TECHNICAL_COLLEGE, VIRTUAL, DJJ, ADULT + is_charter, is_schools_of_hope, charter_open_year | Adapt. |
| B.3 | admission category | catchment / lottery / special admission / … | NEIGHBORHOOD, MAGNET_SCHOOL, MAGNET_PROGRAM_HOST, CHARTER_LOTTERY, CHOICE, ALTERNATIVE from MSID MAGNET_STATUS (Z/P/S) + CS_SCHOOL_CHOICE | Adapt. Smaller enum; magnet-within-school is a flag, not a type. |
| B.4 | subgroup enum | PA set incl. NON_BINARY | Florida set: ALL, WHITE, BLACK, HISPANIC, ASIAN, AMER_INDIAN_AK_NATIVE, HAWAIIAN_PAC_ISL, TWO_OR_MORE_RACES, ECON_DISADV, ELL, SWD, MIGRANT, AT_RISK, MALE, FEMALE; CRDC adds IDEA, SECTION_504 | Adapt. No NON_BINARY, no temp-housing (EDFacts has HOM as a subgroup — carry HOMELESS for that source). |
| B.5 | long-format metrics | same | same; population_cut stays n/a (Florida publishes one cut) | Same. |
| B.5a | first-class designations | essa_designation | school_grade (A–F, per year, as a metric and the latest on schools), title_i, sir_rating, plp_flag, school_of_excellence | New columns / metrics. |
| B.5b | growth indicator | PVAAS | FLDOE learning gains (all + lowest 25%) | Adapt. learning_gains_* keys. |
| B.5c | facilities | — | fish_capacity_permanent, fish_capacity_school, utilization_*, cofte as metrics; MiamiSchoolEvent table | New. 06_…md §3. |
| B.6 | metric_definitions | ~90 keys | ~110 keys (FAST × grade × subject, EOCs, gains, grad × subgroup, two absenteeism, SESIR rates, survey rollups, enrollment/capacity/PPE) + MetricUnit.USD | Same structure. |
| B.7 | peer groups | KNN k=40 | KNN k=40 (flip-diff 20/30), re-tuned features (M.9) | Same code, new config. |
| B.8 | survey raw table | PSES shape | climate workbook shape (school × item × respondent: agree/disagree/DK %, N) | Same. |
| B.9 | geography | learning network, council district | region (N/C/S), board_district, municipality, zip, attendance-boundary polygon id | Adapt. |
C. Analysis pipeline
| # | Capability | Change |
|---|---|---|
| C.1–C.2 | engine, outliers | Same. |
| C.3 | comparability regimes | Adapt. FCAT 2.0 / FSA / FAST; EOC NGSSS→FSA→B.E.S.T.; grad rate unchanged; absenteeism 2017-18 definition start, 2020-21 COVID; School Grades 2023 baseline + 2026 scale change. |
| C.4 | assemble | Adapt to miami_*. |
| C.5 | outcomes config | New file. ~16 outcomes. |
| C.6 | covariates | Adapt. pct_black, pct_hispanic, pct_ell, pct_econ_disadv, cep_flag, log_enrollment, magnet_flag, sector. |
| C.7 | closure-track analysis (lift table, decomposition, cost curve, ablations) | New. 06_…md §4. |
D. Validation harness
| # | Check | Miami posture |
|---|---|---|
| D.1 | (i) completeness | source-only school numbers vs MSID active list; per-year rosters vs School Grades history. |
| D.2 | (ii) base case | re-read of the exact FLDOE/M-DCPS file; PDF tables via two parsers. |
| D.3 | (iii-a) presentation | School Grades components; Report Cards spot sample; ABC book capacity blocks; Statistical Highlights totals. |
| D.4 | (iii-b) independent | CCD (enrollment, staff), EDFacts (absenteeism), CRDC (absenteeism, discipline), county GIS (capacity). |
| D.5 | (iv) computed | utilization, change rates, survey rollups, SESIR rates — packet recomputation. |
E. Front-end
| # | Surface | Change |
|---|---|---|
| E.1 | navbar city tabs | Adapt. CITIES gains { key: "miami", label: "Miami", basePath: "/miami" }; search API returns a miami bucket. |
| E.2 | Outliers / Movers / school page / methodology / proof | Same components under app/miami/*, Miami labels (Region badge instead of Learning Network; School Grade badge instead of ESSA designation). |
| E.3 | stories | All new (~20 stubs; 00_overview.md §4). |
| E.4 | enrollment/capacity working analysis + map | New page (06_…md §6). Needs one new chart type (utilization histogram with marked schools) and a map component (attendance boundaries + points) — the first map in the product. |
| E.5 | routes test | add app/miami/* routes to tests/unit/routes.test.ts when they land. |
F. Bottom line
- Source layer: mostly new connectors, but simpler universe handling than Philly (one publisher covers everyone).
- Schema: same shape; new facilities metrics and an events table.
- Analysis / verification / front-end: lifted; one new analytical track and one new map surface.
- Effort: roughly 50% of the NYC build — more schools and more files than Philly, PDF parsing, and a new page, offset by the solved methodology and reusable components. See
03_integration_plan.mdfor the per-item estimate.