# 05 — Validation Spec (Miami-Dade)

Adaptation of `verify/METHODOLOGY.md` (NYC) and
`docs/cities/philly/05_validation_spec.md` to Florida sources. Same four
checks, same posture, different publishers.

## The four checks

- **(i) Completeness** — every school in the MSID active list (and every school in the year's School Grades roster) appears in each served (metric, year); source-only and served-only cells reported.
- **(ii) Base case** — the served value equals an independent re-read of the exact ingested file (xlsx/xls by `pandas`; PDF tables by a second parser). *The binding gate.*
- **(iii-a) Spot check, same publisher** — FLDOE School Grades components; Florida Report Cards on a stratified sample; M-DCPS ABC book and school dashboards for the closure track.
- **(iii-b) Independent publisher** — NCES CCD, EDFacts (EdDataExpress), CRDC, county GIS.
- **(iv) Computed values** — derived metrics recomputed in Python from validated inputs (packet recomputation, as in the Philly full QA).

## Per-metric expectations

| Metric | (ii) source | (iii-a) | (iii-b) | Tolerance (ii / iii) |
|---|---|---|---|---|
| `fast_ela_g{3..10}_proficiency`, `fast_math_g{3..8}_proficiency` | SRS xls, per grade | School Grades `ELA/Math Achievement` (all grades pooled, ±2 pp; only a sanity check) | — (no independent publisher for FL assessments) | ±1.0 / ±2.0 |
| `eoc_{alg1,geometry,biology,civics,ushistory}_proficiency` | SRS xls (spring administration) | School Grades `Science` / `Social Studies Achievement` where applicable | — | ±1.0 / ±2.0 |
| `learning_gains_{ela,math}`, `_l25` | Gains xls | School Grades gains components (exact expected) | — | ±0.5 / ±0.5 |
| `school_grade`, components | School Grades xlsx | Report Cards | — | exact |
| `graduation_rate_4yr` (+ subgroups) | FedGradRate xls | School Grades `Graduation Rate` (lagged one year — align cohorts) | — | ±0.5 / ±1.0 |
| `chronic_absenteeism_rate` (10%) | ABS21Days10Comparison xlsx | Report Cards ESSA absenteeism | EDFacts FS195 (ESSA 10%; perturbed) | ±0.5 / ±1.5 |
| `absent_21plus_rate` | ABS21Days10Comparison / ABS21DAYSTDistSchl | — | CRDC 15+ days: correlation only (different cut) | ±0.5 / n.a. |
| `attendance_rate` (ADA/ADM) | ADAADMSchl xlsx | dashboards ADA | — | ±0.2 |
| `sesir_*_per_100` | SESIR xlsx | — | CRDC harassment/bullying counts (correlation) | exact counts / n.a. |
| `survey_*` rollups | climate workbooks | Power BI dashboard spot sample | — | ±0.0 (exact from raw) |
| `enrollment_oct` | MembBySchoolByGrade xlsx | Statistical Highlights district totals; dashboards | **CCD membership** (federal copy of the same submission; mismatch is a finding) | exact / ±0.5% district |
| `fish_capacity_permanent`, `fish_capacity_school`, `cofte`, `utilization_school_capacity_pct` | FISH LOS PDF (two parsers) | ABC book `Perm FISH Cap` for targeted schools (exact) | county GIS `CAPACITY` (permanent; vintage differences expected and reported) | exact |
| `per_pupil_expenditure_*` | ESSA PPE xlsx | Report Cards PPE | — | exact |
| `teacher_fte`, `students_per_teacher` | CCD staff file | dashboards staff counts (headcount vs FTE — report both) | — | exact / n.a. |

## Known reconciliation hazards (stated before the first load)

1. **School-number padding** (`0026` vs `26`) — normalize before any join; a padding bug shows up as (i) failures, so the completeness check runs first.
2. **Virtual provider numbers** — accountability files carry an extra key column; the virtual programs must not collide with the district's own rows.
3. **Denominators differ by survey**: October membership (Survey 2) vs annual unduplicated enrollment (Survey 5, ~6% larger; students at two schools counted at each) vs February FTE (Survey 3) vs COFTE (funding FTE). Every served rate names its denominator.
4. **Three utilization numerators** (COFTE in FISH LOS; headcount in the ABC book; a 2024-25 count in GIS) and two capacities (permanent; school capacity incl. relocatables). Stored separately; never mixed.
5. **Graduation-year indexing**: `2425` = cohort entering grade 9 in 2021-22; School Grades reports it lagged one year.
6. **Suppression tokens** vary by file and year (`*`, `**`, blank, `Less_10`); the token census is part of every load.
7. **Assessment regime breaks** (2015 FSA, 2023 FAST; no 2020) and the 2023 "informational baseline" — (ii) is per-year and unaffected; (iii-a) against School Grades must use the same year's file.
8. **School Grades achievement pools grades**; SRS files are per grade. The (iii-a) proficiency check is a sanity bound, not an oracle.
9. **CEP and the 1.6 multiplier** in economic-disadvantage shares; comparisons across CEP status flagged.
10. **CCD lag** (~1.5 years): (iii-b) on enrollment covers years ≤ 2023-24 only.

## What lands first

1. (ii) base case for every C1 and P1 metric, full coverage, before anything is published — NYC's bar.
2. (iii-b) CCD-vs-FLDOE enrollment on every school × year in the overlap (this is the strongest check available in any city so far, since both copies derive from the same submission).
3. (iii-a) School Grades components and ABC-book capacity on the affected schools.
4. (iv) packet recomputation for the closure page (mirrors `pipeline_philly/verify/absenteeism_full_qa.py`).

## Authority

Authoritative for Miami-Dade verification expectations; divergences from
the NYC spec are intentional and listed above.