# pipeline_miami/

Python sidecar for the Miami-Dade build. Same shape as `pipeline_philly/`:
`discover.py` turns `docs/cities/miami/sources.yaml` entries into fetched
files with provenance + a findings markdown (`discover_srs.py` registers
the per-grade assessment files); `verify/` is the read-only validation
harness (closure and outcomes tracks, cross-publisher checks, residual-z
checks); `parse/` holds the PDF-table parsers for the facilities sources and
the climate-survey normaliser; `analysis/` holds the residual-z engine and
the survey-versus-enrollment test; `export_metric_index.py` writes the
metric index the methodology pages read.

Two Miami-specific fetch rules live in `discover.py`:

- **`www.fldoe.org` → `origin.fldoe.org`.** The public host answers Akamai
  403 to cloud egress; the origin host serves the identical path. The
  canonical `www` URL is what `provenance.json` records as `url`; the host
  actually used is recorded as `fetched_from`.
- **MSID needs a session cookie.** `eds.fldoe.org/EDS/MasterSchoolID/Downloads/All_schools.cfm`
  rejects a cold request; `discover.py` loads `index.cfm` first with a
  `requests.Session()`.

## Setup

```bash
python3 -m venv pipeline_miami/.venv && source pipeline_miami/.venv/bin/activate
pip install -r pipeline_miami/requirements.txt
pip install 'psycopg[binary]'     # verify/ only
```

## Usage

```bash
python -m pipeline_miami.discover                          # list sources + state
python -m pipeline_miami.discover --source fldoe_school_grades_2024-25
python -m pipeline_miami.discover --all                    # every resolved source
python -m pipeline_miami.parse.fish_los fldoe_fish_los_dade_2024-25   # PDF → csv
python -m pipeline_miami.parse.abc_book mdcps_abc_book_2026-27
python -m pipeline_miami.parse.climate_survey                    # 20 survey workbooks → per-item tables + item crosswalk
python -m pipeline_miami.analysis.residual_z                     # WLS residual-z per outcome × year → CSV + packet
python -m pipeline_miami.analysis.survey_enrollment              # pre-registered sentiment-vs-enrollment test → packet
python -m pipeline_miami.export_metric_index                     # sources per metric + served coverage → metric_index.json
python -m pipeline_miami.verify.run_all                          # harness; writes docs/qa_reports/miami/*
```

## Outputs

- `data/cities/miami/sources/<source_id>/<YYYY-MM-DD>/<file>` — the file (git-ignored)
- `data/cities/miami/sources/<source_id>/<YYYY-MM-DD>/provenance.json` — sha256 + retrieved_at + bytes + url (+ fetched_from)
- `pipeline_miami/findings/<source_id>.md` — sheets / columns / sample rows
- `data/cities/miami/reference/parsed/<source_id>.csv` — parsed PDF tables (committed: they are the audit trail)
- `data/cities/miami/reference/parsed/survey/<source_id>.csv` (+ `.school.csv`) and `survey_items.csv` — normalised survey items and the crosswalk (committed)
- `data/cities/miami/reference/residual_z.csv`, `data/analysis/miami-residual-z.json`, `data/analysis/miami-survey-enrollment.json` — analysis outputs (committed)