pipeline_miami/

Python sidecar for the Miami-Dade build. Same shape as pipeline_philly/: discover.py turns docs/cities/miami/sources.yaml entries into fetched files with provenance + a findings markdown (discover_srs.py registers the per-grade assessment files); verify/ is the read-only validation harness (closure and outcomes tracks, cross-publisher checks, residual-z checks); parse/ holds the PDF-table parsers for the facilities sources and the climate-survey normaliser; analysis/ holds the residual-z engine and the survey-versus-enrollment test; export_metric_index.py writes the metric index the methodology pages read.

Two Miami-specific fetch rules live in discover.py:

• www.fldoe.org → origin.fldoe.org. The public host answers Akamai 403 to cloud egress; the origin host serves the identical path. The canonical www URL is what provenance.json records as url; the host actually used is recorded as fetched_from.
• MSID needs a session cookie. eds.fldoe.org/EDS/MasterSchoolID/Downloads/All_schools.cfm rejects a cold request; discover.py loads index.cfm first with a requests.Session().

Setup

python3 -m venv pipeline_miami/.venv && source pipeline_miami/.venv/bin/activate
pip install -r pipeline_miami/requirements.txt
pip install 'psycopg[binary]'     # verify/ only

Usage

python -m pipeline_miami.discover                          # list sources + state
python -m pipeline_miami.discover --source fldoe_school_grades_2024-25
python -m pipeline_miami.discover --all                    # every resolved source
python -m pipeline_miami.parse.fish_los fldoe_fish_los_dade_2024-25   # PDF → csv
python -m pipeline_miami.parse.abc_book mdcps_abc_book_2026-27
python -m pipeline_miami.parse.climate_survey                    # 20 survey workbooks → per-item tables + item crosswalk
python -m pipeline_miami.analysis.residual_z                     # WLS residual-z per outcome × year → CSV + packet
python -m pipeline_miami.analysis.survey_enrollment              # pre-registered sentiment-vs-enrollment test → packet
python -m pipeline_miami.export_metric_index                     # sources per metric + served coverage → metric_index.json
python -m pipeline_miami.verify.run_all                          # harness; writes docs/qa_reports/miami/*

Outputs

• data/cities/miami/sources/<source_id>/<YYYY-MM-DD>/<file> — the file (git-ignored)
• data/cities/miami/sources/<source_id>/<YYYY-MM-DD>/provenance.json — sha256 + retrieved_at + bytes + url (+ fetched_from)
• pipeline_miami/findings/<source_id>.md — sheets / columns / sample rows
• data/cities/miami/reference/parsed/<source_id>.csv — parsed PDF tables (committed: they are the audit trail)
• data/cities/miami/reference/parsed/survey/<source_id>.csv (+ .school.csv) and survey_items.csv — normalised survey items and the crosswalk (committed)
• data/cities/miami/reference/residual_z.csv, data/analysis/miami-residual-z.json, data/analysis/miami-survey-enrollment.json — analysis outputs (committed)