# probe/ — measurement scripts behind `findings.md`

Small, read-only Python scripts that produced the numbers in
`../findings.md`. They are the Miami-Dade analogue of the first-fetch
batches recorded in `docs/cities/philly/findings.md`, kept as code so every
figure in the plan is reproducible (house rule: derive quantities from the
process, don't quote them).

They run against a scratch directory (default
`data/cities/miami/sources/_probe/`, git-ignored; override with
`MIAMI_PROBE_DIR`) and need only `pandas`, `openpyxl`, `requests`, `scipy`
(`pipeline_philly/requirements.txt` + scipy).

| Script | Fetches | Produces |
|---|---|---|
| `ccd_probe.py` | NCES CCD 2023-24 directory (`029`), staff (`059`), lunch (`033`) zips (download them first, see below) | universe counts by status / type / charter / level; ID-scheme checks |
| `ccd_membership_year.py <url> <out.csv>` | one CCD school-membership zip (`052`, ~200 MB), streamed | the Miami-Dade (ST_LEAID `FL-13`) rows only |
| `ccd_small.py` | — (uses the two above) | size buckets by sector × level; the under-300 list; race composition; grade totals; `mdc_directory_enrollment_2324.csv` |
| `ccd_trend.py` | CCD 2018-19 directory zip | five-year school-level change, by sector / level / grade; `mdc_district_regular_trend_1819_2324.csv` |
| `gis_util.py` | — (uses `gis/layer_0.json`) | utilization distribution from the county GIS layer; capacity-definition and enrollment-vintage tests |
| `closure_overlap.py` | — | lift of the district's 2026 closure / review lists over the CCD size tail (`closure_overlap.json`) |

Run order, from the repo root:

```bash
export MIAMI_PROBE_DIR=data/cities/miami/sources/_probe
mkdir -p $MIAMI_PROBE_DIR/ccd $MIAMI_PROBE_DIR/gis
cd $MIAMI_PROBE_DIR/ccd
for f in ccd_sch_029_2324_w_1a_073124 ccd_sch_059_2324_l_1a_073124 ccd_sch_033_2324_l_1a_073124 ccd_sch_029_1819_w_1a_091019; do
  curl -sS -o $f.zip https://nces.ed.gov/ccd/Data/zip/$f.zip && unzip -o -q $f.zip -d unz_$f
done
mv unz_ccd_sch_029_1819_w_1a_091019 unz_1819_dir
cd -
curl -sS "https://services.arcgis.com/8Pc9XBTAsYuxx9Ny/arcgis/rest/services/SchoolSite_gdb/FeatureServer/0/query?where=1%3D1&outFields=*&returnGeometry=true&outSR=4326&f=json&resultRecordCount=2000" -o $MIAMI_PROBE_DIR/gis/layer_0.json
python docs/cities/miami/probe/ccd_probe.py
python docs/cities/miami/probe/ccd_membership_year.py https://nces.ed.gov/ccd/Data/zip/ccd_sch_052_2324_l_1a_073124.zip ccd/mdc_membership_2324.csv
python docs/cities/miami/probe/ccd_membership_year.py https://nces.ed.gov/ccd/Data/zip/ccd_sch_052_1819_l_1a_091019.zip ccd/mdc_membership_1819.csv
python docs/cities/miami/probe/ccd_small.py
python docs/cities/miami/probe/ccd_trend.py
python docs/cities/miami/probe/gis_util.py
python docs/cities/miami/probe/closure_overlap.py
```

The sha256 / bytes / retrieved-at of every file fetched on 2026-09-15 are in
`data/cities/miami/sources/<source_id>/2026-09-15/provenance.json`; the
derived per-school tables are committed under `data/cities/miami/reference/`
as the audit trail (same convention as Philly's reference CSVs).

These scripts are throwaway probes, not loaders. The real P1 loaders will
live under `scripts/loaders/miami/` and write to `miami_*` tables, per
`03_integration_plan.md`.