probe/ — measurement scripts behind findings.md
Small, read-only Python scripts that produced the numbers in
../findings.md. They are the Miami-Dade analogue of the first-fetch
batches recorded in docs/cities/philly/findings.md, kept as code so every
figure in the plan is reproducible (house rule: derive quantities from the
process, don't quote them).
They run against a scratch directory (default
data/cities/miami/sources/_probe/, git-ignored; override with
MIAMI_PROBE_DIR) and need only pandas, openpyxl, requests, scipy
(pipeline_philly/requirements.txt + scipy).
| Script | Fetches | Produces |
|---|---|---|
ccd_probe.py | NCES CCD 2023-24 directory (029), staff (059), lunch (033) zips (download them first, see below) | universe counts by status / type / charter / level; ID-scheme checks |
ccd_membership_year.py <url> <out.csv> | one CCD school-membership zip (052, ~200 MB), streamed | the Miami-Dade (ST_LEAID FL-13) rows only |
ccd_small.py | — (uses the two above) | size buckets by sector × level; the under-300 list; race composition; grade totals; mdc_directory_enrollment_2324.csv |
ccd_trend.py | CCD 2018-19 directory zip | five-year school-level change, by sector / level / grade; mdc_district_regular_trend_1819_2324.csv |
gis_util.py | — (uses gis/layer_0.json) | utilization distribution from the county GIS layer; capacity-definition and enrollment-vintage tests |
closure_overlap.py | — | lift of the district's 2026 closure / review lists over the CCD size tail (closure_overlap.json) |
Run order, from the repo root:
export MIAMI_PROBE_DIR=data/cities/miami/sources/_probe
mkdir -p $MIAMI_PROBE_DIR/ccd $MIAMI_PROBE_DIR/gis
cd $MIAMI_PROBE_DIR/ccd
for f in ccd_sch_029_2324_w_1a_073124 ccd_sch_059_2324_l_1a_073124 ccd_sch_033_2324_l_1a_073124 ccd_sch_029_1819_w_1a_091019; do
curl -sS -o $f.zip https://nces.ed.gov/ccd/Data/zip/$f.zip && unzip -o -q $f.zip -d unz_$f
done
mv unz_ccd_sch_029_1819_w_1a_091019 unz_1819_dir
cd -
curl -sS "https://services.arcgis.com/8Pc9XBTAsYuxx9Ny/arcgis/rest/services/SchoolSite_gdb/FeatureServer/0/query?where=1%3D1&outFields=*&returnGeometry=true&outSR=4326&f=json&resultRecordCount=2000" -o $MIAMI_PROBE_DIR/gis/layer_0.json
python docs/cities/miami/probe/ccd_probe.py
python docs/cities/miami/probe/ccd_membership_year.py https://nces.ed.gov/ccd/Data/zip/ccd_sch_052_2324_l_1a_073124.zip ccd/mdc_membership_2324.csv
python docs/cities/miami/probe/ccd_membership_year.py https://nces.ed.gov/ccd/Data/zip/ccd_sch_052_1819_l_1a_091019.zip ccd/mdc_membership_1819.csv
python docs/cities/miami/probe/ccd_small.py
python docs/cities/miami/probe/ccd_trend.py
python docs/cities/miami/probe/gis_util.py
python docs/cities/miami/probe/closure_overlap.py
The sha256 / bytes / retrieved-at of every file fetched on 2026-09-15 are in
data/cities/miami/sources/<source_id>/2026-09-15/provenance.json; the
derived per-school tables are committed under data/cities/miami/reference/
as the audit trail (same convention as Philly's reference CSVs).
These scripts are throwaway probes, not loaders. The real P1 loaders will
live under scripts/loaders/miami/ and write to miami_* tables, per
03_integration_plan.md.