probe/ — measurement scripts behind the NYC design docs
Read-only Python scripts that produce the numbers quoted in the numbered design
docs, kept as code so every figure is reproducible (house rule: derive
quantities from the process, don't quote them). The Miami analogue is
docs/cities/miami/probe/.
They fetch into a scratch directory — default
data/reference/sources/_probe/, git-ignored; override with NYC_PROBE_DIR —
and cache every download, so a rerun is offline and the cache is the audit
trail. Both also read the committed parse of the SCA's PDFs
(data/cities/nyc/reference/parsed/) and refuse a year whose two-parser gate
did not pass.
| Script | Reads | Produces |
|---|---|---|
bluebook_util.py | SCA Blue Book (gkd7-3vk7, q9xk-w9iv) + the five PDF-parsed years + the 2019-20 school directory (wg9x-4ke6) + three DOE Demographic Snapshot vintages (s52a-8aq6, c7ru-d68s, the current InfoHub xlsx) | §1–§4 of 10_nyc_enrollment_capacity_utilization.md: panel shape, snapshot → school-year date-stamping, join rate and its failure modes, computed-value check, utilization distributions at both grains, enrollment-vs-capacity decomposition, the two capacity definitions, district breakdown, and the disappearance lift table |
bluebook_verify.py | the same Blue Book datasets + NCES CCD (via the Urban Institute API) + the DOE Building Space Usage room inventory (wavz-fkw8) | the cross-source verification in 10 §5.1–§5.3: enrollment against an independent publisher with the DOE's own file as a control, capacity against the physical room inventory (a room set fixed before any comparison; square feet per seat and seats per room are measured, not assumed), and the retired extract against the live dataset |
What neither script produces, and 10 says so where each appears: the §2.1
parse figures (in the gate report beside each parsed CSV), the §4.3 segment
table (the packet's decomposition.segments), the §4.2 Miami-Dade and
Philadelphia figures (those cities' packets), the §4.2 charter and District 75
counts, and the Pre-K measurements in §5.2 and §5.4.
Requires requests, openpyxl, scipy.
python docs/design/probe/bluebook_util.py # what the SCA publishes
python docs/design/probe/bluebook_verify.py # is it right
Roughly 30 MB cold for bluebook_util.py and another ~250 MB for the room
inventory in bluebook_verify.py; both are seconds warm. Every section
prints its own method line, so the output is readable without the doc beside it.