probe/ — measurement scripts behind the Philly design docs

Read-only Python scripts that measure the numbers quoted in the numbered docs, kept as code so every figure is reproducible (house rule: derive quantities from the process, don't quote them). Same idiom as docs/cities/miami/probe/ and docs/design/probe/.

They fetch into a scratch directory — default data/cities/philly/sources/_probe/, git-ignored; override with PHILLY_PROBE_DIR — and cache every download, so a rerun is offline and the cache is the audit trail.

Requires requests, websocket-client, scipy.

python docs/cities/philly/probe/facilities.py

Two things to know before changing it:

• The Qlik endpoint returns 403 without a browser User-Agent. The rejection happens at the load balancer, before the engine sees the request, so it looks like an auth failure and is not one. Anonymous access is otherwise open (mustAuthenticate: false).
• GetHyperCubeData caps a page at roughly 10,000 cells. PAGE is set to 250 rows; at the probe's 14 columns, raising it past ~700 returns Result too large (error 6001).

The app id (2d47abaf-…) is read out of the dashboard's env-config Nuxt plugin and is not documented anywhere; if the dashboard is rebuilt it may change.

The loader path is a different script

facilities.py measures; nothing it writes reaches the database. The extract the pipeline runs on is pipeline_philly/fetch_facilities.py, which talks to the same Qlik app and writes CSVs that are committed rather than cached:

pip install websocket-client                            # the Engine API is WebSocket, not HTTP
python -m pipeline_philly.fetch_facilities
python -m pipeline_philly.fetch_facilities --dry-run    # fetch, report, write nothing

The CSVs are committed because the source is an undocumented Qlik app that can change or disappear without notice, so the extract is the audit trail rather than the dashboard. The fetcher refuses to write a short table — a truncated pull exits non-zero instead of quietly shrinking a committed file.

Both scripts hit the two limits described above: the 403 without a browser User-Agent, and the ~10,000-cell page cap. They differ in how much of the app they read. facilities.py pulls a cube at site grain (with School Year), where FCI and gross_square_footage arrive as site measures. It caches the rest separately in fpp_building_basis.json:

• the site cube on the fetcher's own dimensions;
• a per-site cube that aggregates the building fields with Aggr(…, id_site_eos, id_bldg_eos), one value per building, for floor area (§6.3) and BLDGCAPACITY (§6.5);
• a (site × programme) cube;
• every record of the app table that holds BLDGCAPACITY, with the number of current buildings each record links to;
• one row per current building, with no ULCS dimension, so no programme repeat;
• the app's own measure definitions and labels.

The fetcher pulls the site cube and a second one at building grain, the source of year built, use type, per-building area and the older capacity figure (BLDGCAPACITY, §6.5). The analysis packet joins the two on site_id; the loader stores no building field. Adding a building dimension to the site cube instead would silently change its grain: capacity, enrollment and utilization are published on each site row, and a site can hold a garage and a fieldhouse alongside its classroom block. A school on several sites carries its whole enrollment on each of them (doc §3.2), so a school figure counts it once.

The building grain has a second trap one level down. ULCS Code is a field of the app's Buildings table, which holds one record per building per programme, so a building on a site hosting two programmes is emitted twice by the fetcher's building cube and is counted twice by a plain Sum of gross_square_footage inside Qlik itself. BLDGCAPACITY sits in a separate table keyed by building, so a plain Sum([BLDGCAPACITY]) is safe, but summing the fetcher's rows is not. Section I prints the row-summing reading (median, share under 50%, ratio spread, by level) beside the per-building one, labelled [counts a repeated building twice], and section H1 does the same for floor area.

make philly-facilities-fetch runs the fetcher through the pipeline_philly venv; make capacity-help prints where it sits in the load order.