# probe/ — measurement scripts behind the Philly design docs

Read-only Python scripts that measure the numbers quoted in the numbered docs,
kept as code so every figure is reproducible (house rule: derive quantities
from the process, don't quote them). Same idiom as
`docs/cities/miami/probe/` and `docs/design/probe/`.

They fetch into a scratch directory — default
`data/cities/philly/sources/_probe/`, git-ignored; override with
`PHILLY_PROBE_DIR` — and cache every download, so a rerun is offline and the
cache is the audit trail.

| Script | Reaches | Produces |
|---|---|---|
| `facilities.py` | SDP Facilities Data Dashboard (Qlik Sense Engine API, anonymous) + SDP Master School List + NCES CCD via the Urban Institute API | the measured figures in [`06_enrollment_capacity_utilization.md`](../06_enrollment_capacity_utilization.md): what the app holds, the single-vintage negative result, the ULCS join, the computed-value check, utilization distributions, the lift of the district's own Close recommendation over utilization, the CCD and floor-area checks (§6.2, §6.3), and the older, 2017-labelled capacity figure of §6.5 (`BLDGCAPACITY` counted once per building against Site Capacity, with the app's own table and labels for it). Sections H1 and I also print what a sum over the extract's repeated rows reads, so the size of that hazard is a printed number. The §4.1–4.3 site-grain figures are the packet's (`scripts/analysis/philly-capacity-packet.ts`), on its distinct-site universe; the §3.2 school roll-up checks are the loader's |

Requires `requests`, `websocket-client`, `scipy`.

```bash
python docs/cities/philly/probe/facilities.py
```

Two things to know before changing it:

- **The Qlik endpoint returns 403 without a browser `User-Agent`.** The
  rejection happens at the load balancer, before the engine sees the request,
  so it looks like an auth failure and is not one. Anonymous access is
  otherwise open (`mustAuthenticate: false`).
- **`GetHyperCubeData` caps a page at roughly 10,000 cells.** `PAGE` is set to
  250 rows; at the probe's 14 columns, raising it past ~700 returns `Result too
  large` (error 6001).

The app id (`2d47abaf-…`) is read out of the dashboard's `env-config` Nuxt
plugin and is not documented anywhere; if the dashboard is rebuilt it may
change.

## The loader path is a different script

`facilities.py` measures; nothing it writes reaches the database. The extract
the pipeline runs on is `pipeline_philly/fetch_facilities.py`, which talks to
the same Qlik app and writes CSVs that are **committed** rather than cached:

```bash
pip install websocket-client                            # the Engine API is WebSocket, not HTTP
python -m pipeline_philly.fetch_facilities
python -m pipeline_philly.fetch_facilities --dry-run    # fetch, report, write nothing
```

|  | `probe/facilities.py` | `pipeline_philly/fetch_facilities.py` |
|---|---|---|
| Purpose | reproduce every figure in [`06_…`](../06_enrollment_capacity_utilization.md) | produce the tables the loader and the packet read |
| Reaches | Qlik + Master School List + NCES CCD (Urban Institute API) | Qlik only |
| Writes | a scratch cache under `data/cities/philly/sources/_probe/`, git-ignored | `data/cities/philly/reference/parsed/sdp_facilities_sites.csv` and `…_buildings.csv`, plus `sources/sdp_facilities_dashboard/<date>/provenance.json` — all committed |
| Consumed by | the doc, by a reader rerunning it | `npx tsx scripts/loaders/philly/facilities.ts` (the sites CSV; its hash pin covers both), and `scripts/analysis/philly-capacity-packet.ts` (both) |
| Needs | `requests`, `websocket-client`, `scipy` | `websocket-client` (in `pipeline_philly/requirements.txt`) |

The CSVs are committed because the source is an undocumented Qlik app that can
change or disappear without notice, so the extract is the audit trail rather
than the dashboard. The fetcher refuses to write a short table — a truncated
pull exits non-zero instead of quietly shrinking a committed file.

Both scripts hit the two limits described above: the 403 without a browser
`User-Agent`, and the ~10,000-cell page cap. They differ in how much of the app
they read. `facilities.py` pulls a cube at site grain (with `School Year`),
where FCI and `gross_square_footage` arrive as site measures. It caches the
rest separately in `fpp_building_basis.json`:

- the site cube on the fetcher's own dimensions;
- a per-site cube that aggregates the building fields with
  `Aggr(…, id_site_eos, id_bldg_eos)`, one value per building, for floor area
  (§6.3) and `BLDGCAPACITY` (§6.5);
- a (site × programme) cube;
- every record of the app table that holds `BLDGCAPACITY`, with the number of
  current buildings each record links to;
- one row per current building, with no ULCS dimension, so no programme repeat;
- the app's own measure definitions and labels.

The fetcher pulls the site cube and a **second one at building grain**, the
source of year built, use type, per-building area and the older capacity figure
(`BLDGCAPACITY`, [§6.5](../06_enrollment_capacity_utilization.md)). The
analysis packet joins the two on `site_id`; the loader stores no building
field.
Adding a building dimension to the site cube instead would silently change its
grain: capacity, enrollment and utilization are published on each site row, and
a site can hold a garage and a fieldhouse alongside its classroom block. A
school on several sites carries its whole enrollment on each of them (doc
§3.2), so a school figure counts it once.

The building grain has a second trap one level down. `ULCS Code` is a field of
the app's Buildings table, which holds one record per building **per
programme**, so a building on a site hosting two programmes is emitted twice by
the fetcher's building cube and is counted twice by a plain `Sum` of
`gross_square_footage` inside Qlik itself. `BLDGCAPACITY` sits in a separate
table keyed by building, so a plain `Sum([BLDGCAPACITY])` is safe, but summing
the fetcher's rows is not. Section I prints the row-summing reading (median,
share under 50%, ratio spread, by level) beside the per-building one, labelled
`[counts a repeated building twice]`, and section H1 does the same for floor
area.

`make philly-facilities-fetch` runs the fetcher through the `pipeline_philly`
venv; `make capacity-help` prints where it sits in the load order.