Find What Works: Schools
← All source documentsRaw file
Source document

pipeline_philly/README.md

Served verbatim from the project repository. Internal working document conventions apply: documents may reference file paths, branch names, and findings-ledger anchors from the repo.

pipeline_philly/

Python sidecar for the Philly build. v0: a discover.py script that turns the TBD URLs in docs/cities/philly/sources.yaml into actual fetched files with provenance + a discovered-columns/sheets manifest.

Mirrors KX's verify/ pattern: read-only outward calls, all artifacts under data/cities/philly/sources/ and pipeline_philly/findings/. No DB access yet — that lands when the schema decision (O.1) is made and P1 proper begins.

Setup

python3 -m venv pipeline_philly/.venv && source pipeline_philly/.venv/bin/activate
pip install -r pipeline_philly/requirements.txt

Usage

# List sources and their resolution state (TBD vs fetchable)
python -m pipeline_philly.discover

# Fetch one source, record provenance, write a findings markdown
python -m pipeline_philly.discover --source pde_pssa

# Fetch every resolved (non-TBD) source
python -m pipeline_philly.discover --all

Outputs

  • data/cities/philly/sources/<source_id>/<YYYY-MM-DD>/<file> — the downloaded file
  • data/cities/philly/sources/<source_id>/<YYYY-MM-DD>/provenance.json — sha256 + retrieved_at + bytes + url
  • pipeline_philly/findings/<source_id>.md — sheets, columns, sample first row

Discipline

  • Source files are git-ignored (.gitignore rules added). Only the provenance.json and findings/*.md are committed.
  • Each successful discovery should result in a sources.yaml PR replacing TBDs with the real values found.