# pipeline_philly/

Python sidecar for the Philly build. v0: a `discover.py` script that turns
the `TBD` URLs in `docs/cities/philly/sources.yaml` into actual fetched files
with provenance + a discovered-columns/sheets manifest.

Mirrors KX's `verify/` pattern: read-only outward calls, all artifacts under
`data/cities/philly/sources/` and `pipeline_philly/findings/`. No DB access
yet — that lands when the schema decision (O.1) is made and P1 proper begins.

## Setup

```bash
python3 -m venv pipeline_philly/.venv && source pipeline_philly/.venv/bin/activate
pip install -r pipeline_philly/requirements.txt
```

## Usage

```bash
# List sources and their resolution state (TBD vs fetchable)
python -m pipeline_philly.discover

# Fetch one source, record provenance, write a findings markdown
python -m pipeline_philly.discover --source pde_pssa

# Fetch every resolved (non-TBD) source
python -m pipeline_philly.discover --all
```

## Outputs

- `data/cities/philly/sources/<source_id>/<YYYY-MM-DD>/<file>` — the downloaded file
- `data/cities/philly/sources/<source_id>/<YYYY-MM-DD>/provenance.json` — sha256 + retrieved_at + bytes + url
- `pipeline_philly/findings/<source_id>.md` — sheets, columns, sample first row

## Discipline

- Source files are git-ignored (`.gitignore` rules added). Only the `provenance.json` and `findings/*.md` are committed.
- Each successful discovery should result in a `sources.yaml` PR replacing TBDs with the real values found.