05 — Validation Spec (Philly)

Adaptation of KX's NYC validation spec (docs/design/05_validation_spec.md + the as-built verify/METHODOLOGY.md) to Philly sources. Same four checks, same posture, different publishers and source files.

The four checks (terms used throughout)

Identical to KX's framing:

• (i) Completeness — are years or schools missing? Surfaced as "source-only" rows (in the source file but not in the served DB).
• (ii) Correctness — base case — does the DB match an independent re-read of the exact source file the loader ingested? (the binding correctness gate)
• (iii) Correctness — spot check — does the DB match the District's public-facing per-school report (SPREE for Philly; NYC School Snapshot for NYC)? (presentation cross-reference, not an independent oracle)
• (iv) Correctness — computed values — are derived metrics (composites, survey rollups) computed correctly from validated inputs? (deferred)

Sources by check

Posture for Philly's two-publisher fact

Philly is unusual in having two independent authoritative publishers for the same metric family:

• PDE publishes school-level PSSA + Keystone + cohort grad files. This is the ingest source of truth (the file the loader actually reads).
• SDP publishes its own copies plus the SPREE composite. This is the presentation reference (what families see).
• Future Ready PA Index publishes PDE's accountability bundle. This is the cross-publisher independent oracle — analogous to KX's deferred NYSED check, but available day one.

So Philly's (iii) spot check has two flavors:

• (iii-a) DB vs SPREE — same-publisher presentation cross-reference. Mirrors NYC's Snapshot check. Caveats from verify/METHODOLOGY.md apply (it's a presentation check, not an oracle).
• (iii-b) DB vs Future Ready PA Index — different-publisher independent check. The "deferred NYSED" gap NYC has does not exist here.

Both should ship. (iii-b) is the stronger check; (iii-a) is the one families will recognize.

Per-metric expectations

For each v1 metric, the table records the (ii) base case source + the (iii-a)

• (iii-b) references. Tolerances match KX's defaults (±1.0pp for (ii), ±1.5pp for (iii)) until Philly-specific tuning is needed.

TBD rows resolve when the source files are first fetched (P1) and the variable-name maps are written into verify-philly/snapshot.yaml.

Known reconciliation hazards (lessons from NYC)

From verify/METHODOLOGY.md:

• Different population cuts of the same data. NYC's published Snapshot uses an accountability cohort (Oct-31-register-attributed) while the ingested InfoHub file uses all tested students. Differences in either direction are expected. Philly may have the same hazard: SPREE uses PDE's accountability cohort; the bulk PSSA file uses all tested. Document whichever rule SDP uses in SPREE's Educator Guide and reconcile per-cell with the rule, not the headline.
• N differences with both directions. NYC's N can be larger or smaller than the source by year; same will be true in Philly. Don't assume "served is a subset of source."
• Test-recalibration discontinuities. PSSA was recalibrated 2014-15 (PA Core) and 2022-23 (post-COVID). Across-regime trend comparisons need the comparability config from pipeline/analysis/comparability.yaml. (i) and (ii) checks are per-year and unaffected; (iii) checks against SPREE that also report standardized scores may diverge across the regime boundary.

What lands first

The KX precedent: (ii) base case ships first, full coverage, all metrics; then (iii-a) SPREE spot check on a stratified sample; then (iii-b) Future Ready PA Index cross-publisher check; (iv) computed values last.

Per-step gates:

1. (ii) base case PASS for every metric at ±1.0pp tolerance, full coverage, before any analysis is published from that metric. NYC's bar.
2. (iii-a) SPREE spot check PASS at ±1.5pp on a 300-school stratified sample per metric. NYC's bar.
3. (iii-b) Future Ready PA Index PASS at ±1.5pp on a 300-school stratified sample per metric. New for Philly; sets a higher quality bar than NYC.
4. (iv) computed values: deferred to a follow-up PR, same as NYC.

Acct vs Actual — the population-cut gap NYC had, Philly publishes both

KX's verify/METHODOLOGY.md documents an important NYC reconciliation hazard: NYC's published Snapshot reports an accountability cohort (Oct-31-register-attributed) while the InfoHub bulk file the NYC loader ingests reports all tested students. The two are different population cuts of the same data, with N differences in either direction. The (iii) spot check against the Snapshot has a wide tail (NYC absenteeism agreement ~67% at ±1.5pp; PSSA-style proficiency ~62-73%) — not because the data is wrong, but because the population definitions diverge and the attribution rules aren't fully documented at the cell level.

Philly has the same gap, but publishes both cuts natively (per findings.md §F.6 and 00_overview.md §O.8):

• PSSA_Keystone_Schools_Acct_Suppressed.csv — accountability cohort
• PSSA_Keystone_Schools_Actual_Suppressed.csv — all tested students

This doesn't eliminate the gap; it makes it observable and reconcilable at ingest time rather than something that has to be reverse-engineered from validation reports. The Philly posture:

1. Canonical metric value comes from the Acct file. Matches SDP's own SPREE composite, matches public-facing school reports, matches what families see.
2. <metric_key>_actual is co-loaded as a parallel metric for every PSSA + Keystone subject × grade × subgroup row. Same DB grain.
3. An additional reconciliation gate runs at ingest time: a per-school diff of Acct vs Actual, written to docs/qa_reports/<metric>__acct_vs_actual.md. This is structurally analogous to the (ii) base case (it compares two cells against each other) but conceptually new — a population-cut report rather than a re-read fidelity report.
4. Reporting on the gap is honest, not buried. The story-page recipes and the methodology page surface the Acct/Actual distinction prominently. Where Acct and Actual differ meaningfully for a story's featured school, both numbers appear with the population each represents.

NYC has the same conceptual gap but cannot ship the Acct cut without first reverse-engineering the attribution rules. The Philly build's infrastructure for #3 could, in principle, be back-ported to a future NYC deep-dive that reconstructs an Acct cohort from NYC's published register files — but that's a separate NYC follow-up project, not a Philly deliverable.

Authority

This document is authoritative for Philly verification expectations. Where it diverges from KX's NYC spec, the divergence is intentional and documented in the sections above.