Validation methodology — Philadelphia

Per-metric definitions, validation results, and known gaps. Mirrors verify/METHODOLOGY.md (NYC) in shape.

The four checks (terms used throughout)

Identical to KX's NYC framing:

• (i) Completeness — are years or schools missing? Surfaced as "source-only" / "served-only" cells.
• (ii) Correctness — base case — does the DB match an independent re-read of the exact source file the loader ingested? (the binding correctness gate)
• (iii) Correctness — spot check — does the DB match a public-facing per-school report? Two flavors for Philly:
  • (iii-a) DB vs SDP SPREE — same publisher, presentation cross-reference. Deferred until SPREE URL is discovered + ingested.
  • (iii-b) DB vs Future Ready PA Index — different publisher, independent oracle. Available day-one for Philly (the gap NYC had to defer).
• (iv) Correctness — computed values — derived metrics (composites, survey rollups). (deferred)

Plus the Acct vs Actual population-cut diff specific to Philly's O.8 decision — see __acct_vs_actual.md.

Validation results — 2026-06-04 snapshot

(i) Completeness — PASS

All 6 currently-validated components show clean coverage. The single source-only graduation row is a year-encoding edge case (a cohort year that falls outside our default reconciliation window). See __completeness.md for the full table.

(ii) Base case — PASS across all 6 components

Per-cell agreement at ±1.0 pp (±0.5 pp for graduation + attendance):

* matches less than common = suppressed cells (correctly excluded from the match count). No actual disagreements at any cell.

(iii-b) Cross-publisher cross-check — sanity-PASS

SDP PSSA Grades 3-8 (ELA) vs Future Ready Keystone Literature (HS ELA): 169 pairs, median Δ=0.03 pp, 94% within ±1 pp, 99% within ±5 pp.

NB: PSSA grades 3-8 and Keystone Literature HS measure ELA at different grade levels, so they're not expected to agree exactly — only to correlate. The tight agreement is consistent with both being honest ELA measures.

A tighter (iii-b) check would compare PSSA grade-by-grade against Future Ready's per-grade breakouts; the per-grade FRPI columns aren't loaded yet, so deferred to v2.

Acct vs Actual population-cut diff (O.8 gate) — TRIVIAL

Every one of 4,535 paired cells shows Δ = 0. At the school × subgroup rollup grain for 2024-25, SDP's Acct (accountability cohort) and Actual (all tested) cuts are byte-identical.

This is itself a substantive finding. The conceptual cohort-vs-tested distinction that drove NYC's reconciliation gap (per verify/METHODOLOGY.md — 67–73% snapshot agreement with wide tails) simply doesn't move the published number in Philly for 2024-25. Two interpretations to test in future years:

1. SDP's pipeline produces both files from the same intermediate — the "two cuts" naming is a vestige of a possibly-different reporting intent that never produced numerically distinct outputs at the school-aggregate grain.
2. The Acct/Actual difference is real at the student level but disappears at school aggregation because the in/out flips cancel.

Either way: for 2024-25 Philly, Acct = Actual at school-level rollup. The harness retains both load paths so that if 2025-26 or later shows a divergence (e.g. SDP changes attribution rules) it surfaces immediately.

Per-metric definitions

PSSA proficiency (all-grades + per-grade)

• Ours (loaded from SDP's PSSA Zip): % Proficient + Advanced = profadv_score column, per (school, year, testname, subject, grade, group). Ingested in both Acct and Actual cuts.
• (ii) base case: PASS at ±1.0 pp across 320 paired all-grades cells (ELA + Math), 150 grade-3 cells.
• Coverage: SDP district + alternative only (220+ schools). Charter PSSA aggregate values come from Future Ready (Future Ready aggregate column populated).

Keystone (Algebra / Literature / Biology)

• Ours (loaded from SDP's PSSA Zip and Future Ready Performance file): % Proficient + Advanced. Per O.5, latest-attempt all-students.
• (ii) base case: PASS at ±1.0 pp on Algebra (56 paired cells).
• Coverage: HS schools only.

PVAAS growth (Algebra / ELA-Literature / Science-Biology)

• Ours (loaded from Future Ready Performance file): growth-index value per (school, subject, subgroup).
• (ii) base case: not yet wired (deferred — PVAAS data structure makes per-cell reconciliation noisier; first add to v2).
• Coverage: all 298 Philly schools where reported by PDE.

Graduation rate (4-/5-/6-year)

• Ours: score column from SDP's grad CSV, indexed by cohort year (9th-grade start year). Future Ready charter-side covers report-year indexed values for the same metric_key (intentional — see loader notes).
• (ii) base case: PASS at ±0.5 pp on 4-year SDP cohort 2021-22 (81 paired cells).

Attendance ≥90% rate

• Ours (SDP attendance CSV): % with 90%+ Attendance (Yearly).
• (ii) base case: PASS at ±0.5 pp (216 paired cells, 2024-25).

Persistent attendance rate (Future Ready)

• Ours (Future Ready On Track Measures sheet): PercentPersistentAttendance (the metric PDE actually publishes; their PercentChronicAbsenteeism column exists but is NULL statewide in 2024-25).
• (ii) base case: not yet wired in components.yaml (the file structure makes the reconciliation trivial; pending one more iteration).

Out-of-school suspensions (derived: any / multiple / chronic)

• Ours (SDP suspensions CSV → derived per F.10): oss_pct_any = 100 − % zero; oss_pct_multiple = % 2 + % 3 + % 4+; oss_pct_chronic = % 4+ (residual).
• (ii) base case: not yet wired — these are computed metrics so v1 reconciliation against the raw distribution columns is straightforward but pending.
• (iv) computed-value check: needed once we add it.

PSES survey rollups (teacher + student × 6 domains)

• Ours (PSES topic+subtopic Zip): per-respondent topic-level score derived as mean of subtopic scores per (school, year, topic, respondent).
• (ii) base case: deferred — KX's NYC equivalent took several iterations; mirror that timing.
• Coverage: 207-298 schools per metric.

What's still TBD

• (iii-a) SPREE spot check — needs the SPREE URL discovered + an ingest pass; then a per-metric mapping (__snapshot.yaml equivalent) and a comparison runner.
• (ii) base case for PVAAS (attendance_persistence_rate — the state's Regular Attendance, stored by attendance year — is now covered by absenteeism_full_qa.py §B; OSS and PSES by __computed_value).
• (iv) computed-value checks for the OSS derivations and the PSES rollups.
• A "1-of-N anchor" set of hand-verified schools for each metric (KX's known_cases: pattern), so a known-good cell is always in the sample.

Authority

This document is authoritative for Philly verification results as of 2026-06-04, with the attendance checks superseded on 2026-09-07 by absenteeism_full_qa.py and the year-aligned cross_publisher_attendance.py (see the corrections log on /philly/methodology/metrics/chronic-absenteeism). Reports are regenerated by python pipeline_philly/verify/reconcile.py --all and the companion scripts. Where this diverges from KX's NYC framing, the divergence is intentional and documented in 05_validation_spec.md.