# 05 — Validation Spec (Philly)

Adaptation of KX's NYC validation spec (`docs/design/05_validation_spec.md` +
the as-built `verify/METHODOLOGY.md`) to Philly sources. Same four checks,
same posture, different publishers and source files.

---

## The four checks (terms used throughout)

Identical to KX's framing:

- **(i) Completeness** — are years or schools missing? Surfaced as "source-only" rows (in the source file but not in the served DB).
- **(ii) Correctness — base case** — does the DB match an independent re-read of the **exact source file the loader ingested**? *(the binding correctness gate)*
- **(iii) Correctness — spot check** — does the DB match the District's **public-facing per-school report** (SPREE for Philly; NYC School Snapshot for NYC)? *(presentation cross-reference, not an independent oracle)*
- **(iv) Correctness — computed values** — are derived metrics (composites, survey rollups) computed correctly from validated inputs? *(deferred)*

## Sources by check

| Check | NYC source | Philly source |
|---|---|---|
| (i) and (ii) — ingest baseline | NYC DOE InfoHub xlsx files | **PDE bulk xlsx + OpenDataPhilly Socrata + Future Ready PA Index xlsx** |
| (iii) — spot check | NYC School Snapshot per-school JSON API | **SPREE per-school xlsx + School Profile dashboard JSON** (philadelphiaschools.org/research) |
| Independent-publisher reference (deferred in NYC) | NYSED bulk file | **Future Ready PA Index** (PDE-published; independent of SDP). Available day one — meaningful improvement over NYC. |

## Posture for Philly's two-publisher fact

Philly is unusual in having **two independent authoritative publishers** for
the same metric family:

- **PDE** publishes school-level PSSA + Keystone + cohort grad files. This is
  the *ingest source of truth* (the file the loader actually reads).
- **SDP** publishes its own copies plus the SPREE composite. This is the
  *presentation reference* (what families see).
- **Future Ready PA Index** publishes PDE's accountability bundle. This is
  the *cross-publisher independent oracle* — analogous to KX's deferred NYSED
  check, but available day one.

So Philly's (iii) spot check has two flavors:

- **(iii-a) DB vs SPREE** — same-publisher presentation cross-reference.
  Mirrors NYC's Snapshot check. Caveats from `verify/METHODOLOGY.md` apply
  (it's a presentation check, not an oracle).
- **(iii-b) DB vs Future Ready PA Index** — different-publisher independent
  check. The "deferred NYSED" gap NYC has does not exist here.

Both should ship. (iii-b) is the stronger check; (iii-a) is the one families
will recognize.

## Per-metric expectations

For each v1 metric, the table records the (ii) base case source + the (iii-a)
+ (iii-b) references. Tolerances match KX's defaults (±1.0pp for (ii),
±1.5pp for (iii)) until Philly-specific tuning is needed.

| Metric (provisional name) | (ii) source | (iii-a) SPREE varname | (iii-b) Future Ready varname | Tolerance |
|---|---|---|---|---|
| `pssa_grade3_ela_proficiency` | PDE PSSA results xlsx (sheet `ELA Grade 3`) | TBD — first-fetch | TBD — first-fetch | ±1.0 / ±1.5 |
| `pssa_grade3_math_proficiency` | PDE PSSA results xlsx (sheet `Math Grade 3`) | TBD | TBD | ±1.0 / ±1.5 |
| `pssa_grade7_ela_proficiency` | PDE PSSA results xlsx (sheet `ELA Grade 7`) | TBD | TBD | ±1.0 / ±1.5 |
| `pssa_grade7_math_proficiency` | PDE PSSA results xlsx (sheet `Math Grade 7`) | TBD | TBD | ±1.0 / ±1.5 |
| `pssa_all_ela_proficiency` | PDE PSSA results xlsx (all-grades aggregate) | TBD | TBD | ±1.0 / ±1.5 |
| `pssa_all_math_proficiency` | PDE PSSA results xlsx (all-grades aggregate) | TBD | TBD | ±1.0 / ±1.5 |
| `keystone_algebra_proficiency` | PDE Keystone results xlsx | TBD | TBD | ±1.0 / ±1.5 |
| `keystone_literature_proficiency` | PDE Keystone results xlsx | TBD | TBD | ±1.0 / ±1.5 |
| `keystone_biology_proficiency` | PDE Keystone results xlsx | TBD | TBD | ±1.0 / ±1.5 |
| `graduation_rate_4yr` | Future Ready PA Index xlsx | TBD | (self) | ±1.0 / ±1.5 |
| `chronic_absenteeism_rate` | Future Ready PA Index xlsx | TBD | (self) | ±1.0 / ±1.5 |
| `survey_*_*` (PSES rollups) | SDP PSES Zip — per-respondent CSV | n/a — derived | n/a | ±0.0 (exact, from raw) |

`TBD` rows resolve when the source files are first fetched (P1) and the
variable-name maps are written into `verify-philly/snapshot.yaml`.

## Known reconciliation hazards (lessons from NYC)

From `verify/METHODOLOGY.md`:

- **Different population cuts of the same data.** NYC's published Snapshot
  uses an *accountability cohort* (Oct-31-register-attributed) while the
  ingested InfoHub file uses *all tested students*. Differences in either
  direction are expected. **Philly may have the same hazard:** SPREE uses
  PDE's accountability cohort; the bulk PSSA file uses all tested. Document
  whichever rule SDP uses in SPREE's Educator Guide and reconcile per-cell
  with the rule, not the headline.
- **N differences with both directions.** NYC's N can be larger or smaller
  than the source by year; same will be true in Philly. Don't assume "served
  is a subset of source."
- **Test-recalibration discontinuities.** PSSA was recalibrated 2014-15 (PA
  Core) and 2022-23 (post-COVID). Across-regime trend comparisons need the
  comparability config from `pipeline/analysis/comparability.yaml`. (i) and
  (ii) checks are per-year and unaffected; (iii) checks against SPREE that
  also report standardized scores may diverge across the regime boundary.

## What lands first

The KX precedent: **(ii) base case ships first, full coverage, all metrics**;
then (iii-a) SPREE spot check on a stratified sample; then (iii-b) Future
Ready PA Index cross-publisher check; (iv) computed values last.

Per-step gates:

1. (ii) base case **PASS** for every metric at ±1.0pp tolerance, full coverage,
   before any analysis is published from that metric. NYC's bar.
2. (iii-a) SPREE spot check **PASS** at ±1.5pp on a 300-school stratified
   sample per metric. NYC's bar.
3. (iii-b) Future Ready PA Index **PASS** at ±1.5pp on a 300-school stratified
   sample per metric. New for Philly; sets a higher quality bar than NYC.
4. (iv) computed values: deferred to a follow-up PR, same as NYC.

## Acct vs Actual — the population-cut gap NYC had, Philly publishes both

KX's `verify/METHODOLOGY.md` documents an important NYC reconciliation
hazard: NYC's published Snapshot reports an **accountability cohort**
(Oct-31-register-attributed) while the InfoHub bulk file the NYC loader
ingests reports **all tested students**. The two are different population
cuts of the same data, with N differences in either direction. The (iii)
spot check against the Snapshot has a wide tail (NYC absenteeism agreement
~67% at ±1.5pp; PSSA-style proficiency ~62-73%) — *not* because the data
is wrong, but because the population definitions diverge and the
attribution rules aren't fully documented at the cell level.

**Philly has the same gap**, but publishes both cuts natively (per
`findings.md §F.6` and `00_overview.md §O.8`):

- `PSSA_Keystone_Schools_Acct_Suppressed.csv` — accountability cohort
- `PSSA_Keystone_Schools_Actual_Suppressed.csv` — all tested students

This doesn't eliminate the gap; it makes it **observable and reconcilable
at ingest time** rather than something that has to be reverse-engineered
from validation reports. The Philly posture:

1. **Canonical metric value comes from the Acct file.** Matches SDP's own
   SPREE composite, matches public-facing school reports, matches what
   families see.
2. **`<metric_key>_actual` is co-loaded as a parallel metric** for every
   PSSA + Keystone subject × grade × subgroup row. Same DB grain.
3. **An additional reconciliation gate** runs at ingest time: a per-school
   diff of Acct vs Actual, written to
   `docs/qa_reports/<metric>__acct_vs_actual.md`. This is structurally
   analogous to the (ii) base case (it compares two cells against each
   other) but conceptually new — a population-cut report rather than a
   re-read fidelity report.
4. **Reporting on the gap is honest, not buried.** The story-page recipes
   and the methodology page surface the Acct/Actual distinction
   prominently. Where Acct and Actual differ meaningfully for a story's
   featured school, both numbers appear with the population each
   represents.

NYC has the same conceptual gap but cannot ship the Acct cut without first
reverse-engineering the attribution rules. The Philly build's
infrastructure for #3 could, in principle, be back-ported to a future NYC
deep-dive that reconstructs an Acct cohort from NYC's published register
files — but that's a separate NYC follow-up project, not a Philly
deliverable.

## Authority

This document is **authoritative for Philly verification expectations**.
Where it diverges from KX's NYC spec, the divergence is intentional and
documented in the sections above.