# NYC School Quality Snapshot — chronic-absenteeism census & charter provenance

The full-scale **(iii) presentation cross-reference** for NYC chronic
absenteeism: the served values vs the NYC School Quality Snapshot
(tools.nycenet.edu), exhaustively, for every comparable school-year — replacing
the earlier 300-school sample (`docs/qa_reports/chronic_absenteeism_rate__snapshot.{md,json}`,
kept for the record). Plus a separate provenance study of the Snapshot's
**charter** values, which our site cannot compare like-for-like (charters are
served from NYSED).

Contents:
- `CENSUS.md` — the district census (6,570 school-year pairs, SY 2018-19 →
  2022-23) and the basis findings: the Snapshot displays the **same-year
  grade-band rate** (grades K–8, kindergarten-inclusive, on EMS reports;
  grades 9–12 on HS reports), **not** the school-wide figure; and the
  2024/2025 report-years' renamed feed varnames store the **complement**
  (`round(100 − band rate)`). Hand-verified end-to-end cases in §6; the
  charter census (descriptive only) in §7.
- `census-stats.json` — machine-readable panels (bucket shape compatible with
  `docs/qa_reports/nysed/stats.json`), candidate-interpretation tables,
  worst-case lists, verdicts.
- `PROVENANCE.md` — are the Snapshot's charter chronic-absenteeism values an
  independent series? Verdict: **computed, not copied** from NYSED's published
  rates, but reporting outages co-move with the state chain 82/82 —
  **partially independent** as a cross-check.
- `provenance-stats.json` — all numbers behind PROVENANCE.md.

Method & lineage: the Snapshot side of both studies is
`data/quality/snapshot-values.json` — a committed (~4 MB) bulk fetch of the
Snapshot data feed for every school × report years 2019–2025 **× report type**
(band-spanning schools have both an EMS and an HS page; both are fetched),
produced by `scripts/quality/fetch_snapshot_values.py` against the
tools.nycenet.edu API via the `verify/` connector (raw per-DBN API responses
cached under `data/reference/snapshot/{EMS,HS}/<reportYear>/`, gitignored).
The artifact is committed because it is the census's source of record and the
verification card reads it directly. The served side is the warehouse (SELECT-only); the
NYSED side of PROVENANCE.md is the committed `data/sources/nysed/SRC*_CA_nyc.csv`
extracts. The census/provenance computation scripts (`run_census.py`,
`augment_census.py`, `build_provenance.py`, referenced from the reports) live
in the analysis workspace, not this repo; the reports were written there and
copied verbatim.

Status: one-off studies ahead of harness integration. Follow-up (not done
here): `verify/snapshot.yaml` should re-point the chronic-absenteeism check at
the K–8 / 9–12 band aggregates (and decode the 2024/2025 complement varnames)
so the recurring (iii) check runs on the established basis.

Headline: ledger **A.6 largely explained** — school-wide agreement within
±1.5pp is 0.6534 (6,570 pairs; the sample's "~67%" confirmed), rising to
**0.8639 pooled / 0.9223 EMS-report (≥0.94 for EMS report-years 2020–2023)**
on the matching grade band; the entire worst-15 list, including 28Q157
(−26.84pp school-wide = its 35-student grade-9 rate exactly), is
band-explained. Residual NOT established: HS reports in SY 2019-20/2021-22
(~0.49, one-sided +1.5pp median shift) and SY 2018-19 (0.80–0.83 both report
types). **A.7 decoded** — `val_chronic_absent_{ems,hs}_all` =
`round(100 − same-year K–8 / 9–12 rate)`, 0.97–0.98 agreement (raw-value
candidates max out at 0.042); page-label semantics still unestablished.
**A.19 settled** — charter values are computed (17.5–19.8% rounding-match vs
~100% expected for republication; values exist where NYSED suppresses;
K-inclusive enrollment verified at scale), track NYSED only loosely (median
|diff| 1.79pp, r 0.73–0.83 vs district 0.99 — why not established), and
**82/82** `DATA_REP_FLAG=N` charter school-years have no Snapshot CA value:
**partially independent** — catches gross serving errors, cannot catch
state-chain reporting failures, cannot arbitrate <2pp.

Addendum (2026-06-10): `census-stats.json` carries two computed sections —
`band_basis_panels` (the established band-basis comparison as signed
distribution buckets + `worst10` lists, pooled / per-year / per report type)
and `decoded_2024_2025_panels` (same shape for the decoded 2024/2025
complement varnames). First built by `augment_band_distributions.py` on the
one-value-per-school-year artifact (`CENSUS.md` §9: 8 of the 10 pooled
extremes were dual-cached schools whose feed value matched their 9–12 rate —
tie-break artifacts), then **re-censused type-matched** by
`recensus_both_types.py` after the both-report-types fetch for the 201
band-spanning schools (`CENSUS.md` §10): every report-type value is compared
against its own band, spanning schools contribute both pairs. Pooled band
agreement 0.8639 → **0.8695** (n 6,964), EMS 0.9261, decode 0.97–0.98; the
tie-break artifacts are gone from the worst lists, and the residual pockets
(HS reports SY 2019-20/2021-22 ≈0.48–0.50, SY 2018-19 0.80–0.83) **persist
under type-matching** — real, still unexplained. These panels drive the
band-basis strips and the extreme-cases list on the metric page's open
issue (3).