Snapshot-vs-served census — NYC chronic absenteeism (exhaustive; replaces the 300-school sample, ledger A.6)

Date: 2026-06-10. Computed by run_census.py + augment_census.py in this directory; machine-readable results in census-stats.json. Repo was read-only; database access was SELECT-only.

Sources and conventions

• Snapshot side: nyc-edu-data/data/quality/snapshot-values.json (bulk fetch of the tools.nycenet.edu NYC School Snapshot data feed; raw per-DBN API responses cached at nyc-edu-data/data/reference/snapshot/{EMS,HS}/<reportYear>/<DBN>.json). Values are whole percents. reportYear = school-year END (2023 ↔ SY 2022-23). JSON carries reportYears 2019–2025 for 2,131 DBNs.
• Served side: local Postgres edu_data_local, school_year_metrics with metric_key='chronic_absenteeism_rate', subgroup='ALL', NOT suppressed, joined to schools for school_type/grade_band/name (exports: served_schoolwide.csv, served_grades.csv).
• diff = served − snapshot (pp). Headline tolerance ±1.5pp (matches verify/snapshot.yaml and the old sample). Bucket convention (matches docs/qa_reports/nysed/stats.json panels): below_-5pp: d<−5 | -5_to_-2: −5≤d<−2 | -2_to_-1: −2≤d<−1 | within_±1: −1≤d≤1 | +1_to_+2: 1<d≤2 | +2_to_+5: 2<d≤5 | above_+5: d>5. share_within_Xpp = |d|≤X.
• Old sample being replaced: docs/qa_reports/chronic_absenteeism_rate__snapshot.{md,json} — n_sampled 300, checked 212, match 141, agreement 0.6651 at ±1.5pp.
• District = school_type != 'CHARTER' (DOE_PUBLIC + D75_SPECIAL_ED). Charters are a separate section (their served values are NYSED-derived; do not mix).

1. District census — old varname student_chronic_absent, SY 2018-19..2022-23

Served district rows in scope: 7,608 (7,329 DOE_PUBLIC + 279 D75). Compared: 6,570. Snapshot-missing: 1,038, all "dbn present in the feed but no entry for that reportYear" — 279 are D75 schools (zero D75 school-years have Snapshot entries, so the compared set is 100% DOE_PUBLIC) and 759 are DOE_PUBLIC school-years.

Pooled (n = 6,570)

Buckets (pp): below_-5pp 130 | -5_to_-2 187 | -2_to_-1 185 | within_±1 3,531 | +1_to_+2 1,184 | +2_to_+5 1,229 | above_+5 124.

Per year

Per-year buckets are in census-stats.json → district_census.per_year[<SY>].distribution.buckets_pp.

Worst 15 (pooled, by |diff|)

The 15 rows are 11 distinct schools, every one of them spanning a grade-band boundary or fetched under the HS report: seven serve grades 6–12, two serve 9–12, one serves 6–9 (28Q157, four of the 15 rows), one is K–12 (03M859) — exactly the population where a school-wide rate diverges from a band rate; see §3 (verified end-to-end for 28Q157 in §6).

2. Does the census confirm the old sample?

Yes on the headline; the sample understated the extremes and hid year-to-year spread.

• Pooled census agreement 0.6534 vs the sample's 0.6651 — within 1.2pp of each other; "~67%" is confirmed as "~65%" at 31× the sample's compared n (6,570 vs 212).
• Per-year, agreement ranges 0.5756 (2021-22) to 0.7824 (2020-21) — the sample pooled years and could not see this. 2020-21 is the outlier-good year; 2021-22 the worst.
• Tail: the sample's mismatch range (−12 to +7.7pp, per ledger A.6) is real but truncated. Census full range −26.84 to +15.83pp; p1/p99 = −7.09/+5.72; 130 school-years below −5pp and 124 above +5pp (3.9% of compared beyond ±5pp).
• The signed shape is the same: the distribution is right-shifted (served > snapshot; mean +0.73, weighted +0.58pp) with a thinner but longer negative tail.

3. Basis narrowing (new; replaces A.6's "suspected accountability-cohort denominator")

A.6 left the cause "not established." The census data establishes most of it: the Snapshot's student_chronic_absent tracks the grade-band rate, not the school-wide rate — grades K–8 (K-inclusive) for schools cached under the EMS report, grades 9–12 for HS-report schools, same school year. Report type per school-year was read from raw-cache file presence (data/reference/snapshot/EMS|HS/<reportYear>/; 16 of ~7,650 dbn-years are dual-cached, EMS preferred).

Replacing the school-wide served candidate with the served grade-band rate (reconstructed from non-suppressed chronic_absenteeism_rate_grade_* rows, denominator-weighted):

Year-alignment check (same vs prior vs next school year, per report type): same-year wins in every panel by a wide margin (e.g., EMS ry2021: same 0.9424 vs next 0.0446 vs prior 0.1261) — there is no year offset.

Established: the band basis explains the bulk of the A.6 disagreement, including the entire worst-15 list pattern (middle schools with a tiny 9th grade served the HS report get a grades-9-12 figure — §6 case 2 — and K-8 schools' school-wide rates differ from the Snapshot by their PK enrollment — §6 case 1). The EMS basis is K-inclusive: at the 2024/25 varnames where the test is sharpest, K-8 fits far better than 1-8 or PK-8 (§5).

Not established: the residual. Two HS panels — ry2020 (SY 2019-20, w±1.5 = 0.4845) and ry2022 (SY 2021-22, 0.4895) — carry a one-sided +1.5pp median level shift (served band rate above snapshot) that year alignment does not fix; and ry2019 (SY 2018-19) sits at only 0.80–0.83 for both report types. Whether this is a denominator rule (e.g., the Educator-Guide/accountability population), a different attendance cut for those years, or something else, is not determined by this census. This narrows A.6's open item to: "HS reports in SY 2019-20/2021-22 + both report types in SY 2018-19."

4. Comparison-group average (cavg_student_chronic_absent) — descriptive only

Not a validation. Distribution of school − comparison-group-average, district compared rows with a non-null cavg (note: the feed has zero non-null cavg values in reportYear 2021 / SY 2020-21):

Description only: the median school sits 3–6pp above its NYC "similar schools" average on both the Snapshot's own value and our served value, and only ~35–40% of schools sit below their comparison-group average; the spread is wide (sd ~14pp). No claim about how NYC constructs the groups is made here.

5. ReportYears 2024/2025 — renamed varnames (val_chronic_absent_ems_all / val_chronic_absent_hs_all), ledger A.7

A.7 said the values "match neither adjacent InfoHub year" (4-school check); basis unestablished. The exhaustive test confirms the raw values match nothing — and then resolves the basis.

Raw-value candidates all fail. Tested per variant × report year against served school-wide same-year/prior-year/next-year/two-prior and served band rates (26 candidate panels, n = 377–1,089 each): best share within ±1.5pp is 0.042; medians −5.8 to −32.1pp. Full table in census-stats.json → a7_2024_2025_candidates.

The complement reconciles. Interpreting the value as 100 − (chronic absenteeism rate):

(Remaining candidates: census-stats.json → a7_complement_interpretation.)

Verdict (established, arithmetic only): val_chronic_absent_ems_all = round(100 − same-report-year K-8 chronic-absenteeism rate) and val_chronic_absent_hs_all = round(100 − same-report-year 9-12 rate) for 97–98% of schools at ±1.5pp. The series did NOT change population or period in 2024 — it changed sign convention (the feed now stores the complement, i.e., the share NOT chronically absent). K-8 beats 1-8 and PK-8 decisively, consistent with A.19's K-inclusive observation. The earlier 4-school "matches neither year" result was an artifact of comparing the raw value instead of its complement. EMS/HS variants are band-specific by construction (K-8 vs 9-12 fits); no school in the bulk fetch carries both variants in the same reportYear (the fetch pulled one report type per school — 0 overlaps — so same-school EMS-vs-HS comparison is not possible from this feed). What label the rendered page attaches to this number is not established here (per A.7, the page does not visibly display it); pavg_chronic_absent_*_all was not validated against anything and is noted only as present and on the same complement scale (e.g., 01M184: val 91, pavg 89).

6. Hand verifications (end-to-end)

Case 1 — 01M015 (P.S. 015 Roberto Clemente), reportYear 2023 = SY 2022-23, EMS report, old varname. Raw DOE file (public-school-attendance-results-2019-2025.xlsx, All Students): All Grades contributing = 206, chronically absent = 104 → 104/206 = 50.4854%. Served DB row: value 50.48543549, denominator 206 — exact. K-8 portion (grades K,1–5; school serves no 6–8): contributing 37+34+28+33+25+36 = 193, CA 16+21+11+14+16+16 = 94 → 94/193 = 48.705%. PK row: 13 contributing, 10 CA. Snapshot cache EMS/2023/01M015.json: student_chronic_absent='49%'; snapshot-values.json: 49.0. Result: served − snapshot = +1.49pp (a "match" by 0.01); the K-8 rate rounds to 49 exactly; the gap is the PK rows in the school-wide rate.

Case 2 — 28Q157 (J.H.S. 157 Stephen A. Halsey), reportYear 2022 = SY 2021-22, HS report, old varname; the single worst diff in the census (−26.84pp). Raw file: All Grades contributing = 1,624, CA = 441 → 27.155% (= served DB value, denominator 1,624). Grade 9 only: contributing = 35, CA = 19 → 19/35 = 54.286%. Snapshot cache: no EMS/2022/28Q157.json exists; HS/2022/28Q157.json has student_chronic_absent='54%' (cavg 42%). Result: the snapshot value is the school's grades-9-12 figure computed on 35 students — the school was served the HS report, and the worst tail case is fully explained by the band basis, not by a data error on either side.

Case 3 — 01M184 (P.S. 184m Shuang Wen), reportYear 2024 = SY 2023-24, EMS report, renamed varname. Raw file: All Grades contributing = 746, CA = 72 → 9.651% (= served DB value, denominator 746). K-8 = All Grades minus PK (51 contributing, 11 CA): 695 contributing, 61 CA → 61/695 = 8.777%; 100 − 8.777 = 91.22. Snapshot cache EMS/2024/01M184.json: val_chronic_absent_ems_all='91%' (pavg 89%); snapshot-values.json: 91.0. Result: complement of the K-8 rate rounds to 91 exactly; the raw value 91 matches no CA rate of any nearby year, reproducing why the A.7 spot check failed.

7. CHARTERS — separate, descriptive only

Served charter values are NYSED-derived (grades 1–12); the Snapshot's charter values appear DOE-computed on a K-inclusive basis (A.19) — a basis mismatch on both grade range and publisher chain, and charter HS 2020-21→2022-23 inherits the A.20 NYSED HS anomaly. Provenance is a parallel workstream; nothing below is a provenance claim.

Served charter rows SY 2018-19..2022-23: 890 (no 2019-20 — NYSED never published it). Compared: 796; missing 94 (72 no reportYear entry, 22 null varname).

Pooled: w±0.5/1.0/1.5 = 0.157 / 0.279 / 0.3781 (301/796); mean +5.04, median +0.89, sd 14.27; p10/p90 −2.89/+17.34; min/max −87.80/+93.43; weighted aggregate served 32.02% vs snapshot 27.08% (+4.94pp); correlation 0.775. Buckets: below_-5pp 34 | -5_to_-2 86 | -2_to_-1 64 | within_±1 222 | +1_to_+2 79 | +2_to_+5 110 | above_+5 201.

The worst-15 list (census-stats.json → charter_census.worst_15) is dominated by served values of exactly or near 100.0 against snapshot values of 3–37 (e.g., 84K927 2022-23 served 100.0 vs snapshot 8; 84X419 2020-21 served 99.43 vs snapshot 6), plus one inverse case (84M433 2018-19 served 0.2 vs snapshot 88). These served extremes come from the NYSED side of the served pipeline and overlap the A.20 phenomenon; their cause is not established here.

8. Files

• census-stats.json — all panels (stats.json-compatible bucket shape), candidate-interpretation tables, worst-case lists, hand verifications, verdicts.
• run_census.py, augment_census.py — reproducible computation (read-only inputs).
• served_schoolwide.csv, served_grades.csv — SELECT-only exports used as inputs.

9. Addendum (2026-06-10): distribution panels on the band basis

Added by augment_band_distributions.py (same analysis workspace as the other scripts; inputs unchanged). Every summary share was asserted equal to the §3/§5 numbers before writing — pooled band 0.8639, EMS pooled 0.9223, decoded 0.9665–0.9814 all reproduce exactly; no recomputation drift.

Two new computed sections in census-stats.json, both in the same distribution-bucket shape as §1 and the docs/qa_reports/nysed/stats.json panels, each panel carrying a worst10 list (largest |diff|; our_rate = the served band rate, band_denominator = the band's contributing-student sum):

• band_basis_panels.panels — the §3 comparison (served grade-band rate − Snapshot, district schools, old varname, SY 2018-19..2022-23): pooled, per-SY, EMS_pooled/HS_pooled, and per-SY × report type (<SY>_EMS/<SY>_HS). Pooled buckets (pp): below_-5pp 16 | -5_to_-2 98 | -2_to_-1 225 | within_±1 5,043 | +1_to_+2 751 | +2_to_+5 418 | above_+5 18 (n 6,569) — versus the school-wide pooled buckets in §1 (130 | 187 | 185 | 3,531 | 1,184 | 1,229 | 124), i.e. the beyond-±5pp tail shrinks from 254 to 34 school-years on the correct basis.
• decoded_2024_2025_panels.panels — the §5 decode (served band rate − (100 − feed value)), reportYears 2024/2025 × EMS/HS, same school year. The worst10 rows are exactly the schools where even the complement decode fails; the largest residual anywhere is −6.27pp (13K301, 2023-24, K-8 47.73 vs decoded 54) and 3 of the 4 panels max out below ±5.1pp.

Dual-cache annotation (new, verified per row). Each worst10 row carries dual_cached (both EMS/<ry>/<dbn>.json and HS/<ry>/<dbn>.json exist in the raw cache — the §3 tie-break prefers EMS) and, where dual, alt_band_rate (the served rate of the other band). Result: 8 of the 10 pooled band-basis worst rows are dual-cached schools whose feed value matches their grades-9-12 rate to within 2.5pp (7 of 8 within ±1.5pp; e.g. 15K448 2021-22: compared K-8 35.77 vs snapshot 64, but 9-12 = 63.74) — for those dbn-years the bulk feed evidently stored the HS-report figure, so the top of the band-basis tail is largely a tie-break artifact, itself band-explained under the HS reading. The two non-dual extremes (09X260 2020-21, −15.02; 08X452 2020-21, +12.85; both HS-report 9-12 comparisons) remain genuinely unexplained and stay with the §3 residual. Which report a dual-cached school's page actually displays is not established here.

These panels feed the metric page's open issue (3) (band-basis strips + the linked extreme-cases list); §§1–8 above are unchanged.

10. Addendum (2026-06-10): both report types fetched for band-spanning schools; panels re-censused type-matched

§9 left the top of the band-basis tail as a tie-break artifact (the census compared ONE feed value per school-year, routed by cache presence with EMS preferred). Closed by completing the fetch: scripts/quality/fetch_snapshot_values.py moved to a per-report-type artifact schema ({dbn: {reportYear: {"EMS": …|null, "HS": …|null}}}; v1 → v2 migration verified value-preserving against the v1 artifact, 10,809 blocks, 0 drifts) and a targeted --spanning mode fetched the missing type for all 201 band-spanning schools (grade_levels_served has a grade ≤8 AND one of 09–12) × report years 2019–2025 × both types: 2,814/2,814 triples recorded, 0 fetch errors (live fetches distinguish errors — retried once, left unrecorded — from genuine no-report responses). Coverage: 780 of 1,407 spanning school-years carry values under BOTH types (110–114 per year); 322 of the 627 no-value-pair school-years are D75 (the feed has no D75 reports, §1), 14 ALT_PROGRAM, the rest EMS-only/HS-only or unopened years.

The dual-page reading of §9 is confirmed directly: each endpoint serves its own band's figure. 15K448 ry2022: EMS page 36 ↔ served K-8 35.77, HS page 64 ↔ 9-12 63.74. 28Q157 (HS-banded 6–9 school; only its HS endpoint had ever been fetched): EMS 26 ↔ K-8 26.56 and HS 54 ↔ 9-12 54.29 in SY 2021-22 — its −26.84pp §1 worst-case is now a pair of sub-0.6pp matches — and the same pattern holds for all its years.

band_basis_panels and decoded_2024_2025_panels were regenerated type-matched by recensus_both_types.py (analysis workspace): every report-type value a school's feed carries is compared against ITS band (EMS ↔ K-8 K-inclusive, 9–12 ↔ HS), so spanning schools contribute both pairs. Before replacement the script reproduced the superseded selection exactly from the v1 artifact + the pre-fetch cache map (every band_basis_test share incl. pooled 0.8639 / EMS 0.9223, and every a7_complement_interpretation decode share — hard asserts), and verified that 6,553 of the 6,569 old rows carry over diff-identical; the 16 replaced rows are exactly the in-scope dual-cached tie-break artifacts.

Results (old → new):

Pooled buckets (pp): below_-5pp 3 (was 16) | -5_to_-2 101 | -2_to_-1 243 | within_±1 5,380 | +1_to_+2 789 | +2_to_+5 430 | above_+5 18; mean +0.34, median +0.20, sd 1.08. The 8 dual-cache mismatches are gone from the pooled worst10, which is now headed by the two §9 genuine residuals (09X260 2020-21 −15.02; 08X452 2020-21 +12.85) followed by single-band gaps of ≤±9.3pp — the largest remaining decode failure is −6.27pp (13K301, unchanged).

Residual pockets: PERSIST — they are real, not tie-break artifacts. With correct type-matching, HS-report agreement is 0.4834 in SY 2019-20 (median +1.57) and 0.4974 in SY 2021-22 (median +1.48) (was 0.4845/0.4895), and SY 2018-19 stays at 0.8297 EMS / 0.7995 HS. The one-sided +1.5pp shift in those HS panels is a property of the published HS-report values for those years; its cause remains not established (A.6's open residual is unchanged in substance).