# Absenteeism deep dive — findings ledger

*Living registry of findings from the chronic-absenteeism deep dive (plan: [06](06_absenteeism_deep_dive_plan.md), metric memo: [07](07_absenteeism_metric_alignment.md)). One entry per finding, never deleted — status changes are edited in place with a date. Statuses: **verified** (established, evidence linked) · **fixed** (was a defect; resolution linked) · **open** (real, cause/resolution not established) · **ruled out** (investigated, not the case) · **preliminary** (analysis result pending validation/checkpoint).*

*Where things live: validation methodology = `verify/METHODOLOGY.md`; validation evidence = `docs/qa_reports/` (NYSED cross-publisher under `docs/qa_reports/nysed/`); analysis memos = `docs/analysis/absenteeism/`; this file = the index of what we know.*

## Data & ingestion

| # | Finding | Status |
|---|---|---|
| A.1 | The attendance loader silently dropped five source categories ("Not Poverty", "Not ELL", "Not STH", "Other", "Neither male nor female") via exact-string mismatch — ~39k school-year cells, including the comparison sides of the poverty/ELL/housing gaps. | **fixed** — PR #9 (2026-06-09) |
| A.2 | 148 attendance school-years exist in the source file but not the warehouse: DBNs missing from the `schools` directory (stale 2019-20 vintage); loaders skip unknown DBNs. Same root cause confirmed for the test-metric gaps. | **fixed (locally — pending prod re-ingest)** — 2026-06-11, branch `kx/bakeoff-prep`: directory seed now unions wg9x-4ke6 with the LIVE LCGMS export (nycenet.edu postback; newest tabular Socrata vintage is still 2019-20). +96 schools, 73 wg9x-only schools marked closed (sentinel date), grade-span changes logged not silent. Re-ingest on the dev replica: **141/148 source-only school-years resolve** (residual: 15K418 ×7 years — the D15 code of dual-coded inclusion school 75K372, which reports separately; 15K418 is absent from BOTH directory vintages) and **33/34 unmatched charter BEDS now match** (residual 332200861039 SA Sheepshead: NYSED county prefix 3322 vs LCGMS 3313 and name variant; its only year, 2024-25, is SA non-reporting → suppressed anyway, A.13). District-invariant verified: 0 previously-served metric rows changed or lost, +15,575 added. |
| A.3 | The source file carries per-grade rows (PK–12) and average-daily-attendance columns that were never ingested. Grade-level All-Students cells are only 3.9% suppressed (per-school grade trajectories and pseudo-cohorts viable); grade×subgroup runs 23–50% suppressed. | **fixed** — PR #12 ingests all of it (~2.25M cells) |
| A.4 | `pct_econ_dis` in `school_year_demographics` is top-coded at 95% and "Above 95%" is stored as **NULL** for 8–18% of schools per year — and those schools have *higher* absenteeism, so dropping them biases any SES adjustment. The demographics loader also hard-codes `pct_temp_housing` to null, and no Economic Need Index is ingested. | **fixed (locally — pending prod re-ingest)** — 2026-06-11, branch `kx/bakeoff-prep`: (1) the exact source marker is the string `Above 95%`; now stored as **0.95 with new flag `pct_econ_dis_topcoded`** — 0 NULL `pct_econ_dis` remain; per-year topcoded 2020-21→2024-25: 172/145/201/334/316. (2) The file DOES carry an **Economic Need Index** column (same `Above 95%` top-code, 934 rows) — ingested as new `eni` column, populated on every row. (3) `pct_temp_housing` now derived per school from the attendance file's STH sheet by new `sth-shares.ts` (share = STH_N/(STH_N+NotSTH_N), attendance-contributing basis, both cells non-suppressed): 7,405 existing rows filled; full 7-year series (2018-19→2024-25, wider than the demographics file) committed at `data/analysis/sth-shares.json`; citywide weighted share 10.68% in 2018-19 (≈1 in 9, matches published) → 15.35% in 2024-25. Schema changes additive, pushed to the LOCAL replica only. |

## Validation — base case (ii) and presentation (iii)

| # | Finding | Status |
|---|---|---|
| A.5 | Base case is **exact** for `chronic_absenteeism_rate` (10,639 cells, 0 mismatches) and now for `average_daily_attendance` (10,639 cells, 0 mismatches) + per-grade components. | **verified** — `docs/qa_reports/*__validation.md`, PR #13 |
| A.6 | **The Snapshot's chronic-absenteeism figure is the same-year GRADE-BAND rate, not the school-wide rate** — grades K–8 (kindergarten-inclusive) on the EMS report, grades 9–12 on the HS report — **and a band-spanning school has BOTH report pages, each carrying its own band's figure** (2026-06-10: both report types fetched for all 201 band-spanning schools; the bulk-fetch artifact is now per report type). Established by the exhaustive census (6,570 district school-year pairs, SY 2018-19 → 2022-23; replaces the 300-school sample): school-wide agreement within ±1.5pp is 0.6534 (the sample's "~67%" confirmed; full tail −26.84 to +15.83pp); against the matching band, type-matched, it rises to **0.8695 pooled (6,964 report-band pairs), 0.9261 EMS-report, ≥0.94 for EMS report-years 2020–2023**. The entire worst-15 list is band-explained, including the single worst case (28Q157 SY 2021-22, −26.84pp school-wide: its HS page's 54% is its 35-student grade-9 rate, 19/35, exactly — while its EMS page's 26% matches its grades-6–8 rate, 26.56%). Supersedes the "suspected accountability-cohort denominator" for district schools. **Residual not established — and unchanged by type-matching (2026-06-10):** HS-report values in SY 2019-20 and 2021-22 (band agreement 0.4834/0.4974 with a one-sided +1.5pp median shift, served above Snapshot) and both report types in SY 2018-19 (0.80–0.83). | **open (largely explained)** — census + type-matched re-census 2026-06-10, `docs/qa_reports/snapshot/` (CENSUS.md §10); residual = HS SY 2019-20/2021-22 + SY 2018-19, persists under type-matching |
| A.7 | Snapshot report-years 2024/2025 carry chronic absenteeism under **renamed varnames** (`val_chronic_absent_ems/hs_all`, with peer averages `pavg_*`). **Decoded (2026-06-10): the value is the COMPLEMENT — `round(100 − same-report-year grade-band chronic-absenteeism rate)`** (EMS: grades K–8, K-inclusive; HS: grades 9–12) — i.e., the share NOT chronically absent. Decoded agreement at ±1.5pp: 0.9710/0.9740 (ry2024 EMS/HS) and 0.9665/0.9814 (ry2025); every raw-value candidate maxes out at 0.042, so the earlier 4-school "matches neither adjacent InfoHub year" result was an artifact of comparing the raw value instead of its complement. The series changed sign convention in 2024, not population or period. Human review (2026-06-10) found the rendered Snapshot page does not visibly display chronic absenteeism for 2024-25 — the field lives in the data feed; **what label the page attaches to it remains unestablished** (the one open thread). | **verified (complement decoded)** — 2026-06-10, `docs/qa_reports/snapshot/CENSUS.md` §5; open thread: page-label semantics |

| A.19 | **Charters DO appear in the NYC Snapshot tool** with school-specific chronic-absenteeism values and comparison-group averages — despite being absent from the End-of-Year file (zero charter rows, full-file scan). **Settled at full scale (2026-06-10):** the values are **computed, not copied** — only 17.5–19.8% match whole-percent roundings of NYSED's published band rates (vs ~100% expected for republication), values exist in 39 school-years where NYSED's public file suppresses, and the enrollment basis is K-inclusive at scale (95.8% of K-serving charters show an enrollment surplus over NYSED's grades-1–8 band, median +9.8% ≈ one grade; non-K charters −2.4%). They track NYSED only loosely (median \|diff\| 1.79pp on the matched band, r 0.73–0.83 every year, vs district r 0.99 — why not established). Reporting outages co-move perfectly with the state chain: **82/82 `DATA_REP_FLAG=N` charter school-years (2021–2025) have no Snapshot CA value — zero counterexamples, ≥5 networks** (all 31 SA schools wink out exactly when SIRS reporting stops, the Snapshot report itself still published). **Practical rating: PARTIALLY independent** — can catch gross serving errors (independently computed, r ~0.7–0.8) and coverage gaps; cannot catch failures originating in the state attendance-reporting chain (it goes dark in lockstep); cannot arbitrate differences under ~2pp. Not established: the exact computation basis (denominator rules, thresholds, as-of date); whether DOE receives charter attendance through SIRS itself or a parallel submission failing simultaneously; why charter correlation is so much weaker than district. | **verified** — 2026-06-10, `docs/qa_reports/snapshot/PROVENANCE.md`; unestablished sub-points named in the verdict |

## Validation — cross-publisher (iii-b), NYSED

| # | Finding | Status |
|---|---|---|
| A.8 | NYSED publishes school-level chronic absenteeism for **2017-18, 2018-19, 2020-21, 2021-22 → 2024-25** (SRC2025 released 2026-04-30), including NYC charters — so an all-schools universe holds for every analysis year. 2019-20 was never published (COVID). Format: Access-only Report Card Database zips; tables `ACC EM/HS Chronic Absenteeism`. | **verified** — `docs/qa_reports/nysed/FORMAT_NOTES.md` |
| A.9 | BEDS→DBN is **deterministic for district schools** (county→borough letter + CSD + last-3-of-building-number; zero collisions across all panels). Charter BEDS codes do not encode the DBN → crosswalk needed (NYC OD `wg9x-4ke6` + current LCGMS). | **verified** — `docs/qa_reports/nysed/RECONCILIATION.md` |
| A.10 | **Ten of twelve (year × grade-band) panels reconcile across publishers at the median** — all comparable years (2018-19, 2020-21 → 2024-25 × grades 1–8 / 9–12; 2019-20 is not comparable: NYSED never published it and the DOE year is truncated; 2017-18 is NYSED-only): median per-school diff ≤0.4pp in the ten; r 0.987–0.997 in every grades-1–8 panel except 2024-25. Still the strongest external validation of the attendance data. The two failures: **2024-25 EM** (median −1.84pp → A.11) and **2022-23 HS** (median −0.60pp, only 47% within ±1pp, r 0.779 → A.20). Median agreement hides tails: the 2020-21/2021-22 HS panels pass at the median yet carry the same one-sided tail (A.20; weighted aggregates −2.1/−3.4pp, r 0.94/0.92), and 2020-21 carries the remote-attendance caveat. NYSED-only HS entities are transfer/alternative schools absent from the DOE file's school-level rows. | **verified** — `docs/qa_reports/nysed/RECONCILIATION.md` (expanded to 12 panels 2026-06-10, §9) |
| A.11 | **2024-25 grades 1–8 diverges systematically**: DOE shows EM chronic absenteeism falling (30.5→29.0% weighted) while NYSED shows it rising (30.0→30.8%) on the same ~1,115 schools; numerator-side; broad-based; not our aggregation; not a NYSED revision. Document review rules out any documented 2024-25 definition change on either side, and NYSED's standing suspension/medical exclusion predicts the **opposite** sign. One documented candidate remains: SIRS defaults missing/unpaired attendance records to "absent, unexcused". **Implication: 2024-25 EM CA is publisher-dependent — carry source attribution.** The expanded panels (2026-06-10) show the break has **no precursor**: EM medians in time order +0.39 → −0.13 → +0.09 → +0.13 → +0.38 → −1.84pp — a clean single-year break, broad-based (470 of ~1,115 joined schools below −2pp vs 51–61 in the COVID-era years), different in shape from the tail-concentrated A.20 phenomenon. | **open** — `docs/qa_reports/nysed/NARROWING.md`; RECONCILIATION.md §9; NYSED inquiry drafted |
| A.12 | D75 and D79 have **zero rows** in NYSED's CA tables in any year — NYSED cannot extend coverage there (D75 stays DOE-only; D79 has no public school-level data from either publisher). | **verified** |
| A.13 | **Success Academy stopped reporting attendance to NYSED network-wide in 2023-24** (`DATA_REP_FLAG=N`; 35/35 EM + 3/3 HS schools in 2024-25; zero flagged in 2018-19/2021-22). Charter-universe analyses must disclose this hole. | **verified** |
| A.20 | **COVID-era HS tail: a large one-sided population of high schools sits far below NYSED in 2020-21 → 2022-23.** 62 / 112 / 129 of ~414 joined high schools are >5pp below NYSED in 2020-21 / 2021-22 / 2022-23 (vs 0 / 0 / 3 in 2018-19 / 2023-24 / 2024-25), largely on byte-identical denominators; 86 schools persist across 2021-22 → 2022-23; extremes reach NYSED 99.8% vs DOE 15.3% on identical N. Verified publisher-level, not a reconstruction artifact: DOE's own published school totals match the per-grade reconstruction exactly at tail schools. Median-based panel summaries hide this tail (the 2020-21/2021-22 HS panels pass at the median; 2022-23 HS fails — median −0.60pp, r 0.779). **Cause not established.** Charter implication: charter values are served from NYSED, so charter HS chronic absenteeism for 2020-21 → 2022-23 inherits whatever affected NYSED's HS data in those years — unverifiable directly for charters (no second publisher). | **open** — `docs/qa_reports/nysed/RECONCILIATION.md` §9 (2026-06-10) |

## Analysis (preliminary — pre-Chalkbeat-checkpoint; see `docs/analysis/absenteeism/01_descriptives.md`)

| # | Finding | Status |
|---|---|---|
| A.14 | Recovery is real but decelerating (weighted CA 25.8% → 39.5% → 32.3%; −4.4/−1.3/−1.5pp) and **concentrated in older grades**: grade 12 is the only grade back to pre-pandemic; grades 1–8 remain +7–9pp. (2024-25 portion carries the A.11 asterisk for grades 1–8.) | **preliminary** |
| A.15 | School-level CA is mostly signal (YoY r 0.89–0.94); 58 schools worst-decile and 83 best-decile in all four post-COVID years; small schools ~2× as volatile; classic regression-to-the-mean at the extremes. Outlier claims need multi-year + size-aware gates. | **preliminary** |
| A.16 | Demographics explain ~46% of school-level variation (ES 57%, HS 42%); regression residuals are stable year-over-year (r 0.88); raw rates rank schools substantially by poverty (r 0.57) while residuals are poverty-orthogonal — the Detroit-PEER lesson reproduced on NYC data. Positive-residual tail clusters in small D2 high schools → school-type confound to control in the bake-off. | **preliminary** |
| A.17 | **The poverty gap is at its 7-year maximum** (18.7pp weighted vs 14.7 pre-pandemic; STH gap 19.5pp); recovery has been faster for advantaged students; the ELL gap flipped from ~0 to 5.5pp. Roughly ¼–⅓ of citywide gaps is between-school sorting. Uses the A.1-recovered complement subgroups — data that exists nowhere else publicly. | **preliminary** |
| A.21 | **Charter chronic absenteeism worsened in 2023-24 on a constant panel:** among the 183 charters with non-suppressed values in all of 2021-22 → 2024-25 (Success Academy excluded by construction — no state-reported attendance from 2023-24, A.13), weighted CA rose **38.7% → 43.2%** (2022-23 → 2023-24) before falling to 38.0% in 2024-25; 58.5% of 193 same-school year-over-year pairs worsened (median +1.3pp), so the rise is broad-based — **not** explained by the Success Academy exit, which mechanically raises the naive all-reporting series ~2pp (SA was 14.3% of 2022-23 charter students at weighted CA 24.7 vs 38.7 for other charters). NYSED-only basis: no second publisher exists to verify against (A.13/A.20 caveats apply). | **preliminary** — descriptives checkpoint packet (internal); lineage per `docs/analysis/absenteeism/01_descriptives.md` conventions; cite only after the packet lands and editorial review |
| A.22 | **Stability benchmark vs test scores, and pseudo-cohort vs grade-environment decomposition (rev 2026-06-11 packet additions):** (1) At school level, chronic absenteeism is **about as stable year-over-year as test proficiency, not more** — on identical school sets (schools with CA, ELA and Math All-Grades % Level 3+4 in both years; n = 1,024/1,038/1,046), consecutive-pair Pearson r runs CA 0.917/0.930/0.943 vs ELA 0.925/0.941/0.894 and Math 0.932/0.935/0.920 — CA is the most stable of the three only in the latest pair. These are raw levels, not value-added: the literature's attendance-beats-tests stability gap (Liu 2022: attendance VA r = 0.878 vs test VA 0.774) does not transfer to levels; the 2021-22→2022-23 pair spans the NextGen recalibration (flagged). Tests computed from the ingested InfoHub files (no test rows in the local replica). (2) **Pseudo-cohort decomposition** (per-grade CA, within school): the cohort correlation (grade g year y vs grade g+1 year y+1) exceeds the same-grade correlation (grade g both years) in **31 of 36 transition × year-pair cells**, by ~+0.04–0.08 at every transition from K→1 through 9→10 except the structural 8→9 (e.g. **g3 2022-23 → g4 2023-24: cohort r 0.862, n = 772, vs same-grade r 0.758**) — cohorts carry their attendance correlation into the next grade beyond what the school's grade environment alone shows; exceptions are late-HS (11→12 in all three pairs, one 10→11, one 8→9). Both correlations are high (≈0.74–0.89), so most persistence is school-level. Caveats: school-level pseudo-cohorts, not linked students; 5→6/8→9 cells cover only the minority of schools spanning the transition (n 80–141, churn 44–71%); school-wide CA r and the memo-01 g3→g4 cell reproduce exactly. | **preliminary** — working-analysis page rev 2026-06-11 + packet JSON (`stability_vs_tests`, `pseudo_cohort`); cite only after editorial review |

## Environment / tooling

| # | Finding | Status |
|---|---|---|
| A.18 | The stack could not run against a local/dev Postgres (hard-coded SSL in three pg pools; percentile scripts missing `DATABASE_URL` fallback; Supabase auth threw on every route when env unset). | **fixed** — PR #11 (local replica + dev recipe in PR body) |