# Metric alignment: measuring absenteeism/attendance for NYC schools

*Phase 0 deliverable of the [absenteeism deep dive](06_absenteeism_deep_dive_plan.md) — kx-jin, 2026-06-09. Under review with the Chalkbeat NY team; sources cited inline.*

## 1. Question

Chalkbeat NY wants to use per-school attendance data to explain post-pandemic absenteeism: overall trends, how much schools vary, and which schools over/under-perform socioeconomically similar peers. Before building anything we need to align on **which metric(s)** to measure, verify the candidate sources **agree with each other**, and fix the **analysis window**.

## 2. Candidate metrics

| Metric | Definition | Best source | Verdict |
|---|---|---|---|
| **Chronic absenteeism rate** | % of students absent ≥10% of their enrolled days | NYC DOE InfoHub End-of-Year file | **Recommend: headline** |
| **Average daily attendance (ADA)** | days present ÷ days on register | Same file (`% Attendance` column) | **Recommend: companion** |
| % students with >90% attendance | SQR-scored attendance metric | SQR PDFs only — not in any bulk file | Not usable at scale |
| Severity tiers (moderate 10–19% / severe ≥20% absence) | Attendance Works framework | Not published per school by NYC | Not available |

### Why chronic absenteeism leads

1. **It's the official measure, everywhere.** NYC DOE adopted NYSED's definition in 2020-21 and applied it retroactively to the whole published series: a student is chronically absent if attendance ≤90% of enrolled days, counted only if enrolled ≥10 days and present ≥1 day (source NOTES sheet of the InfoHub file; same definition in NYC Open Data dataset `sgsi-66kk`). The Mayor's Management Report, NYSED report cards, IBO, the State Comptroller, and Chalkbeat's own coverage all report this measure.
2. **It's the sensitive instrument.** Citywide, chronic absenteeism moved 26.5% → 40.3% → 33.3% (2018-19 → 2021-22 → 2024-25) while ADA moved only 91.5% → 88.1% → 89.8%. ADA averages away exactly the concentration that matters: a school where most students miss a little looks like a school where some students miss a lot.
3. **It's already ingested and validated.** 7 years × ~1,520 schools × 16 subgroups in the warehouse, with denominators; base-case validation vs the source file is exact (10,639 cells, 0 mismatches — `docs/qa_reports/chronic_absenteeism_rate__validation.md`).

### Why ADA still matters as a companion

It's the MMR's other indicator, it's in the same source file (not yet ingested — Phase 1), and divergence between the two is itself diagnostic (rising ADA + flat chronic absenteeism ⇒ improvement concentrated among students who were already mostly attending).

## 3. Do the sources agree?

**Citywide, % chronically absent (All Students, All Grades):**

| School year | InfoHub file (ours) | MMR (FY) | School-level unweighted mean* |
|---|---|---|---|
| 2018-19 | 26.5 | — | 29.8 |
| 2019-20† | 25.1 | — | 28.2 |
| 2020-21† | 29.7 | 29.7 | 31.8 |
| 2021-22 | 40.3 | **40.7** | 44.0 |
| 2022-23 | 36.2 | 36.2 | 39.5 |
| 2023-24 | 34.8 | 34.8 | 38.0 |
| 2024-25 | 33.3 | 33.3 | 36.6 |

*\* mean over ~1,520 school-level rates, computed from our warehouse (`data/stories/deepdive-data.json`). † 2019-20 truncated at 3/13/20; 2020-21 includes remote attendance.*

- **InfoHub = MMR** exactly for FY23–FY25; one small discrepancy in FY22 (40.3 vs 40.7). Cause not established; MMR p.301 notes its own FY22 revision (40.6→40.7). Both round to Chalkbeat's reported "41%". Not material for school-level work; flagged for the validation backlog.
- **Citywide vs school-mean differ by ~3pp by construction** (student-weighted vs school-weighted). Both are correct; published work mixes them silently. We will always label which one we're showing.
- **School Quality Snapshot disagrees at school level**: only 66.5% of sampled school-years match within ±1.5pp; signed gaps −12 to +7.7pp including at large N (`docs/qa_reports/chronic_absenteeism_rate__snapshot.md`). Working hypothesis (not established): the Snapshot uses NYC's accountability cohort (Oct-31 audited register) rather than the all-enrolled InfoHub population. **Implication: the InfoHub file is our ground truth; the Snapshot is a labeled secondary reference.** Resolving the gap definitively is Phase 2 work (Educator Guide denominator; the 2024+ Snapshot's renamed varnames).
- **NYSED publishes the same definition independently** (per-school, through 2024-25 incl. charters; Access-format Report Card Database, tables `ACC EM/HS Chronic Absenteeism`) — NYC adopted NYSED's definition, so values should reconcile up to enrollment-window details (NYSED reports grades 1–8 and 9–12 separately; min-N 30, dropped to 20 in 2024-25; suppression marker `s`). This is the one truly independent (different-publisher) check; BEDS↔DBN is deterministic for district/D75 schools and needs a crosswalk only for charters.
- **NYC Open Data** mirrors of the same DOE series (`sgsi-66kk`; school-level `gqq2-hgxd` ends 2020-21) — same publisher, useful as a format check only.

## 4. Recommendation

1. **Headline metric: chronic absenteeism rate** (NYSED definition), school-level, from the InfoHub End-of-Year file — already ingested as `chronic_absenteeism_rate` (LOWER_BETTER).
2. **Companion: ADA** (`% Attendance` from the same file) — ingest in Phase 1; report alongside, never as the headline.
3. **Grade-level chronic absenteeism** (PK–12 rows exist in the file, currently unloaded) — ingest in Phase 1. Citywide the pattern is U-shaped (K 37.8%, grade 5 26.8%, grade 12 43.7% in 2024-25), so school-level grade mix matters for fair comparisons.
4. **Subgroups**: keep all published cuts, including the complements recovered in [PR #9](https://github.com/mwg1378/edu-data/pull/9) (Not Poverty / Not ELL / Not STH / Other / non-binary) so within-school gaps (e.g. STH 48.7% vs Not-STH 30.7% citywide) are computable.
5. **Universe: all NYC public schools, for every analysis year.** The InfoHub file covers **districts 1–32 + 75 only** — it is NYC DOE reporting on schools it operates from its own ATS system, so charters (~281 schools, ~148k students, >15% of public-school enrollment in 2024-25), D79, and home/hospital are out of scope *for that publisher*, not missing. Charter chronic absenteeism is published per school by **NYSED** (Report Card Database, data.nysed.gov): scouting confirmed school-level tables for **2017-18, 2018-19, 2020-21, and 2021-22 through 2024-25** — SRC2025 (released 2026-04-30) already carries 2024-25, so the earlier assumption of a one-year NYSED lag is moot. The only NYSED hole is 2019-20 (never published — COVID), which our window excludes anyway. Decision: ingest NYSED and run an **all-schools universe across the full window**; every chart/table still states its universe and per-school publisher. Settled by scouting: **D79 programs do not appear in NYSED data** → D79 stays excluded, stated in methodology. Known coverage wart: **Success Academy stopped reporting attendance to NYSED in 2023-24** (`DATA_REP_FLAG=N`, values suppressed in 2023-24 and 2024-25) — network scope to be quantified on ingest and disclosed wherever charters are shown.

## 4b. Grade coverage by source

| Source | Grade granularity | Grades covered | School-wide aggregate |
|---|---|---|---|
| NYC DOE End-of-Year file | true per-grade rows (PK, K, 1–12) × subgroup | PK–12 (PK only where a K-12 school offers it) | "All Grades" (also includes students missing a grade label) |
| NYSED Report Card Database | two bands only: grades 1–8 ("EM") and 9–12 ("HS") | 1–12 — **no PK/K** | none published (we derive grades 1–12 for charters) |
| NYC School Quality Snapshot | one school-wide number | whole school | that single number |

Implications: like-for-like cross-publisher comparison = our grades-1–8/9–12 aggregates vs NYSED's bands (possible only because the per-grade DOE rows are ingested); PK/K carry the highest absenteeism (48.2%/37.8% citywide 2024-25), so DOE school-wide mechanically exceeds a grades-1–12 basis at elementary schools — the documented basis caveat on charter headline values; NYSED also lacks gender and temporary-housing subgroups, so within-school STH gaps remain DOE-only.

## 5. Analysis window & comparability

- **Pre-pandemic baseline: 2018-19** (earliest year in the series).
- **Exclude from trend fits: 2019-20** (year ends 3/13/20) **and 2020-21** (remote attendance counted) — consistent with `docs/design/reference-scaffold/config/comparability.yaml`.
- **Trend window: 2021-22 → 2024-25** (4 clean post-pandemic years), with 2018-19 as the recovery yardstick. This matches the Detroit PEER study's window and Chalkbeat's framing.

## 6. Known caveats to carry through all phases

- Students attending multiple schools in a year count at **each** school (school-level rows don't sum to citywide).
- FERPA suppression: rows with ≤5 students or ≤900 total days; chronic-absence value additionally suppressed if ≤5 students contribute ≥20 days.
- D75 short-term-stay schools enter the file in 2022-23 (small series break).
- PK rows only for K-12 schools that offer PK (no NYCEEC/PreK centers).
- 148 school-years exist in the source but not the warehouse (stale school directory; re-ingest requested alongside PR #9).
- **NYSED (charter) data is not perfectly apples-to-apples with the DOE file**: same core definition (DOE adopted NYSED's), but NYSED accountability reporting uses grades 1–8 / 9–12 splits (DOE file includes PK/K), and the State Comptroller describes NYSED's measure as excluding out-of-school-suspension and extended-medical absences (not yet verified at the rule-book level). Reconcile and document before mixing publishers in one chart; the overlap on district schools (both publishers report them) is itself the validation instrument.
- **Crosswalk**: NYC charters carry district-84 DBNs in city directories; whether the `schools` table covers all ~281 charters is to be confirmed when wiring the NYSED ingest (same BEDS↔DBN work the independent check needs).

## 7. Open decisions (checkpoint with Chalkbeat)

1. Approve chronic absenteeism as headline + ADA companion + grade-level cut.
2. Approve the analysis window (2018-19 baseline; trends on 2021-22→2024-25; 2019-20/2020-21 excluded).
3. Confirm the **all-schools universe** (district via DOE + charter via NYSED, every analysis year incl. 2024-25; D79 not published anywhere → excluded with a methodology note); universe and per-school publisher labeled on every output.
4. Flag any additional cuts they want first-class (e.g. borough/district rollups, STH focus).