Find What Works: Schools
← All source documentsRaw file
Source document

data/sources/nysed/README.md

Served verbatim from the project repository. Internal working document conventions apply: documents may reference file paths, branch names, and findings-ledger anchors from the repo.

NYSED Report Card chronic absenteeism — committed NYC subset

NYC school-level rows from the NYSED Report Card Database (SRC) tables ACC EM Chronic Absenteeism (grades 1-8) and ACC HS Chronic Absenteeism (grades 9-12). Backs scripts/loaders/nysed-attendance.ts (charter-school chronic_absenteeism_rate) and scripts/loaders/dbn-beds-crosswalk.ts.

Why a committed extract (exception to the "no committed source bytes" rule)

The upstream SRC zips are ~300-385 MB each (~1.4 GB for the four years here) and contain only Access databases (.accdb/.mdb) — no CSV alternative exists. Reproducing the extract requires mdbtools (not a repo dependency). Committing the small NYC subset (~16 MB) with full hash lineage keeps loader runs reproducible for anyone without Access tooling. Flagged for review in the ingest PR; the full-state bytes stay gitignored as usual.

Lineage

  1. Download zips from https://data.nysed.gov/downloads.php (retrieved 2026-06-09; sha256 + bytes in provenance.json → parent_zips).
  2. Extract the .mdb from each zip; export both CA tables: mdb-export <SRC.mdb> "ACC EM Chronic Absenteeism" > SRC{YYYY}_EM_CA.csv (same for HS). Full-state CSVs live (gitignored) at data/reference/sources/nysed_src/2026-06-09/extracted/; their sha256s are recorded in provenance.json → full_state_exports_sha256.
  3. python3 scripts/pipeline-nysed/filter_nyc_ca.py filters to NYC school-level rows (ENTITY_CD county prefix 31-35 or 3075…, building digits 9-12 != '0000') and writes the *_nyc.csv files here, columns unchanged.

File facts

  • SRC{2019,2022,2024,2025}_{EM,HS}_CA_nyc.csv. Each file carries two YEAR values (current + prior reporting year). YEAR is the school-year END: 2019 = 2018-19 SY. Combined coverage: 2017-18 through 2024-25, except 2019-20 (never published — COVID waiver).
  • Column layouts differ by era (SRC2019 has MIP-goal columns and no INSTITUTION_ID; SRC2025 drops LEVEL/OVERRIDE) — see docs/qa_reports/nysed/FORMAT_NOTES.md §3.
  • All values are quoted text. Suppression marker is a literal s. DATA_REP_FLAG = "N" means the school did not report attendance data (notably the entire Success Academy network from 2023-24 on).
  • ENTITY_CD is the 12-digit BEDS code: county(2) + city/town(2) + district(2) + LEA type(2; 86 = charter) + building(4).