Methodology and approach documents
The working documents behind this site — source registries, validation methodologies, QA reports, analysis memos, statistics packets, and the scripts that generate them — served directly from the project repository. Start from the NYC or Philly methodology overviews for narrative context.
Philadelphia
Source registry, integration plans, per-source first-fetch findings, and the validation methodology for the Philly build.
- docs/cities/philly/00_overview.md · 7 KB
- docs/cities/philly/01_inventory.md · 8 KB
- docs/cities/philly/02_gap_analysis.md · 10 KB
- docs/cities/philly/03_integration_plan.md · 9 KB
- docs/cities/philly/04_contribution_plan.md · 5 KB
- docs/cities/philly/05_validation_spec.md · 9 KB
- docs/cities/philly/README.md · 2 KB
- docs/cities/philly/findings.md · 18 KB
- docs/cities/philly/sources.yaml · 39 KB
- pipeline_philly/README.md · 1 KB
- pipeline_philly/discover.py · 8 KB
- pipeline_philly/findings/futurereadypa_performance_2018-19.md · 1 KB
- pipeline_philly/findings/futurereadypa_performance_2019-20.md · 1 KB
- pipeline_philly/findings/futurereadypa_performance_2020-21.md · 1 KB
- pipeline_philly/findings/futurereadypa_performance_2021-22.md · 70 KB
- pipeline_philly/findings/futurereadypa_performance_2022-23.md · 70 KB
- pipeline_philly/findings/futurereadypa_performance_2023-24.md · 70 KB
- pipeline_philly/findings/futurereadypa_performance_2024-25.md · 76 KB
- pipeline_philly/findings/futurereadypa_school_fast_facts_2024-25.md · 2 KB
- pipeline_philly/findings/pses_2024-25.md · 689 B
- pipeline_philly/findings/pses_topic_subtopic_scores_all_years.md · 587 B
- pipeline_philly/findings/sdp_attendance_90_district_monthly.md · 926 B
- pipeline_philly/findings/sdp_attendance_90_school.md · 944 B
- pipeline_philly/findings/sdp_attendance_ada_school.md · 713 B
- pipeline_philly/findings/sdp_enrollment_demographics_school_2024-25.md · 2 KB
- pipeline_philly/findings/sdp_graduation_school_2014-2025.md · 884 B
- pipeline_philly/findings/sdp_master_school_list_2024-25.md · 3 KB
- pipeline_philly/findings/sdp_pssa_keystone_2023-24.md · 684 B
- pipeline_philly/findings/sdp_pssa_keystone_2024-25.md · 694 B
- pipeline_philly/findings/sdp_spree_2023-24.md · 1 KB
- pipeline_philly/findings/sdp_suspensions_school.md · 2 KB
- pipeline_philly/verify/METHODOLOGY.md · 8 KB
- pipeline_philly/verify/__init__.py · 0 B
- pipeline_philly/verify/absenteeism_full_qa.py · 52 KB
- pipeline_philly/verify/common.py · 2 KB
- pipeline_philly/verify/completeness.py · 4 KB
- pipeline_philly/verify/connectors/__init__.py · 0 B
- pipeline_philly/verify/connectors/served_db.py · 2 KB
- pipeline_philly/verify/connectors/source_file.py · 6 KB
- pipeline_philly/verify/cross_publisher_attendance.py · 15 KB
- pipeline_philly/verify/cross_publisher_pssa.py · 5 KB
- pipeline_philly/verify/reconcile.py · 7 KB
- pipeline_philly/verify/reconcile_acct_actual.py · 6 KB
- pipeline_philly/verify/reconcile_computed.py · 12 KB
- pipeline_philly/verify/reconcile_spree.py · 9 KB
- pipeline_philly/verify/test_compare_cells.py · 1 KB
- scripts/loaders/philly/_lib.ts · 7 KB
- scripts/loaders/philly/compute_percentiles.ts · 7 KB
- scripts/loaders/philly/derive_peer_groups.ts · 9 KB
- scripts/loaders/philly/futurereadypa_longformat_attendance.ts · 7 KB
- scripts/loaders/philly/futurereadypa_performance.ts · 12 KB
- scripts/loaders/philly/master_school_list.ts · 8 KB
- scripts/loaders/philly/metric_definitions.ts · 11 KB
- scripts/loaders/philly/pses.ts · 7 KB
- scripts/loaders/philly/sdp_attendance_and_suspensions.ts · 9 KB
- scripts/loaders/philly/sdp_enrollment_demographics.ts · 6 KB
- scripts/loaders/philly/sdp_graduation.ts · 5 KB
- scripts/loaders/philly/sdp_pssa_keystone.ts · 8 KB
- scripts/loaders/philly/sdp_pssa_keystone_2023-24.ts · 8 KB
NYC data quality and verification
The four-check validation methodology, per-metric QA reports, and cross-publisher reconciliation evidence for the NYC build.
- docs/qa_reports/chronic_absenteeism_rate__snapshot.json · 384 B
- docs/qa_reports/chronic_absenteeism_rate__snapshot.md · 3 KB
- docs/qa_reports/chronic_absenteeism_rate__validation.json · 1 KB
- docs/qa_reports/chronic_absenteeism_rate__validation.md · 1017 B
- docs/qa_reports/ela_all_proficiency__snapshot.json · 376 B
- docs/qa_reports/ela_all_proficiency__snapshot.md · 3 KB
- docs/qa_reports/ela_all_proficiency__validation.json · 1 KB
- docs/qa_reports/ela_all_proficiency__validation.md · 1017 B
- docs/qa_reports/ela_grade3_proficiency__validation.json · 1 KB
- docs/qa_reports/ela_grade3_proficiency__validation.md · 983 B
- docs/qa_reports/ela_grade7_proficiency__validation.json · 1 KB
- docs/qa_reports/ela_grade7_proficiency__validation.md · 1019 B
- docs/qa_reports/graduation_rate_4yr__validation.json · 1 KB
- docs/qa_reports/graduation_rate_4yr__validation.md · 860 B
- docs/qa_reports/math_all_proficiency__snapshot.json · 378 B
- docs/qa_reports/math_all_proficiency__snapshot.md · 3 KB
- docs/qa_reports/math_all_proficiency__validation.json · 1 KB
- docs/qa_reports/math_all_proficiency__validation.md · 1020 B
- docs/qa_reports/math_grade3_proficiency__validation.json · 1 KB
- docs/qa_reports/math_grade3_proficiency__validation.md · 985 B
- docs/qa_reports/math_grade7_proficiency__validation.json · 1 KB
- docs/qa_reports/math_grade7_proficiency__validation.md · 1022 B
- docs/qa_reports/nysed/FORMAT_NOTES.md · 15 KB
- docs/qa_reports/nysed/NARROWING.md · 18 KB
- docs/qa_reports/nysed/README.md · 2 KB
- docs/qa_reports/nysed/RECONCILIATION.md · 23 KB
- docs/qa_reports/nysed/stats.json · 59 KB
- docs/qa_reports/philly/__absenteeism_full_qa.json · 14 KB
- docs/qa_reports/philly/__absenteeism_full_qa.md · 7 KB
- docs/qa_reports/philly/__acct_vs_actual.json · 15 KB
- docs/qa_reports/philly/__acct_vs_actual.md · 9 KB
- docs/qa_reports/philly/__completeness.json · 2 KB
- docs/qa_reports/philly/__completeness.md · 774 B
- docs/qa_reports/philly/__computed_value.json · 3 KB
- docs/qa_reports/philly/__computed_value.md · 3 KB
- docs/qa_reports/philly/__cross_publisher_attendance.json · 10 KB
- docs/qa_reports/philly/__cross_publisher_attendance.md · 6 KB
- docs/qa_reports/philly/__cross_publisher_pssa.json · 3 KB
- docs/qa_reports/philly/__cross_publisher_pssa.md · 2 KB
- docs/qa_reports/philly/__spree_spot_check.json · 6 KB
- docs/qa_reports/philly/__spree_spot_check.md · 3 KB
- docs/qa_reports/philly/attendance_rate_above_90__validation.json · 390 B
- docs/qa_reports/philly/attendance_rate_above_90__validation.md · 407 B
- docs/qa_reports/philly/graduation_rate_4yr__validation.json · 403 B
- docs/qa_reports/philly/graduation_rate_4yr__validation.md · 434 B
- docs/qa_reports/philly/keystone_algebra_proficiency__validation.json · 391 B
- docs/qa_reports/philly/keystone_algebra_proficiency__validation.md · 412 B
- docs/qa_reports/philly/pssa_all_ela_proficiency__validation.json · 391 B
- docs/qa_reports/philly/pssa_all_ela_proficiency__validation.md · 408 B
- docs/qa_reports/philly/pssa_all_math_proficiency__validation.json · 392 B
- docs/qa_reports/philly/pssa_all_math_proficiency__validation.md · 410 B
- docs/qa_reports/philly/pssa_grade3_ela_proficiency__validation.json · 394 B
- docs/qa_reports/philly/pssa_grade3_ela_proficiency__validation.md · 414 B
- docs/qa_reports/snapshot/CENSUS.md · 23 KB
- docs/qa_reports/snapshot/PROVENANCE.md · 11 KB
- docs/qa_reports/snapshot/README.md · 5 KB
- docs/qa_reports/snapshot/census-stats.json · 114 KB
- docs/qa_reports/snapshot/provenance-stats.json · 39 KB
- docs/qa_reports/survey_student_2021-22__validation.json · 370 B
- docs/qa_reports/survey_student_2021-22__validation.md · 442 B
- docs/qa_reports/survey_student_2022-23__validation.json · 371 B
- docs/qa_reports/survey_student_2022-23__validation.md · 443 B
- docs/qa_reports/survey_student_2023-24__validation.json · 371 B
- docs/qa_reports/survey_student_2023-24__validation.md · 443 B
- docs/qa_reports/survey_teacher_2021-22__validation.json · 376 B
- docs/qa_reports/survey_teacher_2021-22__validation.md · 448 B
- docs/qa_reports/survey_teacher_2022-23__validation.json · 375 B
- docs/qa_reports/survey_teacher_2022-23__validation.md · 447 B
- docs/qa_reports/survey_teacher_2023-24__validation.json · 375 B
- docs/qa_reports/survey_teacher_2023-24__validation.md · 447 B
- verify/FINDINGS.md · 12 KB
- verify/METHODOLOGY.md · 18 KB
- verify/README.md · 4 KB
Analysis working papers
Design memos, descriptive analyses, findings ledgers, committed statistics packets, and the scripts that generate them.
- data/analysis/absenteeism-descriptives.json · 93 KB
- data/analysis/california_replication.json · 6 KB
- data/analysis/chalkbeat_verification.json · 94 KB
- data/analysis/chalkbeat_verification_annex.json · 74 KB
- data/analysis/dd1_grade_distortion.json · 18 KB
- data/analysis/dd2_distribution_tail.json · 16 KB
- data/analysis/dd3_units.json · 21 KB
- data/analysis/philly-absenteeism-descriptives.json · 70 KB
- data/analysis/philly_outliers_trends.json · 16 KB
- data/analysis/philly_replication.json · 7 KB
- data/analysis/sth-shares.json · 254 KB
- docs/analysis/absenteeism/01_descriptives.md · 44 KB
- docs/analysis/absenteeism/03_bakeoff_design.md · 18 KB
- docs/analysis/absenteeism/bakeoff/04_bakeoff_results.md · 50 KB
- docs/analysis/absenteeism/bakeoff/README.md · 3 KB
- docs/analysis/absenteeism/deep-dives/00_PLAN.md · 14 KB
- docs/design/01_inventory.md · 12 KB
- docs/design/02_gap_analysis.md · 15 KB
- docs/design/03_integration_plan.md · 15 KB
- docs/design/04_contribution_plan.md · 6 KB
- docs/design/05_validation_spec.md · 20 KB
- docs/design/06_absenteeism_deep_dive_plan.md · 8 KB
- docs/design/07_absenteeism_metric_alignment.md · 10 KB
- docs/design/08_absenteeism_findings.md · 17 KB
- docs/design/README.md · 2 KB
- docs/design/nyc-school-outcomes-spec.md · 32 KB
- docs/design/reference-scaffold/README.md · 7 KB
- docs/design/reference-scaffold/config/comparability.yaml · 2 KB
- docs/design/reference-scaffold/config/covariates.yaml · 847 B
- docs/design/reference-scaffold/config/outcomes.yaml · 3 KB
- docs/design/reference-scaffold/docs/outcome_cards/ela_math_g3.md · 4 KB
- docs/design/reference-scaffold/docs/qa_reports/ela_math_g3__checks.md · 279 B
- docs/design/reference-scaffold/docs/qa_reports/ela_math_g3__reconcile.md · 545 B
- scripts/analysis/RESEARCH_NOTES.md · 6 KB
- scripts/analysis/_probe-data.ts · 3 KB
- scripts/analysis/bakeoff.py · 12 KB
- scripts/analysis/cross_outcome.py · 3 KB
- scripts/analysis/diagnostics.py · 4 KB
- scripts/analysis/export-dataset.ts · 6 KB
- scripts/analysis/inspect_sqr.py · 2 KB
- scripts/analysis/philly-absenteeism-descriptives.ts · 64 KB
- scripts/analysis/run.py · 8 KB
Source data (NYC / NYSED)
The exact NYSED source tables the chronic-absenteeism reconciliation ran against, with provenance.
- data/sources/nysed/README.md · 2 KB
- data/sources/nysed/SRC2019_EM_CA_nyc.csv · 2.8 MB
- data/sources/nysed/SRC2019_HS_CA_nyc.csv · 1.3 MB
- data/sources/nysed/SRC2022_EM_CA_nyc.csv · 2.6 MB
- data/sources/nysed/SRC2022_HS_CA_nyc.csv · 1.2 MB
- data/sources/nysed/SRC2024_EM_CA_nyc.csv · 2.7 MB
- data/sources/nysed/SRC2024_HS_CA_nyc.csv · 1.2 MB
- data/sources/nysed/SRC2025_EM_CA_nyc.csv · 2.7 MB
- data/sources/nysed/SRC2025_HS_CA_nyc.csv · 1.2 MB
- data/sources/nysed/provenance.json · 4 KB
Everything else
Specs and remaining documents.
- SPECS.md · 18 KB
- docs/agents/ai-writing-filter-knowledge.md · 3 KB
- docs/agents/ai-writing-filter.md · 8 KB
- docs/agents/editor-knowledge.md · 3 KB
- docs/agents/editor.md · 5 KB
- docs/agents/education-expert.md · 5 KB
- docs/agents/education-knowledge.md · 5 KB
- docs/agents/pass-log/covid-absenteeism-stuck.md · 11 KB
- docs/agents/pass-log/covid-district-18-comeback.md · 8 KB
- docs/agents/pass-log/covid-test-recalibration.md · 10 KB
- docs/agents/pass-log/ell-gap-disaster.md · 10 KB
- docs/agents/pass-log/punching-above-50.md · 24 KB
- docs/agents/plain-language-knowledge.md · 7 KB
- docs/agents/plain-language.md · 8 KB
- docs/agents/quantitative-knowledge.md · 4 KB
- docs/agents/quantitative.md · 6 KB
- docs/editorial-process.md · 10 KB
- docs/writing-style.md · 9 KB