Find What Works: Schools
← All source documentsRaw file
Source document

docs/agents/quantitative-knowledge.md

Served verbatim from the project repository. Internal working document conventions apply: documents may reference file paths, branch names, and findings-ledger anchors from the repo.

Quantitative — accumulated knowledge

Statistical rigor patterns, sanity-checks, and caveats the quant has built up across stories. Append-only across sessions.

Format

## YYYY-MM-DD — <short title>
Story: <story-id>
Pass: <pass number>

<the rigor check, caveat, or pattern and how to apply it>

2026-05-29 — Top-N residual lists are selected on the outcome

Story: punching-above-50 Pass: 3

Any "top 50 expectation-beaters" list is selected on the outcome (residual) we're claiming to study. We can use it to localize where to look for the mechanism, but we can't use it to back out "what works" without confirming that the residual is stable. Story should always acknowledge the selection.

2026-05-29 — 3-year rolling residual is the right robustness check

Story: punching-above-50 Pass: 3

Single-year proficiency rates carry cohort noise; a top-N list built on one year of data may not survive a second year. The right robustness check is a 3-year rolling residual that requires sustained over-performance to rank. Mention this as a follow-up even when we can't run it right now.

2026-05-29 — Decile-mean baseline vs continuous regression baseline

Story: punching-above-50 Pass: 3

Decile means are a piecewise-constant baseline. They overweight the schools near the ceiling of decile 1 and floor of decile 10 (the compression there means decile means understate the actual range a typical school occupies). A continuous regression baseline shrinks residuals at the extremes by 3-5 points and is the more defensible methodology if anyone asks. Flag for sensitivity.

2026-05-29 — Single-year proficiency jumps materially over citywide bump

Story: punching-above-50 Pass: 3

In a recalibration year, the citywide bump (~4-13 points by grade and subject) is the noise floor for "the number went up." A school-level year-over-year jump materially larger than the citywide bump can be a real instructional improvement, but is also consistent with cohort flipping, opt-out flipping, or DBN reassignment. Flag for confirmation; don't treat as established.

2026-05-29 — Always anchor "over-representation" claims to the underlying share

Story: punching-above-50 Pass: 6

When a story says a group is "over-represented" on a list, the reader needs the underlying base rate to evaluate the claim. "Brooklyn has 28 of the top 50" is meaningless until you say "Brooklyn enrolls ~30% of NYC's elementary students." Always include the anchor in the chart caption or the body, not in a footnote.

2026-05-29 — Linear "won't recover until 20XX" extrapolations are usually wrong

Story: covid-absenteeism-stuck Pass: 7

Three data points of leveling-off don't establish a stable trend. Public-health-style recovery rates are commonly non-linear. The right framing is "at the current rate, X; non-linear recovery could change this in both directions." The structural finding (new floor sits N points above old) is usually more defensible than the extrapolation.

2026-05-29 — Distinguish "stated action" from "inference from data"

Story: covid-test-recalibration Pass: 3

When a story argues a state agency did X (e.g. moved a cutscore), be explicit about whether the agency publicly stated the action or whether we're inferring it from the data signature. Different defensibility profiles. The inference can still be the parsimonious explanation, but the language should make the distinction so readers know what's an established fact vs an interpretation.

2026-05-29 — Subgroup-gap stories systematically sample larger schools

Story: ell-gap-disaster Pass: 6

Cell-suppression below ~5 students means any subgroup-gap analysis filters out small schools. The reported gap is for the population of schools where both subgroups exist in measurable numbers — not all schools. Always state this explicitly; the gap may look different in the suppressed-from-view portion.

2026-05-29 — Small-cell precision is misleading

Story: ell-gap-disaster Pass: 3

A "0% proficient" cell on n=3 students is one student away from "33%." Displayed precision (e.g., "0.0%" for the ELL subgroup at PS 122 Mamie Fay) misleads readers into treating these as stable rates. State the cell size in or near the table, and acknowledge that a single student's score would have moved the rate materially.