Skip to content

Compare two eval arms

astro_weights.eval compares base arm and edit arm on same items from per-item records. Never runs model: caller generates outputs, grades them, writes one Record per item, seed and arm. Numpy only. To generate records from model, see run eval end to end.

Compare

from astro_weights.eval import Record, compare, format_report, prompt_hash

prompts = {f"q{i}": f"question {i}" for i in range(8)}


def record(item_id: str, arm: str, score: float, tokens: int) -> Record:
    return Record(
        item_id=item_id,
        seed=0,
        arm=arm,
        prompt_hash=prompt_hash(prompts[item_id]),
        output="...",
        score=score,
        n_output_tokens=tokens,
        hit_cap=False,
    )


base = [record(k, "base", 1.0 if i < 5 else 0.0, 300) for i, k in enumerate(prompts)]
edit = [record(k, "edit", 1.0 if i < 6 else 0.0, 250) for i, k in enumerate(prompts)]

result = compare(base, edit)
print(format_report(result))

Prints:

n=8 items  base=0.6250  edit=0.7500
score diff=+0.1250  SE=0.1250  95% CI [-0.1200, +0.3700]; not significant; smallest detectable difference at this n is 0.3500
McNemar exact p=1  (base-only right=0, edit-only right=1)
tokens: base=300.0  edit=250.0  diff=-50.0 (-16.7%)  bootstrap 95% CI [-50.0, -50.0]
cap-hit rate: base=0.000  edit=0.000
WARNING: only 8 items (<100); intervals are unreliable

What compare does

  • pair joins arms on (item_id, seed). Raises ValueError when keys differ, key repeats within arm, or prompt_hash differs between arms.
  • Scores and token counts get averaged over seeds per item first; every statistic runs over items.
  • Score difference: paired SE, 95% CI diff ± 1.96 * SE.
  • Exact McNemar p-value only when every per-item score is 0 or 1; otherwise mcnemar_p is None.
  • Token difference: bootstrap 95% CI, n_boot=10000 resamples, seed=0 by default.
  • small_n is True under 100 items; format_report then adds warning line.
  • Minimum detectable effect: score_mde = (1.96 + 0.84) * se, tokens_mde = (1.96 + 0.84) * tokens_se (5% two-sided test, 80% power). When 95% CI contains 0, report line ends with not significant; smallest detectable difference at this n is ....

Noise floor

Run base arm twice under same settings; compare of those two runs is noise floor. exceeds_floor(result, floor) is True per metric when its CI excludes 0 and |diff| > |floor diff| + 1.96 * floor SE; NaN anywhere gives False. format_report(result, floor) adds one line per metric. Continues example above:

from astro_weights.eval import exceeds_floor

# base arm run again; token counts differ by 1 on half the items
rerun = [
    record(k, "base", 1.0 if i < 5 else 0.0, 300 + i % 2) for i, k in enumerate(prompts)
]
floor = compare(base, rerun)
print(exceeds_floor(result, floor))
print(format_report(result, floor).splitlines()[-3:-1])

Prints:

{'score': False, 'tokens': True}
['noise floor (score): |diff|=0.1250 does not exceed |floor diff| + 1.96 SE = 0.0000', 'noise floor (tokens): |diff|=50.0 exceeds |floor diff| + 1.96 SE = 0.9']

Command line: astro-weights eval BASE.jsonl EDIT.jsonl --floor A.jsonl B.jsonl; see run eval end to end.

Clustered items

Items that share source (same passage, same task family) are not independent. Set cluster_id on every record of both arms, same id for item across arms and seeds; compare then uses cluster-robust SE (Miller 2024, with G / (G - 1) factor) for score SE, CI and MDE and for token SE and MDE, and token bootstrap resamples whole clusters. Continues example above:

from dataclasses import replace


def clustered(records: list[Record]) -> list[Record]:
    # two items per cluster: q0 and q1 share c0, q2 and q3 share c1, ...
    return [replace(r, cluster_id=f"c{int(r.item_id[1:]) // 2}") for r in records]


cres = compare(clustered(base), clustered(edit))
print(cres.n_clusters)
print(format_report(cres))

Prints:

4
n=8 items in 4 clusters  base=0.6250  edit=0.7500
score diff=+0.1250  SE=0.1250  95% CI [-0.1200, +0.3700]; not significant; smallest detectable difference at this n is 0.3500
McNemar exact p=1  (base-only right=0, edit-only right=1); ignores clustering
tokens: base=300.0  edit=250.0  diff=-50.0 (-16.7%)  bootstrap 95% CI [-50.0, -50.0]
cap-hit rate: base=0.000  edit=0.000
WARNING: only 4 clusters (<100); intervals are unreliable
  • cluster_id on some records but not all, or one item with two ids: ValueError.
  • Under 2 clusters: SE, CI and MDE are NaN.
  • small_n also flags under 100 clusters. McNemar line ends ; ignores clustering: McNemar always treats items as independent.
  • Without cluster_id, items treated as independent.

Store records

write_jsonl(path, records) writes one JSON object per line. read_jsonl(path) reads them back, skips blank lines, raises ValueError with line number on invalid JSON, missing or unknown keys, wrong value type or non-finite score. prompt_hash(text) is SHA-256 hex digest of UTF-8 prompt.