Paper Autocomplete

Can a model re-derive a paper's held-out experiments from only its primary claims?

⌕

Can a model anticipate a paper's hidden experiments? Each bar = claims re-derived as a share of that paper's held-out set, colored by model, with a budget ramp (pale core at k=4 → deep at the selected budget), run-to-run 95% CI whisker at the tip, and a dashed union ceiling (best any clean model reaches). Hatched bars are excluded for memorization. Hover a bar for the full k-ramp and memorization status.

Proposal budget k

Every model-generated experiment proposal across all papers, models, and budgets. A proposal is recovered (gold tag) when the judge matched it to a held-out claim; otherwise NONE. Filter with the chips, search the free text, then click any row to expand its full manipulation / measurement / establishes (and, when recovered, the matched claim + judge reasoning).

Paper

recall@k = number of held-out claims re-derived among k generated proposals (mean over N=4 runs, 95% t-CI band). Click any point to inspect that (model, k) cell. Contaminated cells (model recognized the paper) are excluded from the curves.

Cell detail

Click a point on the curve to see recovered held-out claims and a sample of non-recovered proposals.

Paper

Claims under the primary-only mask

Shown / "saved" (primary)

Held-out / "not saved" (to be recovered)

Recovery heatmap

Budget k k = 16

Rows = held-out claims · columns = models · cell = number of N=4 runs (0–4) that recovered the claim at the selected k. Click a cell for detail.

Cell detail

Click a heatmap cell to see which runs found the claim and the matching proposal.

Paper

2D UMAP (cosine) of EmbeddingGemma embeddings over all of the paper's claims + generated proposals (trustworthiness ≈0.95; axes are not quantitatively meaningful). Points colored by generating model. Gold-filled = the gold-standard held-out claims; gold-bordered = proposals the judge matched to a held-out claim. Click any point for its text.

Point detail

Click a point to see the proposal or claim text.

Paper

Left model

Right model

Per held-out claim, the difference in recovery rate (% of runs, pooled over all budgets) between the two models. Points left of zero are recovered more reliably by the left model, right of zero by the right model; claims near zero are recovered equally. Gold = exploratory claims. Hover for both rates.

Name the authors, given only the title

Guess detail

Click any cell to read the model's actual forced author guesses across all 4 retries and see exactly who was named.