Pass@k Crossover Tests Tell You When to Ship the Base Model

Every major model family now ships in pairs: a base checkpoint and a sibling tuned with reinforcement learning from verifiable rewards (RLVR). Leaderboards say ship the sibling, because leaderboards grade pass@1 and RLVR training exists to lift pass@1. But if your pipeline draws k samples per task and a verifier picks the best one, the leaderboard has answered a different question than the one you face. The pass@k crossover decides your choice: tuned checkpoints tend to win at small k and lose to their own base model at large k, and where that flip lands depends on your workload, not on the benchmark.

The mechanism is sharpening. RLVR concentrates probability mass on the modes your verifier approves, which raises the odds that the first sample passes while thinning the tail of distinct correct answers. Two curves shaped this way must cross somewhere. A deterministic single-shot pipeline lives to the left of that crossing, where the tuned model usually wins. A verifier-gated best-of-n pipeline can live to the right, where the base model solves more problems per dollar.

This piece delivers what the research abstracts leave out: a decision rule keyed to pipeline shape, a nearly free harness change that reports the whole curve instead of one point, and a paired bootstrap procedure that tells you whether the crossover on your tasks is signal or noise. It also keeps score between settled results and what a preprint posted within the week merely proposes.

What pass@k measures and how to compute it

pass@k is the probability that at least one of k independent samples from a model passes your verifier. It is a curve indexed by budget, and pass@1 is one point on that curve. Most bad checkpoint calls come from comparing points at different k, or from assuming the point predicts the curve.

A worked example from ten stored labels

Suppose you draw n = 10 samples on one problem and c = 4 of them pass. The unbiased pass@k estimator from Chen et al.'s Codex paper is 1 - C(n - c, k) / C(n, k), where the subtracted term is the probability that k draws miss all c correct samples. Applied to the same ten labels:

k	pass@k
1	0.40
2	0.67
4	0.93
8	1.00

The labels that produced the 0.40 also contain the 0.93. Note the last row: once k exceeds n - c, the estimator returns exactly 1 by construction, because any k samples must include a correct one. That saturation is an artifact of small n, so generate n comfortably above the largest k you report; n = 32 covers a grid up to 16 with room to spare.

Why the estimator makes curves nearly free

The labels already exist. Any harness that computes pass@1 has, by definition, scored samples per problem; scoring n instead stores n booleans per problem, and the estimator turns that matrix into pass@k for every k at once. The HumanEval repository ships the reference implementation. The objection that curves are expensive dates from habits formed when each new k meant a new eval run; with the estimator, the marginal cost of a full curve over one point is a loop over labels you already logged.

The pass@k crossover evidence from 2025 studies

The crossover is an established observation now, not a fringe one. Yue et al.'s 2025 study measured RLVR-tuned models against their base checkpoints across math and code benchmarks and found the tuned models ahead at pass@1 while falling behind as k grew, a reversal that independent groups have since reported across model families. The recipe at issue, verifier-driven RL of the kind DeepSeek-R1 popularized, optimizes against unit tests and final-answer checks and produced the reasoning checkpoints now topping pass@1 leaderboards.

The other leg of the evidence predates it. Brown et al.'s coverage results on repeated sampling found that a base code model's coverage on SWE-bench Lite kept climbing with sample count, reaching roughly 95 percent at 250 samples, several times its single-sample pass rate. pass@1 systematically understates what a base model can reach given test-time compute, which is precisely the regime a verifier-gated production pipeline lives in.

The sharpening account of what RLVR does

Put the two legs together and one mechanism explains both facts. RLVR rewards whichever answer modes the verifier accepts, so probability mass concentrates on those modes, and studies of RLVR training dynamics document falling output entropy as training progresses. Concentrated mass raises the chance the first sample lands on an approved mode (pass@1 up) and lowers the number of distinct correct modes reachable within a large budget (tail coverage down). The recurring question, does RLVR add capability or sharpen the distribution, resolves to an evidence pattern: higher pass@1 with thinner tails is what redistribution of existing mass predicts, and it explains the crossover better than benchmark noise does.

Settled results versus this week's proposals

Separate the tiers of evidence before you reorganize a pipeline. Settled: the unbiased estimator (2021), coverage scaling under repeated sampling (2024), and the crossover-plus-sharpening observations (2025).

Then there is the genuinely new tier. A September 2026 preprint, "RLVR is a Kernel, Not a Function," makes three moves worth knowing, all pending replication. First, it argues that a visible pass@k crossover is not automatically a statistical one: across five public RLVR pairs, the initial evaluations established no crossover at conventional significance, while a far larger evaluation on fresh prompts located a reversal whose first loss landed between 11 and 61 samples. Second, its power analysis holds that failing to detect a crossing is weak evidence of absence, and that adding problems can help more than adding answers per problem. Third, it reports that prompts with identical base success rates diverge after RLVR in ways that repeat across independent generation halves, making the base-to-tuned relationship a conditional distribution rather than a single curve.

Treat that tier as proposal, not canon. The practical residue survives the hedge: crossings need statistics, more problems beat more samples per problem, and a non-detection is not a disproof.

A decision rule by pipeline shape

The base model vs tuned model decision comes down to where your operating k lands relative to the crossover. At k = 1 you are buying the sharpest single sample, which the tuned checkpoint usually sells. In best-of-n sampling behind a verifier you are buying tail mass, which the base checkpoint often holds more of per dollar. The break-even k is workload-specific, so treat the table as where to start looking and the next section as how to measure.

Pipeline shape	Typical k	Usual winner	Why
Deterministic single-shot (agent steps, extraction, classification)	1	Tuned	Higher pass@1, tighter instruction following
Verifier-gated best-of-n (code against tests, math against a checker)	8 to 64	Base, often	Thicker tail of distinct correct solutions
Latency-bound chat or copilot	1 to 2	Tuned	Latency and style constraints, no verifier in the loop
Offline batch with cheap verification	50 and up	Measure it	The crossover location decides

The tie is economic as much as statistical. Compute tokens per solved task: k times mean tokens per sample, plus verifier tokens, divided by problems solved at that k. Tuned reasoning models often emit longer chains of thought, so their per-sample token cost can be a multiple of the base model's even before counting the extra samples the base model needs. A base model that wins pass@16 by a hair can still lose on cost once you price the extra completions and verifier calls. When to deploy the base model instead of the RL tuned one is settled by that arithmetic, not by the leaderboard.

How to test for a pass@k crossover on your own tasks

A visible curve crossing in one eval run is a hypothesis. Recent work proposes a formal crossover test for exactly this comparison; the do-it-yourself version below gets you most of the way with tools you already run.

1. Freeze a shared problem set. Same problems, same prompts, same sampling configuration for both checkpoints. Pairing on problems removes difficulty variance, your largest noise source.
2. Draw n samples per problem per checkpoint. Keep n well above your largest reported k, honoring the saturation rule. n = 32 supports the grid 1, 2, 4, 8, 16.
3. Verify every sample and store the binary matrix. This is the only expensive step, and you pay it once.
4. Estimate pass@k per problem with the unbiased estimator, averaged over problems, for both checkpoints on the same k grid.
5. Compute the paired difference d_i(k), tuned minus base, for every problem, then average.
6. Bootstrap the problems. Resample problem indices with replacement, recompute the mean difference at each k, repeat 10,000 times, and read percentile intervals. A bootstrap confidence interval comparing pass@k of two models makes no parametric assumptions; the NIST e-Handbook covers the method.
7. Correct for testing many k values. A grid of five k values at nominal 0.05 each invites one spurious flip; apply Holm or Bonferroni across the grid, or demand a consistent sign across neighboring k before believing a point. The classic A/B testing pitfalls of peeking and repeated looks apply unchanged.
8. Read the band, not a point. A crossover is established when the corrected interval for d(1) sits above zero and the interval for some larger k sits below. If every interval straddles zero, you are underpowered, and the new preprint's power analysis says widen the problem set before you crank n.

The core fits in a page:

import numpy as np
from math import comb

def pass_at_k(labels, k):          # labels[i, j] = 1 if sample j passed
    n, c = labels.shape[1], labels.sum(1)
    return float(np.mean([1 - comb(n - int(x), k) / comb(n, k) for x in c]))

def paired_bootstrap(base, tuned, k, B=10_000):
    diffs = []
    for _ in range(B):
        idx = np.random.choice(len(base), len(base), replace=True)
        diffs.append(pass_at_k(tuned[idx], k) - pass_at_k(base[idx], k))
    lo, hi = np.percentile(diffs, [2.5, 97.5])
    return lo, hi

Two ways to void the test: comparing checkpoints on disjoint problem sets, or at mismatched k. Both destroy the pairing and inflate your intervals.

Report pass@k curves, not single points

The harness change is an afternoon of work. Log per-sample correctness once, then report pass@k at several k alongside pass@1; lm-evaluation-harness already implements the metric, and a homegrown harness needs only the estimator loop above. Storage cost is n booleans per problem per checkpoint. The discipline it buys is larger: two checkpoints whose pass@1 scores differ by less than your interval width are indistinguishable, a checkpoint that trails at k = 1 can lead at k = 16, and reasoning model evaluation that reports one point cannot see either fact. Comparing two checkpoints across sampling budgets is the entire game; a single-number comparison forfeits it.

When the tuned checkpoint still wins

The crossover argument has edges, and honest selection names them.

• Cost per solved task at small k. The tuned model's pass@1 edge means fewer samples, fewer verifier calls, and often lower total tokens per solved problem even with longer chains.
• Anything the verifier cannot grade. Formatting, tone, refusal behavior, and instructions outside verifier scope. Tuned checkpoints most consistently out-ship their base on instruction adherence.
• Latency-bound products. A user-facing latency budget rarely tolerates eight samples, which pins you at small k where tuning wins.
• Chain-of-thought as product. In user-facing settings the tuned model's visible reasoning style is part of the experience, not overhead to optimize away.
• Weak verifiers. Best-of-n inherits your verifier's errors. A noisy or gameable checker converts base-model tail mass into wrong-but-accepted answers and quietly erases the base model's edge.

The shipping checklist

1. Inventory your pipelines by k. Which run deterministic single-shot, which run best-of-n, and at what n.
2. Assemble a shared problem set of a few hundred tasks drawn from your real workload, not from a public benchmark.
3. Fix sampling parameters and draw n = 32 samples per problem from both checkpoints.
4. Store per-sample pass/fail as a matrix. Never average before storing.
5. Compute unbiased pass@k on the grid 1, 2, 4, 8, 16 for both checkpoints.
6. Run the paired bootstrap and plot the confidence band at every k.
7. Apply a multiple-comparison correction before calling any sign flip significant.
8. Compute tokens per solved task at your production k for both checkpoints.
9. Ship the tuned checkpoint if you operate left of the crossover, the base checkpoint if you operate right of it, and widen the problem set if the intervals cannot tell you where the crossing is.

The leaderboard crowns the model that wins one sample. Your sampling budget crowns the model that wins your pipeline, and you can measure that this week.