We Ran the Same Benchmark Twice and Got a Different Answer
If you've ever seen a single benchmark number and made a decision based on it — this model is 8x slower, this configuration is unreliable, this hardware underperforms — this is the experiment that should make you pause before doing that again.
We ran the same test twice, back to back, with nothing changed. The two runs told different stories. This is a short, standalone piece from our private LLM infrastructure series (E002/E002b) about what happened, and why "run it again" turned out to be the single highest-value thing we did in the entire six-experiment series.
The first run
We were benchmarking an 8-billion-parameter model's throughput — how many tokens per second it could generate — to compare against a smaller model we'd already tested. Most of the run looked normal. Then one single request took 564.8 seconds to complete: roughly six times longer than every other request in that same run.
That one request dragged the whole run's measured throughput down to 4.3 tokens per second. Compared against the smaller model's 35.6 tokens per second, that put the larger model at 8.3× slower — a big, clean, reportable number. The run also showed data collection gaps around the same time, and the model failed to release GPU memory cleanly afterward.
Taken at face value, this looked like a real finding: the larger model was dramatically, specifically worse on this hardware.
The rerun
Before trusting that number, we reran the identical test — same model, same prompt, same timeout, same collector, byte-for-byte unmodified. This is the standard check for whether a result reflects the system under test, or reflects a one-time fluke: reproduce it, or don't trust it yet.
The stall did not come back.
Not a smaller stall. Not a partial recurrence. The slowest request in the rerun was 118.5 seconds — the model's normal one-time cold-load cost, the same overhead every run pays on its first request. Everything else about the rerun matched what a clean run should look like:
| Original run | Exact rerun | |
|---|---|---|
| Slowest single request | 564.8 s (~6× the others) | 118.5 s (normal cold-load overhead) |
| Measured throughput | 4.3 tok/s | 6.86 tok/s |
| Generations completed | 11 | 15 |
| Samples collected | 2,001 of ~2,400 (83%) | 2,398 of ~2,400 (essentially complete) |
| Model released memory cleanly? | No | Yes, on schedule |
Once the stalled request is set aside from the original run's own numbers, its "clean" throughput comes out to roughly 6.2 tokens per second — close to the rerun's 6.86. The real gap between the two models was 5–6× slower, not 8.3×. The dramatic number from the first run wasn't wrong about there being a gap. It was wrong about the size of it, by a wide margin, because one transient event was doing most of the work.
We never found out exactly what caused the stall — and that's fine
We don't know, with certainty, what caused the original 564.8-second stall. A scheduling or resource-contention hiccup on the machine is the leading candidate, but we didn't chase it down further, and we didn't need to. The question that mattered for the benchmark wasn't "why did this happen" — it was "does this happen reliably, or was it a one-off." Reproducibility answered that question directly, without requiring a root-cause explanation.
This is worth separating explicitly: explaining an anomaly and confirming whether it recurs are two different investigations, and only one of them is required before you can trust a number. If something can't be explained but also doesn't reproduce, it doesn't get to anchor a conclusion. If it does reproduce, then it's a real property of the system, whether or not you understand why yet.
Why this matters more than it sounds like it should
A single benchmark run, however dramatic the result, is evidence about that one run — not evidence about the hardware, the model, or the configuration in general. That sounds obvious stated plainly. In practice, dramatic single-run numbers are exactly the ones that get shared, screenshotted, and acted on before anyone reruns them, precisely because they're the most interesting-looking result to come out of a session.
If a number is going to justify a decision — a purchase, a configuration change, a claim about reliability — it needs a second, independent run before it's trusted. Not a different model. Not a different configuration. The identical test, run again, checking specifically whether the same result comes back.
A result that doesn't reproduce isn't a wasted run — it's the finding. The original 564.8-second stall was real; it happened, it was measured accurately, and the data collection around it was sound. What it wasn't, was a property of the model or the hardware. Confirming that took the same amount of effort as the original run and completely changed what conclusion was safe to draw.
"Anomalous but explainable" and "anomalous but reproducible" are not the same bar. We didn't explain the stall. We confirmed it wasn't reproducible. That was sufficient to set it aside — explaining it would have been nice to have, but wasn't necessary to reach a trustworthy conclusion.
Where this fits
This is one result from a six-experiment series on private LLM infrastructure — including findings that reversed initial conclusions about GPU efficiency, hardware selection, and cost that looked equally solid on first measurement. The reproducibility discipline established here is what caught every one of them. The full synthesis is in The Metric That Kept Lying to Us.
Related reading: Why We Didn't Upgrade to the Bigger GPU · One Config Change Cut Our Inference Cost 4.8×
If you're evaluating infrastructure or model performance based on a single benchmark run — yours or someone else's — and want it checked properly before it anchors a decision, book a call.