A Code Benchmark Is Only as Strict as Its Tests
Setting: code-generation models are usually scored by running the programs they write against unit tests. A common score is pass@k: let the model make k attempts at a problem and count the problem solved if any attempt passes every test. The score is only as strict as the tests.
Liu, Xia, Wang and Zhang (NeurIPS 2023, "Is Your Code Generated by ChatGPT Really Correct?") asked how strict the tests in the popular HumanEval benchmark really are. Their framework, EvalPlus, generates many extra test inputs automatically, using both an LLM and mutation of existing inputs. Applied to HumanEval it adds 80 times as many tests, producing HumanEval+.
Across 26 popular models they report that HumanEval+ catches a significant amount of previously undetected wrong code, lowering pass@k by up to 19.3 to 28.9% depending on k.
The tests also changed who looks best. They report that WizardCoder-CodeLlama and Phind-CodeLlama outperform ChatGPT on HumanEval+, while neither could on the original HumanEval: test insufficiency can lead to mis-ranking.
Why it matters: a leaderboard gap between two models can come from test coverage, not from the models.
The habit: when you see a pass rate on a code benchmark, ask how many tests each problem has and whether anyone has tried to break them.
Source: Liu et al., NeurIPS 2023, read November 4, 2026.
