Measure Whether a Citation Checker Can Tell Support From Topic
A research agent writes a related-work paragraph: "Method X improves GSM8K accuracy by 4 points [3]." Source [3] is about Method X, and it mentions GSM8K. It may still not say that. An HTTP 200 and a shared keyword are not support.
OpenAI's evaluation guide asks for test data that includes typical, edge and adversarial cases, and it names a failure to avoid: automated metrics never calibrated against human evaluation. A citation checker is exactly such a metric.
The protocol. This course has not run it; here is how you would.
Write 60 claim-source pairs, 20 of each kind:
- Supported. The passage states the claim.
- Contradicted. The passage reports a different number or direction.
- On-topic but silent. Same method, same benchmark, nothing that supports the claim. This is where keyword overlap fools a checker.
Run the checker and require it to quote the passage it relies on, so each verdict can be inspected. Score accuracy separately for each kind; a single pooled number hides a checker that passes the easy kind and fails the silent one.
Put intervals on it. With 20 pairs per kind, an observed 90% has a 95% half-width of about 1.96 × √(0.9 × 0.1 / 20) = 1.96 × 0.067 ≈ ±0.13. That is wide. Treat the per-kind numbers as rough until you add pairs, and compute the n you need before you claim one checker beats another.
Calibrate against people. Have a person label 30 of the pairs without seeing the checker's verdicts, and count agreement. The guide's rule is to scale up an automated judge once it consistently agrees with human annotations.
Chapter 16 derives the intervals and paired comparisons, and the course repo's chapter 16 exercises include an evaluation-statistics notebook. Keep the source URL and retrieval date with every claim, so a later reader can tell a changed page from an unsupported answer.
