AI Evaluation Confidence Interval Calculator
A 65% pass rate from 20 cases and the same rate from 2,000 cases should not lead to the same decision. Estimate the uncertainty around a binary AI evaluation result before treating a small difference as a real improvement.
See the uncertainty behind an AI evaluation pass rate
Enter the number of cases that passed and the total independently evaluated cases. This calculator uses a Wilson score interval to show a plausible range around the observed binary pass rate.
Inputs remain in your browser. Use the same, representative, independently labeled holdout cases—not prompt-tuning examples or repeated conversations.
One percentage is an estimate, not the whole answer
When a team tests an AI system on a sample of pass/fail cases, the observed pass rate is only one possible result from a larger workflow population. A confidence interval shows a range consistent with the sample under stated assumptions, so it makes small-sample uncertainty visible.
This tool uses the Wilson score interval rather than the simple symmetric “rate ± margin” shortcut. NIST notes that the Wilson method does not strongly depend on the pass rate or sample size and avoids impossible negative lower bounds that a common normal approximation can produce.
Use the interval to decide what to inspect next
- 1. Freeze the rubric. Define what a pass means before looking at the candidate's result.
- 2. Count independent cases. Do not inflate precision by counting repeated turns, duplicated documents, or prompt-tuning examples as separate evidence.
- 3. Compare the interval with the decision threshold. If the lower bound does not clear the minimum acceptable pass rate, collect better evidence rather than declaring success.
- 4. Inspect the failures. A narrow interval does not excuse a concentrated error on high-impact cases.
When a binomial interval is the wrong summary
The calculation assumes each result is a binary pass/fail outcome and that the evaluated cases are independent enough to treat as separate observations. It is not a test of whether two AI systems differ, nor a replacement for a multi-label, severity-weighted, or conversation-level analysis.
It also cannot validate a weak rubric or a biased test set. NIST's confidence-interval guidance explains the interval method; teams still need domain review for representative cases, rare severe failures, and release decisions.
Build evidence in the right order
First check whether reviewers apply the same rubric consistently. Then plan enough cases for a meaningful baseline-versus-candidate comparison. Only after that should a workflow move into a pilot.