Here is a box. Inside are black balls and white balls — plenty of both, mixed in some proportion that nobody will tell you. It might be half and half. It might be nine to one. You are not allowed to tip the box out and count. All you may do is shake it, draw one ball, note its colour, drop it back in, and repeat.
The question sounds like a puzzle for children and is secretly the question behind most of applied statistics: how many draws do you need before you can state the true proportion with confidence? A poll asking a thousand voters, a factory testing widgets off the line, an A/B test counting clicks — each is this box wearing a different costume. The balls are people, parts, or page-views; the colours are yes/no, pass/fail, clicked/didn't. So let's answer it properly, for the box, and everything else comes along for free.
01 One ball at a time
The obvious strategy is the right one: keep drawing and keep a running tally. After n draws, your estimate of the proportion of black balls is simply the black count divided by n. The law of large numbers promises that this running estimate settles ever closer to the truth as the draws pile up. What it does not promise is a smooth ride: early on, the estimate lurches drunkenly with every ball.
Try it. The box below has a true mix I have hidden from you. Draw a ball, then a hundred, and watch your estimate wander, then steady. When you think you know the answer, reveal the truth and see how the whole path sits inside a narrowing funnel.
Two things are worth noticing. First, the estimate does settle — always, for any mix. Second, how it settles: violently at first, then grudgingly. Going from 10 draws to 100 transforms your estimate; going from 1,000 to 1,090 barely nudges it. That slowdown is not bad luck. It has an exact shape, and the shape is the answer to our question.
02 The size of the wobble
Suppose you fix a budget of n draws, take your estimate, and stop. How wrong should you expect to be? One run can't tell you — you'd need to run the whole experiment many times and watch where the estimates land. That spread has a name, the standard error, and for a proportion it has a formula: √(p(1−p)/n), where p is the true fraction of black balls.
Below, the box's mix is now known and yours to set. Choose a budget of draws, run the experiment a thousand times, and watch the estimates pile up around the truth.
Play with it and the formula's two levers show themselves. The √n in the denominator is the slowdown from section one made precise: to halve your typical error you must quadruple your draws. And the p(1−p) on top says the box's own mix matters — it is largest at 50/50, where every draw is maximally uninformative, and shrinks as the mix gets lopsided. A 90/10 box gives up its secret with far fewer draws than an even one, which is why close elections are precisely the ones polls struggle with.
03 The price of precision
Now run the logic backwards. You don't want to know how wrong n draws leave you; you want to pick a tolerable error and ask what it costs. Say you want your estimate within ±3 points of the truth, with 95% confidence. Reach out about two standard errors (1.96, to be fussy), set that equal to your margin, and solve for n. Since you don't know the mix in advance, budget for the worst case, 50/50, and the formula collapses to something you can do on a napkin: n = (z / 2m)², for margin m and confidence multiplier z.
Two morals fall out of the arithmetic. First, the famous poll number: ±3 points at 95% confidence costs about 1,100 draws — which is why serious national polls interview roughly a thousand people, whether the nation holds five million or five hundred million. The box's size never entered the formula; only the number of draws did. Second, the tariff on precision is brutal: ±3 costs a thousand draws, ±1 costs ten thousand, ±0.5 costs forty thousand. Every halving of the margin quadruples the bill, forever.
04 So what?
The box gives you a portable answer. A few dozen draws buys you a caricature of the truth; a few hundred buys a decent sketch, within five points or so; a thousand buys the ±3 that news polls call accurate; and past that you are paying quadratically for every extra digit. When someone quotes you a percentage, the first question is always the same: how many draws? A "73% success rate" from eleven trials and one from eleven hundred are different claims wearing the same number.
One loose end deserves its own essay — and already has one. Throughout, "95% confident" carried a precise, slippery meaning: a promise about the long run of the procedure, not about any single interval you happen to hold. What that promise does and does not entitle you to say is the subject of Ninety-Five Per Cent Confident?, which builds the machinery — sampling distributions, the central limit theorem, the t-distribution — that this essay quietly leaned on.
And the box, of course, was on its best behaviour. Every ball went back before the next draw, the mix never changed mid-experiment, and each ball was drawn blind. Real sampling breaks each of those promises in turn — the box that changes while you sample it, the draws that are secretly not independent, the balls someone chooses to show you. Those are the next notes in this series.
You never needed to empty the box. You needed to know how hard to shake it, how long to draw, and what the counting could and couldn't promise.