Hypothesis Testing Made Simple
(for non-dummies)

There are a lot of fiddly details to hypothesis testing, but the basic idea is really simple:

There is some difference between your sample statistics and some baseline. If that difference is too great to be the effect of sample variability, you reject that explanation and conclude that some real effect is at work.

Suppose a control group gets a placebo and an experimental group gets an aspirin tablet every other day. Of 10,000 individuals in each, 192 in the control group get heart attacks, versus 107 in the experimental group. Your null hypothesis is that aspirin has no effect, and your alternative is that it does have an effect.

If aspirin has no effect, you’d expect the two groups to have about the same number of heart attacks — samples do vary, so you wouldn’t expect the numbers to be identical. The actual difference in these samples is 0.85% (1.92%−1.07%). Is that difference small enough that you can just rack it up to sample variability, or is it large enough to be considered unusual (or, as we say, significant)?

If the difference in samples is too large, sample variability isn’t a sufficient explanation. You reject the null hypothesis and accept the alternative, that aspirin does have an influence on heart-attack rates. But if the difference is not that large, it might be just sample variability. In that case, you fail to reject the null hypothesis and you say you can’t determine whether aspirin has an effect or not.

Everything else is just details.