Case Study: Gardasil Vaccine

Summary:  The Gardasil vaccine is marketed by Merck to prevent cervical cancer. What are the statistics behind it? How do women decide whether to get vaccinated? Should the vaccine be mandatory?

Part 1: Is the Vaccine Effective?

A Cortland Standard (Nov 21, 2002) story summarized an article in the New England Journal of Medicine as follows

A new vaccine can protect against Type 16 of the human papilloma virus, a sexually transmitted virus that causes cervical cancer, a new study shows. An estimated 5.5 million people become infected with a strain of HPV [not necessarily this strain] each year in the United States.

Efficiency rate of vaccine and placebo

HPV-16 vaccine:  Group size 768, infection 0

Placebo:  Group size 765, infection 41

Note: The study included 1533 women with an average age of 20.

(Similar studies were done for the vaccine’s effectiveness against another strain, HPV-18. According to the front page of the Wall Street Journal on April 16, 2007, HPV-16 and -18 between them “are thought to cause 70% of cervical-cancer cases.” The vaccine, developed by Merck, would eventually be marketed as Gardasil.)

Hypothesis Test

The samples certainly show an impressive difference, but is it statistically significant? Could the luck of random sampling be enough to account for that difference in infection rates?

The claim is “the vaccine protects against HPV-16.” To translate this into the language of statistics, realize that there are two populations: (1) women who don’t get the vaccine, and (2) women who do get the vaccine.

If the vaccine works, then we expect more women without the vaccine to contract the virus, so we make them population 1. (That’s not necessary; it just usually makes things easier to call population 1 the one with higher numbers.)

Notice that the populations are all women, past, present, and future who do or don’t get vaccinated. The 765 and 768 women are samples, not populations.

The data type is attribute (binomial) because the original question or measurement of each participant is the yes/no question: “Did this woman contract the virus?” Since there are two populations, this is Case 5, Difference of Two Proportions.

Case 5 requires np̂(1−p̂) ≥ 10 for each sample. The samples in the news story don’t actually meet this requirement, but I’ll explain later why it’s still okay to proceed with Case 5 this time.

1. H₀: p₁ = p₂, the vaccine doesn’t make any difference
   H₁: p₁ ≠ p₂, the vaccine makes a difference

   Notice we ask whether the vaccine makes a difference, not whether infection rate is lower. We can’t prejudge. If H₁ was p₁>p₂, we would be asking whether the vaccine was better and in effect saying that if it’s not better then we don’t care whether it’s neutral or actually increases infection rates. That’s a bad way of thinking! (For more, see One-Tailed or Two-Tailed Hypothesis Test?)

α = 0.001, 2-tailed test

Remember that you use lower significance levels when the consequences of a Type I error are more severe. We pick a significance level of α = 0.001 because this is a cancer-causing virus. We don’t want to tell women to get the vaccine if there’s even a one in a thousand chance it won’t make a difference, lest they pass up a more effective treatment.

2. (The test statistic z_o is computed at the same time as the p-value on the TI calculator.)
3. 

   z_o=6.50, p-value=7.90×10^-11. There is less than one chance in ten billion (1×10^-10) of a Type I error. Another way to say it: if the vaccine actually made no difference, there’s less than one chance in ten billion of seeing a sample that shows this much difference between the vaccine and placebo groups. Chance can’t account for the difference in the samples.

   (A Type I error would be announcing that the vaccine makes a difference when actually it doesn’t. A Type II error would be announcing that the vaccine makes no difference when actually it does.)

   By the way, the three p̂s are the sample proportions. The one without the subscript is the blended or pooled proportion: 2.67% of the women in the combined samples got an HPV-16 infection. We’ll use that figure later.

    
4. p-value < α. Reject H₀ and accept H₁.
5. Conclusion: The vaccine does make a difference at the 0.001 level of significance; in fact it lowers the infection rate.

   See p < α in Two-Tailed Test: What Does It Tell You? for when and why to interpret a two-tailed test result in a one-tailed manner.

Assumptions

You’ll recall from class that some assumptions have to be met for a Case 5 hypothesis test or confidence interval to be valid. Different authors express somewhet different rules of thumb:

• The samples must be less than 5% of the population. (Some authors say larger than about 20 and less than about 10% of the population.)
• np̂(1−p̂) ≥ 10 in each sample. (Some authors say there should be at least 5 yes and 5 no in each sample.)

That first assumption is fine, but the second looks like a problem. There are no yeses at all in sample 2. Therefore p̂₂ = 0 and n₂p̂₂(1−p̂₂) = 0. Does that mean that the computed p-value is bogus? Not at all!

Remember that the stated assumptions are rules of thumb — if the conditions are met, your computation is most likely okay, and if they’re not met you may have a problem. In this case we don’t have a problem.

To see why not, think about the null hypothesis. It states that the vaccine makes no difference, in other words that the difference in infection rates between the two samples is just the result of chance in the random selection. If H₀ is true, then both populations have the same true proportion. The proportion in that population is somewhere in the neighborhood of our combined or pooled sample proportion:

p̂ = (0+41) out of (768+765) = 41÷1533 ≈ 0.0267

(It was shown as p̂ without subscript on the TI-83 output screen.)

If H₀ is true, we would expect about 20 or 21 infections in each sample (.0267×768). That expected value is comfortably above the threshold of 5. (Alternatively, using the pooled p̂ we have 765×0.0267×0.9733 = 19.88, comfortably above 10) The assumptions as stated in class are a quick rule of thumb, but the real issue in a test of difference in proportions is the expected value and not the actual sample x’s.

(My thanks to Bruce Weaver for explaining this point in a July 2003 article in sci.stat.edu, archived here.)

Part 2: Should Women and Girls be Vaccinated?

Obviously this test is statistically significant, meaning that p < α, but is it practically significant? In other words, what are the implications for women’s health decisions and for public policy?

The CDC has recommended the vaccine for women aged 11 to 26, and as of spring 2007 Texas and Virginia are requiring it of girls entering sixth grade.

Number Needed to Treat

One important factor in medical decisions is the number needed to treat (NNT). Compare the infection rates for untreated and treated women in the sample: 41/765 = 5.4%, and 0/768 = 0%. The difference is 5.4%. The difference in sample proportions is an estimator for the difference in population proportions. We know that the difference in HPV-16 infection rates for unvaccinated and vaccinated women in general will be in the neighborhood of 5.4%, though most likely not exactly that figure.

Invert the 5.4%: the number needed to treat is 1/0.054 ≈ 18.5. For every 18 or 19 women who get vaccinated, one case of HPV-16 infection will be prevented.

A 95% confidence interval on the difference in proportions gives 3.76% to 6.96%. Inverting those numbers gives a 95% CI for the number needed to treat: 14.4 to 26.6. We’re 95% confident that one case of HPV-16 infection can be prevented for every 14 to 27 women who are vaccinated, on average.

Remember that we’ve analyzed only the data from a trial of HPV-16 prevention. Considering HPV-18 as well, you would need to vaccinate even fewer women to prevent one case of infection by HPV-16 or -18.

Other Factors

There are other factors in the decision whether to have this vaccine, or any vaccine. This page is already long enough, so I’ll just mention a few of them briefly. Some have answers in statistics, some don’t.

• side effects (severity and likelihood): The CDC Web site says no serious side effects have been found.
• cost of the vaccine: $360, though probably covered by insurance if the woman has health insurance
• length of protection: Will a booster shot be needed, and if so when?
• availability of alternative treatments: Regular Pap smears are effective at finding precancerous lesions, but may lead to removing some lesions that never would have become cancerous.
• likelihood that HPV-16 or -18, untreated, will progress to cervical cancer. (We know that 70% of cervical-cancer cases are due to the virus, but we don’t know what percent of virus infections don’t lead to cervical cancer.)
• lifestyle: This is the controversial one, because the virus is sexually transmitted. Medically the ideal time for the vaccination is before a woman becmes sexually active.

So how should a woman decide whether to get the Gardasil vaccine (or any vaccine), and how should parents decide whether to have their girls vaccinated? This page can’t give “the answer” because there’no one right answer for everyone. The statisical evidence is the starting point for discussions with a person’s doctor.