Statistical Shenanigans with Gardasil Research

Something’s hinky in the land of Gardasil surveillance and it’s not just the vaccine. It’s the statistics used in one of their widely reported surveillance studies on Gardasil-suspected autoimmune conditions. From the medical marketing and even the abstract of the study, which is all that most will ever read, it looks like there is no connection between the vaccine and the onset of autoimmune conditions. Indeed, all of the major media reported the appropriate PR:

• HPV vaccine not linked to autoimmune disorders
• Study: No link between HPV vaccine and autoimmune disorders 
• HPV vaccine unlikely triggered autoimmune disease
• No increase in autoimmune conditions from use of HPV vaccine

One organization even failed to change the url before directly copying the press release from another agency; (guess which one, I was surprised). It is unlikely that anyone read this study before blasting the PR far and wide. Otherwise, if they had read the dubious and creatively contorted statistical manipulations, the headlines would have been much different. Perhaps, Researchers Choose Statistics Least Likely to Find Results in Gardasil Study, would have been more appropriate.

I have a nasty habit of reading research before I write about it. I’m sure it will get me in trouble at some point, particularly as this blog grows, but I can’t help it. I want to understand the research. So here you go, my opinion and review on the Surveillance of Autoimmune Conditions Following Routine use of Quadrivalent Human Papillomarvirus Vaccine.

Gardasil and Autoimmune Conditions

Anecdotal evidence abounds suggesting that Gardasil is linked to an increased incidence of a wide array of autoimmune responses, but aside from the convoluted VAERs  tabulations, very little in the way of actual data exist.  In Merck’s FDA-mandated post market surveillance of Gardasil (which should have been pre-market testing if you ask me), the initial study conducted at Kaiser Permanente in California, purports no connection between 16 autoimmune conditions and the Gardasil vaccine.  The conditions investigated included:

• Immune thrombocytopenia
• Autoimmune hemolytic anemia
• Systemic lupus erythematosus
• Rheumatoid arthritis
• Juvenile rheumatoid arthritis
• Type 1 diabetes
• Hashimoto’s disease
• Graves’ disease
• Multiple sclerosis
• Acute disseminated encephalomyelitis
• Other demyelinating diseases of the central nervous system
• Guillain-Barré syndrome
• Optic neuritis
• Uveitis

Though not a complete list, it’s a start and if that were the only flaw in the research, I wouldn’t be writing about this study, but alas, it is not. There are several hinky statistical maneuvers that make it all but impossible to draw any meaningful conclusions from this research.

Statistical Shenanigans

Hinky maneuver #1. The PR reports and the study abstract indicate that the study followed almost 190,000 women for 180 days after each vaccine dose. This would be a great study if it were true. It is not. Researchers reviewed a possible sample of 190,000 medical records (not actual girls and women, but their records, flawed as they may be) from girls and women who had received at least one dose of the Gardasil vaccine. They subsequently removed all records from patients who had not been Kaiser health plan members for at least a year, leaving 149,000. Still a large number of health records, but not the reported 190,000 girls and women followed post vaccine.

Hinky maneuver #2. From those records, they looked for possible new onset cases of the aforementioned diseases within what they defined as a risk period of up to 180 days post vaccine. Read beyond the abstract and we find that, not each dose received the same time frame. The risk period for the first dose was 60 days. This is problematic because a large percentage of patients chose not to receive all three doses and it often takes multiple months to receive a proper autoimmune disease diagnosis. Those data were not reported here and so, it is not clear how many cases may have been missed by essentially compressing the time of onset to within 60 days post first dose.

Hinky maneuver #3. From the 149,000 health plan members, 1014 potential new onset autoimmune cases were identified, 719 were deemed eligible for review (no definition of eligible was given) and only 347 were sampled for case review: “Because of the large number of potential new-onset cases identified for SLE (systemic lupus erythematosus, RA (rheumatoid arthritis ), JRA (juvenile rheumatoid arthritis), Hashimoto’s and Grave’s disease, a random sample of potential cases for these conditions was included for case review.”  For the other conditions, which apparently didn’t have as high an incidence of new onset cases, all cases were included. Beyond the possibility of the admittedly ‘large number of potential new onset cases’ no reasons for reviewing only the sub-sample of the cases was provided, nor was an explanation of their ‘random’ sampling technique. What is clear though, is that by superficially limiting the number of cases, one skews the statistics and limits the ability to detect differences between the vaccine-related autoimmune and the non-vaccine autoimmune diseases.

Hinky maneuver #4 – the granddaddy of hinky maneuvers. Rather than do a matched case control study or a myriad of other possible, more logical and more powerful study designs to compare the rates or risks for onset of these conditions, this study (mis) used statistical techniques designed to manage missing data points (within a larger data sets), to effectively substitute a control group. There was no control group in this study. It was a statistical manipulation and a poor one at that, unless of course, the goal is to not find statistical differences, then it was a pretty creative choice.

Using a statistical procedure called Rubin’s multiple imputation, the researchers effectively (though not technically, for you stats wonks) estimated the data for the entire control group. According to the study authors,  “by treating the actual status of new onset as missing data for the un-reviewed potential cases,”  they imputed (substituted) 500 cases to come up with median, control group data that was then compared to the vaccine group data of 347 cases (the random sample of potential cases).

If this sounds really hinky, that’s because it is. By imputing the control group, they guaranteed that they would not find differences between the unvaccinated and vaccinated group. Indeed, they changed the question entirely. Their comparisons are no longer about the rates of onset between those vaccinated and those not vaccinated, but rather between those vaccinated verses those “estimated to potentially have a condition based on data imputation” which may or may not be the same as for those in the true unvaccinated population. This is completely different finding than what was reported in the study’s abstract, which is probably the sum total of what most PR and media companies read before regurgitating it endlessly through the news streams. And in that regard, this ‘study’ served the vaccine maker well, but it certainly does not serve the health of the public or scientific process at all.

Post Script. Poking through the statistical shenanigans, if we look only at the raw numbers from the abridged case pool, there appeared to be a high rate of Hashimoto’s, rheumatoid and other autoimmune conditions. However, without a properly designed study, it is difficult to delineate an expected incidence in the given population versus the observed incidence.