Monday, September 03, 2007

A Simple Selection Bias Model Explains Generation Rescue's Survey Results

A while back Kev, Prometheus and Orac discussed how underwhelming Generation Rescue's survey results were and some peculiarities of the data that seem to invalidate the survey. What I want to do in this post is go over a model that explains the survey results, including discrepancies between its findings and those of prior phone surveys. Data of this nature can sometimes result in knowledge that wasn't expected, as I will hopefully demonstrate. (I am using data Kev helpfully put into an XLS file here).

ASD - Both Sexes

Strikingly, the survey found that while 3.01% of all vaccinated children had an ASD diagnosis, about 3.73% of all unvaccinated children did. That's right. The survey found autism to be more common among the unvaccinated. While this difference is not statistically significant, I think Kev was correct when he characterized it as disastrous for Generation Rescue's political goals. It also contradicts years of claims by GR itself and Dan Olmstead in regards to the Amish and so on.

Later on I will explain how they managed to spin this result.

[Note: 'Statistical significance' means that we can assert the survey found a difference, with 95% confidence.]

As noted shortly after Kev posted his analysis, both ASD rates are very high relative to the consensus prevalence of autism. Let's compare them to a recent CDC phone survey which found the diagnosed prevalence of autism in the US to be 0.57%.

What might explain the discrepancy? For that, we need to consider the survey's methodology. This was an automated phone survey. That is, households are contacted at random by a computer with a short recorded introductory message, in this case one that read as follows.

This is SurveyUSA calling Sonoma County parents with a private, confidential survey about vaccinations and children's health. If you have a child age 4 to 17, press 1. Otherwise, press 2.


Most persons answering the phone will hang up at this point. But that would be fine if the introduction weren't so obviously biased. Evidently, parents of autistic children who are familiar with the anti-vaccination debate will be more likely to continue with the survey. In fact, given the results of the survey, it is clear that households with autistic children are about 5 times more likely to continue with the survey than households in the general population. (We can also conclude something about SurveyUSA's best-case response rates, but that's not important to this analysis).

This bias in the introductory message ('introduction bias' from now on) easily explains the high prevalence of ASD found in the vaccinated population of children. Consider that the vast majority of children are vaccinated.

I do not, however, believe it fully explains the high rate found in the unvaccinated group. I say this because households with unvaccinated children are no doubt likely to continue with the survey, regardless of the presence of autistic children. This is a key point. In fact, Generation Rescue was somewhat surprised to find 6% of children were completely unvaccinated. Heightened interest in the part of parents of autistic children is not going to be as significant a factor in the unvaccinated group.

Therefore, a second bias, in the opposite direction, seems necessary to explain the high prevalence of autism (6.5 times the CDC prevalence) in the unvaccinated population of children. I will call this the 'genetic bias', one that is well known. That is, parents of autistic children will often not vaccinate subsequent siblings, and such siblings have a considerable higher rate of autism compared to the general population. Even in the relatively small group of parents who blog, I know of two parents who have unvaccinated autistic children: Not Mercury and Kim Stagliano.

ASD - Boys vs. Girls

In boys, vaccination was found to be a non-significant risk factor relative to ASD (OR = 1.17). In girls, however, vaccination was found to be a statistically significant "protective" factor (OR = 0.37). That is, girls are almost 3 times more likely to be diagnosed with ASD if they are not vaccinated, according to the survey results.

The Generation Rescue leadership must not believe the results of their own survey. Otherwise, I'd expect them to post a warning such as the following on their website.

Please Note: We believe vaccines pose a non-significant risk of autism in boys, but they are a protective factor when it comes to girls. We urge parents to vaccinate girls.

OK. They might believe only some of their results, or they might have decided that throwing girls under the bus is the politically beneficial course of action.

It seems far fetched that something would be a risk factor in boys, but a protective factor in girls. From a quick Google search I conclude this would be unprecedented. Nevertheless, the boy vs. girl discrepancy in the survey is of interest. How might it be explained?

I first checked if the introduction bias is the same across boys vs. girls. It turned out to be close. I calculated the CDC prevalence of autism for boys vs. girls given the ratio reported in the press as "nearly 4:1". The resulting prevalence was 0.91% for boys and 0.23% for girls. The resulting bias factors (survey prevalence divided by CDC prevalence) were 5.07 and 5.57 respectively. This tells me that parental interest in the anti-vaccination issue is not significantly skewed by the child's sex.

It follows that the genetic bias must be skewed, and this does appear to be the case. The bias factors in the unvaccinated group were 4.33 for boys and 15.17 for girls. That's quite surprising. To emphasize, unvaccinated girls participating in the survey were found to be diagnosed with ASD 15 times as often as in the general population of girls.

Either not vaccinating girls is a significant cause of ASD, or familial autism is more common in the families of autistic girls. (Any other explanations?) I'm inclined to believe the second explanation, though I haven't found much support for this in the literature. Tsai et al. (2005) comes close. Either way, this seems to be an interesting direction of future research.

"Like" Autism and ADHD

So far we have noted that the survey did not find vaccines to be a significant risk factor for ASD, either in boys or children in general. It did find vaccines to be a "protective" factor in girls. Despite this, Generation Rescue made the following claim.

We surveyed over 9,000 boys in California and Oregon and found that vaccinated boys had a 155% greater chance of having a neurological disorder like ADHD or autism than unvaccinated boys.

(source) (Emphasis mine)

What they did is aggregate the data for ADHD and autism and made a general claim about "neurological disorders". The claim is technically true. I'm sure they could've aggregated asthma and diabetes and made a general claim about health outcomes, but that would've been too transparent. The statement is clearly not a bastion of honesty. (I'm sure readers can tell I'm holding back on my characterization of the statement).

Another thing they did in their survey information page is cherry pick the highest risk ratio they found for boys with ASD, the one for plain autism, which is 1.61. They did not mention the ASD risk ratio, 1.17, which not as impressive, or the risk ratios for PDD-NOS or Asperger's in boys.


The survey did find significant risk ratios in the ADHD population, and these deserve to be analyzed in their own right. About 11.02% of the vaccinated children were reported to have ADHD vs. 5.15% of the unvaccinated children.

[Note: What I'm calling ADHD here corresponds to both ADD and ADHD in the survey results. Concensus in Psychiatry is that there's just one condition called ADHD and not two.]

The rate of ADHD in the vaccinated group is a bit high, although not by much. A 2003 CDC survey found that 7.8% of all children in the US had an ADHD diagnosis. (It did not vary a lot from state to state). This tells me that the introduction bias exists in ADHD too, but is not as significant as in ASD households, and this makes sense. The vaccine hypothesis is not as politically hot in the ADHD world. It follows that the genetic bias will also not be significant, because it's unlikely that parents will stop vaccinating subsequent siblings based on an ADHD diagnosis.

We're still left with a relatively low prevalence of ADHD in the unvaccinated population. But I think this can be easily explained by a third bias, which I will call the 'healthcare bias'. That is, households where children are not vaccinated are also less likely to seek diagnoses of ADHD. This is quite plausible when it comes to ADHD, given that it's such a subjective and controversial diagnosis. This bias probably exists in ASD as well, but I'd suggest it can't be nearly as common.

Partial Vaccination

The survey included a category called "Partially Vaccinated." No doubt Generation Rescue's intention was to find some sort of dose-response relationship between ASD and vaccination. But they did not. What they found is that ASD is over-represented among the partially vaccinated, more so than among both the vaccinated or unvaccinated. This is an obvious finding, in retrospect, considering that many parents will stop vaccinating a child once he or she is diagnosed with ASD. The survey designers must not have considered this problem.

Comment and Predictions

Given that fairly obvious biases explain the survey results, and given that the risk factors involved are underwhelming, I do not believe that public resources should be spent on a methodologically valid follow-up. Generation Rescue could decide to invest on a follow-up, but I see a couple problems with this: (1) Eliminating the introduction bias will result in a much more expensive survey, as more calls (probably 5 times as many) will need to be made to achieve the same level of statistical significance; (2) Generation Rescue probably realizes that by eliminating the introduction bias, vaccines will be found to be a clear "protective" rather than a risk factor for ASD. (Yes, you may consider that a "dare").

Generation Rescue has brushed off the introduction bias. I don't know if they realize this bias is only unimportant if it affects both the vaccinated and unvaccinated populations equally. I have explained why it obviously does not.

Either way, a phone survey can only tell us about the approximate rates of diagnosed ASD in a population. It doesn't tell us much about the true prevalence of ASD. For that, we would need a whole-population screening. Even then, other biases, such as the genetic bias, need to be accounted for.

The most interesting finding of the survey, in my view, is that unvaccinated girls are 15 times as likely to have ASD than expected based on prior surveys. This is probably not explained by the introduction bias. I believe it would have to be explained by an extraordinary genetic bias or some other bias not considered thus far.


  1. Excellent analysis of many of the sources of bias that are not considered (or reported) when these types of surveys come out.


  2. Thanks Kev and Joe.

    Something interesting just posted by Brad Handley (source):

    "Three years later,we are the proud parents of a beautiful baby girl, and we feelwell-armed with the wisdom of other parents and many doctors toprevent her from the same fate her brother experienced..."

    Handley says he will do all sorts of (clearly untested) things to the baby girl to try to prevent her from developing autism. One of them is:

    "- Avoiding all vaccines for at least the first 2 years of life, andthen taking extraordinary caution"

    OK. My first comment is that, again, JB Handley either did not understand the results of his survey, ignores them, or he does not believe them. According to his survey, unvaccinated girls are almost 3 times more likely to develop ASD than vaccinated girls.

    It's very likely the girl will not have ASD, without doing anything special. The odds are something in the order of 90% to 98%. When what is almost certain comes to pass, JB Handley will no doubt say "I told you so." In the very unlikely event the girl gets an ASD diagnosis, we'll say, "WTF?"

  3. About the idea that vaccines could provide some protection against autism, I admit that it wouldn't be my first guess, but it did remind me of something from the New York Times that I read that I don't think you mentioned.

    "A study by the World Health Organization, for example, examined the health records of 109,863 children born in Britain from 1988 to 1997 and found that children who had received the most thimerosal in vaccines had the lowest incidence of developmental problems like autism.
    Another study examined the records of 467,450 Danish children born from 1990 to 1996. It found that after 1992, when the country's only thimerosal-containing vaccine was replaced by one free of the preservative, autism rates rose rather than fell."

    From the New York Times June 25, 2005, On Autism's Cause, It's Parents vs. Research by Gardiner Harris and Anahad O'Connor

  4. Excellent post. Thank you, very much for the analysis.

  5. This is excellent information.

    Thank you!!



  6. The link for Autism Assembly is broken.

  7. Hi Donna,

    Good memory!

    I *think* (but have not gone to the source to check) that in the first study mentioned in the NYT article you quoted, the apparent protective effect of thimerosal went away when socioeconomic factors were controlled for. Which correlates with Joseph's hypothesis about healthcare utilization bias explaining the elevated rate of ADHD diagnosis in the fully vaccinated.


    Fine piece of work here! Handley is lulu. He has long since crossed from his "I'm a Republican just like you" persona to that of a a full-blown conspiracy theorist.

  8. Donna - I don't really believe there's a protective effect. If the child actually gets measles or something because they were not vaccinated, I suppose cognitive disability may occur. But that's not what we're talking about with the vast majority of unvaccinated children (protected by herd immunity).

    The "protective" effect is most likely explained by some bias, as Isles notes.

    In this particular survey, a genetic bias would be enough to explain any protective effect. What's unusual about it is the discrepancy of the effect in boys vs. girls.

    A hypothesis I didn't write about in the post is that parents of autistic children could be aborting male fetuses. This has actually been suggested as a way to "prevent autism". (Maybe parents were doing it even before it was suggested).

  9. "A hypothesis I didn't write about in the post is that parents of autistic children could be aborting male fetuses. This has actually been suggested as a way to "prevent autism". (Maybe parents were doing it even before it was suggested)."

    Some were. Where I live.

  10. @Most posts here have to do with the author's view that autism is not a new man-made phenomenon, but instead a natural part of our species that has always been with us"

    You must be unaware that Leo Kanner, when he wrote his landmark 1943 paper in which autism was first described, announced

    "Since 1938, there have come to our attention a number of children whose condition differs so markedly and uniquely from anything reported so far..."

  11. @Anon: Keep practicing your selective quoting. Emphasis should be in the word reported. Here's what Kanner says afterwards in the same paper:

    "These characteristics form a unique “syndrome,” not heretofore reported, which seems to be rare enough, yet is probably more frequent than is indicated by the paucity of observed cases. It is quite possible that some such children have been viewed as feebleminded or schizophrenic. In fact, several children of our group were intorduced to us as idiots or imbeciles, one still resides in a state school for the feebleminded, and two had been previously considered as schizophrenic."