The Age of Diagnosis Analysis is Also Wrong

I've written a theoretical critique of H-P et al. and also looked at the figures from the paper to see if in fact the artifacts the paper does take into account cannot possibly explain the rise observed (as emphasized in the media by the primary author and other persons associated with the MIND Institute.)

In my first critique I basically skipped the section on Age of Diagnosis. I did not consider it the most important section, and the result (1.2-fold rise for the proportion of diagnoses by age 5) seemed plausible. The more I look at the paper, however, the more I come away thinking it's an exceedingly naive paper that got past peer review who knows how.

So I decided it was probably a good idea to have a closer look at the section on Age of Diagnosis. As it turns out, that section is also wrong.

Wrong Assumptions

What the paper does is compare the proportion of diagnoses before age 5 in the 1990 vs. the 1996 birth year cohorts. It finds that the proportion increased by only 12% in the 1996 cohort. Then it extrapolates from this to 2002. (I'll look at the extrapolation method later.)

That seems fine, right? You basically find out to what extent diagnoses by age 5 have changed, relative to all diagnoses you might expect to have in the cohort.

Except that's not what the paper does, nor would it be able to do that. What the paper looks at is the proportion of diagnoses by age 5 relative to diagnoses by age 10.

If there are few if any diagnoses after the age of 10, then that would work, correct? Intuitively, it seems reasonable that there wouldn't be too many diagnoses after the age of 10. But intuition and reality don't always agree. I knew that was an incorrect assumption because I've been looking at California DDS data for a number of years. (For example, see my post titled The Epidemic of Autism... Among 18-21 Year Olds.)

I have birth year data that California DDS provides on request (a file named Job5028.zip.) Let's look at the number of autistic clients born in 1990 as reported at different times.

In June, 1995 (approx. age 5): 404
In June, 2000 (approx. age 10): 663
In March, 2007 (approx. age 17): 918

Clearly, there is a non-trivial number of diagnoses after the age of 10. Of all the diagnoses by age 17, about 28% occur after the age of 10. There will no doubt be diagnoses after the age of 17 too.

Suppose things have changed since 1990. Perhaps in the 2002 birth year cohort close to 100% of California autistics are diagnosed before age 10. We can't know this, but if this were the case, I estimate that the impact of age of diagnosis would be about 1.6-fold and not 1.2-fold. With this, the total rise explained would get pushed over a factor of 5.

Of course, diagnoses after age 10 are confounded by changes in criteria. Some issues the paper has sort of compensate for one another, and this obviously makes it difficult or impossible to interpret the paper.

Wrong Math

The statistical analysis of age of diagnosis in the paper consists of exactly the following.

A shift toward younger age at diagnosis was clear but not huge: 12% more children were diagnosed before age 5 years in the 1996 birth cohort (the most recent with 10 years of follow-up) in comparison with those in the 1990 cohort.
Extrapolation into the later birth cohorts (eg, 2002) would suggest a 24% rise in the proportion of diagnoses by age 5.

Basically, they do a linear extrapolation: 12% for 1990-1996, then assume it's probably another 12% for 1996-2002, which gives a total of 24%.

Is a linear extrapolation reasonable here? What if there's an acceleration in the age of diagnosis after 1996?

It would be a good idea to look at the trend, wouldn't it? That's why I made the following graph of the proportion of clients at age 5 vs. those at age 10 for birth years 1990-1997.



You tell me, is a linear extrapolation reasonable there?

There are also considerable random fluctuations in the series, so the authors should have calculated a confidence interval on the slope of the linear regression, which is easy to do.

Comment

Let me recap. There appear to be major issues throughout the paper.

Age of Diagnosis - As noted, the assumption that there are few if any diagnoses after age 10 is mistaken, plus the statistical analysis is basically non-existent and naive.

Changes in Criteria - It gets its result from a single Finnish epidemiological study of a population of intellectually disabled children. Finland and California are not necessarily equivalent genetically and environmentally. The ascertainment methods are also not equivalent in the least.

Milder Cases - It assumes that only Asperger's and PDD-NOS would have been missed by a study such as the Finnish one. (There also seems to be a contradiction as to what California DDS says in regards to Asperger's and PDD-NOS, and what the authors believe, which probably needs clarification; the contradiction was noted by Kev.)

What's left?

Awareness - Not considered at all, but noted in the paper as an artifact that should be evaluated later.

Diagnostic substitution - Not addressed at all. The authors probably assume that diagnostic substitution is subsumed by the other artifacts, but it's non-obvious that this would be the case.

Migration - Dismissed in one paragraph as probably not having much of an impact.

Access - There's discussion on access, but no statistical analysis of its impact at all. It's unclear why it's included in the paper.

Statistical Analysis - Basically non-existent. No ranges of statistical confidence are provided. The authors seem to be under the impression that because they are looking at whole population numbers, there's no room for uncertainty in their figures.

Claims about results - The paper claims that artifacts account for a 4.26-fold rise, which does not come close to explaining a 6.85-fold rise. How so? Furthermore, if they had used a 3.6-fold figure for the impact of criteria (a figure from a meta-study), the entire rise would have been explained.

OK. I've read papers having to do with autism epidemiology that are quite poor. For example, I've read several papers by the Geiers. Even so, I'm debating whether H-P et al. is the worst such paper I've ever come across.

In my view, the credibility of the MIND Institute and that of the authors has dropped a notch with this paper. Perhaps a big issue has been the way the paper was described in the media. The language in the paper itself is somewhat skeptical in comparison.

I think they need to think about the implications of being associated with something so naive, mistaken, and so poorly communicated to the public. I wouldn't be surprised if some of the authors decide to retract the paper at some point in the future. That's also something the editors of the journal Epidemilogy should think about.

Some Facts of Interest About the Numbers From the MIND Institute Study

I recently critiqued H-P et al., a paper by the MIND Institute that claims the rise in autism cases in California cannot be fully explained by a number of factors like age of diagnosis, changes in diagnostic criteria, etc. What bothered me most about the paper is an apples-to-oranges comparison between California DDS ascertainment and a Finnish epidemiological study.

You wouldn't know it from the conclusions, but the researchers also admit that they didn't consider a factor that ought to be important in a database like that of California DDS: awareness.

What I want to discuss in this post is something else. It's interesting that the researchers don't mention how much of the rise in "cumulative incidence by 5 years of age" (prevalence at age 5 really) is explained by the factors they did consider, taken together. You have to do the calculation yourself. Let's take a look at the numbers, shall we?

First, what's the extent of the rise? The conclusions of the paper say 7- to 8-fold. But the results from the abstract actually say the following.

Cumulative incidence to 5 years of age per 10,000 births rose consistently from 6.2 for 1990 births to 42.5 for 2001 births.

That's a 6.85-fold increase. Now, how much of this is explained by the factors the researchers considered?

Quantitative analysis of the changes in diagnostic criteria, the inclusion of milder cases, and an earlier age at diagnosis during this period suggests that these factors probably contribute 2.2-, 1.56-, and 1.24-fold increases in autism, respectively, and hence cannot fully explain the magnitude of the rise in autism.

If we multiply 2.2, 1.56 and 1.24, we get a 4.26-fold increase. That's not too bad. With all the problems the paper has, it actually explains 62% of the rise. It doesn't admit to that anywhere, but it does.

(If it's unintuitive why you have to multiply the factors, try the following mental exercise. Suppose the rise in "full syndrome autism" is 3-fold due to changes in criteria, and there currently are also 3 times as many autistics due to inclusion of Asperger's and PPD-NOS. Obviously, you have a 9-fold increase total.)

The interesting aspect of this has to do with the 2.2-fold factor due to changes in diagnostic criteria. I say it's interesting, because the researchers choose this one number, based on a single Finnish study, in favor of the results of a meta-study, Williams et al. (2006).

A meta-analysis of 37 studies of autism prevalence found a 3.6-fold higher risk from DSM-IV or ICD-10 criteria versus other criteria, but this figure would have been confounded by the year of study.

I'm not sure what they mean by the year-of-study confound; they don't explain it. They could very well be right. I don't know.

But just for kicks, let's see what would've happened if the researchers had chosen the 3.6-fold factor instead of the 2.2-fold factor. That is, we multiply 3.6, 1.56 and 1.24.

In this case, the study would explain a 6.96-fold rise. The actual rise was 6.85-fold. In other words, the study would have explained 102% of the rise. Does anyone else find that kind of suspect and hilarious at the same time?

The MIND Institute's Second Attempt: More of the Same Type of Reasoning

The MIND Institute has published a new study titled The Rise in Autism and the Role of Age at Diagnosis (hereby referred to as H-P et al.) Among its conclusions: "Younger ages at diagnosis, differential migration, changes in diagnostic criteria, and inclusion of milder cases do not fully explain the observed increases." As you can imagine, the paper is being cited uncritically in the usual places.

I would like to discuss what it is the paper finds, how it finds it, and whether the findings are accurately characterized. Before I do that, however, I think some background is in order.

Back in 2002, a MIND Institute report to the California Legislature concluded basically the same thing: That a loosening of the criteria had not contributed to the rise in autism diagnoses in California. Back then the MIND Institute report was considered evidence of the "autism epidemic." But it contained a significant error in reasoning, first noted in Gernsbacher, Dawson & Goldsmith (2005).

Basically, the 2002 report found that an earlier cohort of children met DSM-IV criteria at about the same rate as a more recent cohort of children. As Gernsbacher et al. patiently explain, it's not surprising that the more recent DSM-IV criteria is met by nearly all the children, the younger ones and the older ones. What would be more relevant to find out is whether the older children meet some narrower criteria that the younger children do not meet. Gernsbacher et al. illustrate the fallacy using a height analogy and call the researchers' conclusions "imprudent."

There are two main areas of the new paper that I want to discuss: Changes in diagnostic criteria, and inclusion of milder cases (birth cohorts 1990-2006). The paper looks at other areas, but I'd like to focus on the two that are key.

Changes in diagnostic criteria

Finding: Changes in diagnostic criteria have contributed a 2.2-fold increase in the rise of autism incidence in California.

How is this determined? This being the most important area of the paper, my expectation was that I would find new data produced by H-P et al. showing that only a 2.2-fold increase may result from changes in diagnostic criteria. This is not the case at all. H-P et al. rely on prior work.

Specifically, they rely on Kielinen et al. (2000), a study out of Finland based on data "collected from hospital records and the records of the central institutions for the intellectually disabled in the Provinces of Oulu and Lapland in 1996–1997."

The 2.2-fold increase is the difference found by Kielinen et al. when comparing autistics diagnosed with Kanner's criteria vs. those diagnosed with ICD-10 or DSM-IV.

So what's wrong with this result? You need to ask yourself if the results from Kielinen et al. are applicable to changes in the California DDS autism population.

You see, even if we assume that all California autistics from the birth cohort 1990 were diagnosed with Kanner's criteria, why should we assume that all persons matching this criteria were identified and registered with California DDS? Isn't it more likely that only a small minority of such persons were recognized at the time? Furthermore, Kielinen et al. only look at the population with autism or other psychiatric conditions as recorded in known databases. California DDS has a much broader population pool to draw from, which would be more than relevant at the present time.

Also note that the DSM-IV prevalence found by Kielinen et al. is low compared to that of other DSM-IV studies: 20.7 in 10,000.

In essence, I do not believe there is any basis for comparison of Finnish ascertainment done in a 2000 study vs. California DDS ascertainment, either in 1990 or in 2006.

Inclusion of milder cases

Finding: Inclusion of milder cases contributes a 1.5-fold increase in the rise of autism incidence in California.

How is the result determined? H-P et al. again rely on work from a separate study: The Childhood Autism Risks from Genetics and Environment study. In this study, 64% of California cases were confirmed to meet criteria for autism (excluding Asperger's and PDD-NOS) when evaluated using two separate diagnostic instruments.

I was initially confused as to why inclusion of "milder" cases is necessary to take into account, if the paper already claims to have looked at changes in diagnostic criteria. The reason is probably that Kielinen et al. would not have studied PPD-NOS or Asperger's cases.

So what's wrong with the result? If you're trying to account for cases that Kielinen et al. would have missed, I'm not sure that PDD-NOS and Asperger's are enough. What about high functioning autism in general? That's not the same thing.

There's also the question of why only "milder" cases should be considered. Why not more severe cases? Should we assume that Kielinen et al. would've detected all of those? Then there are cases that are neither more severe nor milder but just not diagnosed as autism traditionally. For example, would Kielinen et al. have recorded cases of autism in Down Syndrome or Cerebral Palsy, where California DDS obviously does?

In the end, this particular result does not tell us much. What it says is that if you exclude all autistic children who should not be eligible for California DDS services, the impact on the number of cases is not too great. I could've told you that.

Did They Miss Anything?

One question that comes to mind is why the researchers did not discuss diagnostic substitution, particularly from mental retardation.

Would this be part of the "changes in diagnostic criteria" analysis? Yes and no. To the extent that children previously diagnosed with mental retardation are currently diagnosed with autism because DSM-IV criteria says so, then yes, the analysis by H-P et al. would suffice (assuming that analysis were valid, which I don't believe it was). If there are reasons for the shift that go beyond criteria, then no, it is not enough.

Diagnostic substitution in California is interesting for other reasons. I think there's sufficient data there to do a proper analysis. For example, you can look at cases of mental retardation without autism over time and see if they decline. I have reason to believe such an analysis will be available in the near future.

Comment

H-P et al. is a surprisingly poor paper. It does not produce any new data in order to support its two main results. It makes an apples-to-oranges comparison between a Finnish epidemiological study and California DDS ascertainment over time. It tells us the obvious about "milder" cases. In the end, I don't think this is an improvement over the 2002 MIND Institute report to the California Legislature. In fact, it could very well be worse.

The way H-P et al. have gone about trying to show there's a real rise in autism incidence over time is not a very good way to go about doing things, in my view. There are other ways. For example, I've suggested trying to replicate Lotter (1967) in detail. This would not be as easily challenged.