In my first critique I basically skipped the section on Age of Diagnosis. I did not consider it the most important section, and the result (1.2-fold rise for the proportion of diagnoses by age 5) seemed plausible. The more I look at the paper, however, the more I come away thinking it's an exceedingly naive paper that got past peer review who knows how.
So I decided it was probably a good idea to have a closer look at the section on Age of Diagnosis. As it turns out, that section is also wrong.
Wrong AssumptionsWhat the paper does is compare the proportion of diagnoses before age 5 in the 1990 vs. the 1996 birth year cohorts. It finds that the proportion increased by only 12% in the 1996 cohort. Then it extrapolates from this to 2002. (I'll look at the extrapolation method later.)
That seems fine, right? You basically find out to what extent diagnoses by age 5 have changed, relative to all diagnoses you might expect to have in the cohort.
Except that's not what the paper does, nor would it be able to do that. What the paper looks at is the proportion of diagnoses by age 5 relative to diagnoses by age 10.
If there are few if any diagnoses after the age of 10, then that would work, correct? Intuitively, it seems reasonable that there wouldn't be too many diagnoses after the age of 10. But intuition and reality don't always agree. I knew that was an incorrect assumption because I've been looking at California DDS data for a number of years. (For example, see my post titled The Epidemic of Autism... Among 18-21 Year Olds.)
I have birth year data that California DDS provides on request (a file named Job5028.zip.) Let's look at the number of autistic clients born in 1990 as reported at different times.
In June, 1995 (approx. age 5): 404
In June, 2000 (approx. age 10): 663
In March, 2007 (approx. age 17): 918
Clearly, there is a non-trivial number of diagnoses after the age of 10. Of all the diagnoses by age 17, about 28% occur after the age of 10. There will no doubt be diagnoses after the age of 17 too.
Suppose things have changed since 1990. Perhaps in the 2002 birth year cohort close to 100% of California autistics are diagnosed before age 10. We can't know this, but if this were the case, I estimate that the impact of age of diagnosis would be about 1.6-fold and not 1.2-fold. With this, the total rise explained would get pushed over a factor of 5.
Of course, diagnoses after age 10 are confounded by changes in criteria. Some issues the paper has sort of compensate for one another, and this obviously makes it difficult or impossible to interpret the paper.
Wrong MathThe statistical analysis of age of diagnosis in the paper consists of exactly the following.
A shift toward younger age at diagnosis was clear but not huge: 12% more children were diagnosed before age 5 years in the 1996 birth cohort (the most recent with 10 years of follow-up) in comparison with those in the 1990 cohort.
Extrapolation into the later birth cohorts (eg, 2002) would suggest a 24% rise in the proportion of diagnoses by age 5.
Basically, they do a linear extrapolation: 12% for 1990-1996, then assume it's probably another 12% for 1996-2002, which gives a total of 24%.
Is a linear extrapolation reasonable here? What if there's an acceleration in the age of diagnosis after 1996?
It would be a good idea to look at the trend, wouldn't it? That's why I made the following graph of the proportion of clients at age 5 vs. those at age 10 for birth years 1990-1997.
You tell me, is a linear extrapolation reasonable there?
There are also considerable random fluctuations in the series, so the authors should have calculated a confidence interval on the slope of the linear regression, which is easy to do.
CommentLet me recap. There appear to be major issues throughout the paper.
Age of Diagnosis - As noted, the assumption that there are few if any diagnoses after age 10 is mistaken, plus the statistical analysis is basically non-existent and naive.
Changes in Criteria - It gets its result from a single Finnish epidemiological study of a population of intellectually disabled children. Finland and California are not necessarily equivalent genetically and environmentally. The ascertainment methods are also not equivalent in the least.
Milder Cases - It assumes that only Asperger's and PDD-NOS would have been missed by a study such as the Finnish one. (There also seems to be a contradiction as to what California DDS says in regards to Asperger's and PDD-NOS, and what the authors believe, which probably needs clarification; the contradiction was noted by Kev.)
Awareness - Not considered at all, but noted in the paper as an artifact that should be evaluated later.
Diagnostic substitution - Not addressed at all. The authors probably assume that diagnostic substitution is subsumed by the other artifacts, but it's non-obvious that this would be the case.
Migration - Dismissed in one paragraph as probably not having much of an impact.
Access - There's discussion on access, but no statistical analysis of its impact at all. It's unclear why it's included in the paper.
Statistical Analysis - Basically non-existent. No ranges of statistical confidence are provided. The authors seem to be under the impression that because they are looking at whole population numbers, there's no room for uncertainty in their figures.
Claims about results - The paper claims that artifacts account for a 4.26-fold rise, which does not come close to explaining a 6.85-fold rise. How so? Furthermore, if they had used a 3.6-fold figure for the impact of criteria (a figure from a meta-study), the entire rise would have been explained.
OK. I've read papers having to do with autism epidemiology that are quite poor. For example, I've read several papers by the Geiers. Even so, I'm debating whether H-P et al. is the worst such paper I've ever come across.
In my view, the credibility of the MIND Institute and that of the authors has dropped a notch with this paper. Perhaps a big issue has been the way the paper was described in the media. The language in the paper itself is somewhat skeptical in comparison.
I think they need to think about the implications of being associated with something so naive, mistaken, and so poorly communicated to the public. I wouldn't be surprised if some of the authors decide to retract the paper at some point in the future. That's also something the editors of the journal Epidemilogy should think about.