Tuesday, November 24, 2009

A Word About Hertz-Picciotto & Delwiche (2009)

The recent Chicago Tribune Articles have resulted in a flurry of reactions, particularly from AoA. I'd like to discuss just one of them in this post. Kent Heckenlively was complaining that the Chicago Tribune reporters did not give weight to any of the documentation he had provided them. Specifically, to support the assertion that we are in the midst of an autism epidemic, Heckenlively cited a press release by the MIND Institute.

Why cite the press release and not the paper? Well, that's because the conclusions of the paper said the following.

Other artifacts have yet to be quantified, and as a result, the extent to which the continued rise represents a true increase in the occurrence of autism remains unclear.


A true increase... remains unclear. Is that clear enough?

The paper is titled "The rise in autism and the role of age at diagnosis." I'll refer to it as Hertz-Picciotto & Delwiche (2009). I've already criticized multiple aspects of the paper (last time here) but I'd like to say a few more things about it.

The conclusions of the paper seem refreshingly honest, but I'm guessing they are that way simply to get through peer-review. The missing artifacts are not just any artifacts, either. One of the artifacts that did not enter the calculations is key: awareness. It would be nonsensical to assume that awareness of autism has not changed since the early 1990s.

But what did the press release say?
“It’s time to start looking for the environmental culprits responsible for the remarkable increase in the rate of autism in California,” said UC Davis M.I.N.D. Institute researcher Irva Hertz-Picciotto, a professor of environmental and occupational health and epidemiology, and an internationally respected autism researcher.

Hertz-Picciotto said that many researchers, state officials and advocacy organizations have viewed the rise in autism's incidence in California with skepticism.

(source)

This is completely at odds with the conclusions of the paper, and I find it quite dishonest. To make it perfectly clear, yes, I'm accusing Dr. Hertz-Picciotto of intellectual dishonesty.

Dr. H-P further states that:
“These are fairly small percentages compared to the size of the increase that we’ve seen in the state,” Hertz-Picciotto said.

Is that even true? The paper finds that a 2.2-fold increase may be explained by changes in diagnostic criteria, 1.56-fold due to inclusion of "milder" cases, and 1.24-fold due to changes in age at diagnosis. If you multiply these factors, you come up with a 4.26-fold increase that may be explained by just these 3 artifacts.

That's 62% of the entire rise the authors were attempting to explain. Maybe 62% is a "fairly small percentage." In this case I'm going to give Dr. H-P the benefit of the doubt and say that she likely didn't know the factors needed to be multiplied. There are no indications in the paper that a calculation of the overall contribution of the 3 artifacts combined was even attempted. I won't even get into statistical uncertainty.

The flaws of the analysis are not what bother me the most, however. Consider this. What is the value of this study? Why was it carried out? If you're trying to determine whether artifacts can explain the rise in autism service classifications in California, and you cannot estimate the contribution of all relevant artifacts, what is the point of the analysis?

Here's an analogy. Suppose you wanted to determine if global warming can be explained by the greenhouse effect. In order to do this, you estimate the contribution of methane, water vapor and nitrous oxide to recent temperature increases, but you leave out CO2. You conclude that those 3 gases alone cannot explain the entire rise in temperatures, but perhaps that's because you did not consider the biggest contributor to the greenhouse effect: CO2. Then you tell the media that greenhouse gases cannot fully explain the rise in temperatures.

Wouldn't such a study be better understood as a propaganda effort, rather than a contribution to scientific knowledge?

Someone might complain that Hertz-Picciotto & Delwiche (2009) does contribute to scientific knowledge, because it tells us about the impact of certain artifacts. But does it actually do that?

Take what is perhaps the most important artifact the paper does take into account: changes in diagnostic criteria. What would you do if you wanted to determine the impact of changes in criteria? You might carry out a prevalence study with good case-finding that uses two different criteria on the same population: DSM-IV and DSM-III (or perhaps Kanner criteria.) Hertz-Picciotto & Delwiche (2009) does not do anything that even resembles this. They use data from a separate study, so they didn't even contribute new data. We can't even be sure how well the data from Finland might apply to California. The case-finding of the Finnish study is not necessarily very good either. Plus it's just one data point, with all the uncertainties implied by that.

To determine the impact of inclusion of "milder" cases (i.e. anything that is not "autistic disoder"), what would you do? I think you could evaluate a random sample of CalDDS autistic children and diagnose them with either autistic disorder, PDD-NOS or Asperger's. What the researchers did instead was use data from a separate MIND Institute study where CalDDS children had been evaluated with the Autism Diagnostic Observation Schedule (ADOS) and the Autism Diagnostic Inventory (ADI.) Are these diagnostic tools even able to accurately distinguish between autism spectrum diagnoses? I'm not aware of any evidence that they do.

In other words, the MIND Institute study is not even informative about the impact of the artifacts it did take into account. I frankly can't see this study as a contribution to scientific knowledge at all. It gives the appearance of being part of a propaganda effort.

13 comments:

  1. What do you think about the autism increase that occurred after removal of thimerosal from vaccines?

    Was that a real increase or did that arise from the diagnostic definition changes of 1993-94, increased awareness etc.?

    ReplyDelete
  2. I think I already answered your question a couple times, Harold.

    Evidently, I think any noticeable increase that occurred before and after removal of thimerosal, in passive databases, is probably the result of increased awareness for the most part.

    I have not seen any evidence that removal of thimerosal altered administrative trends at all.

    ReplyDelete
  3. I've found that the two "camps" among autism is really not "Pro-Neurodiversity" and "Anti-Neurodiversity" but "Pro-Vaccine" and "Anti-Vaccine."

    I do wish "Neurodiversity" would spend more time on critical issues that relate to autistic people and the diversity they supposedly advocate for.

    I get tired of the same articles and studies being written about over and over again and the same arguments being made by the same people. I've been in the online autism world for over a year now and it is still the same.

    You left me this response on LB/RB:

    "Institutionalization is just a terrible idea. It’s interesting to read Kanner’s findings on this. No other researcher has ever looked into the institutionalization of autistics, as far as I’m aware.

    Read more: http://leftbrainrightbrain.co.uk/?p=3642#ixzz0XsnpsVn1"

    Can you direct me to Kanner's findings? As a person with autism who was institutionalized I'd like to read them.

    ReplyDelete
  4. @Stephanie: Yes, some of us are interested in certain topics. I don't see anything wrong with this. I'm certainly not going to start to write about other topics because I'm told to.

    There are plenty other blogs about autism, e.g. in the Hub, that rarely engage in debates about causation and so forth. I'm sure you can find one that is more in line with your preferences.

    I've written about Kanner's findings on institutionalization here. I hope you find it informative.

    ReplyDelete
  5. off topic but

    Zack's site's down.

    dunno what's going on there

    anyone got any ideas?

    ReplyDelete
  6. hmm. Sad, once again the desire for fame surpassed the honesty of science. What can you do?

    Sometimes we just have to learn to use the PR to get at the actual science article as a matter of habit, and some times we have to make a choice.

    We could take a red pill and learn about how the Huffington post is providing truly dangerous misinformation about the flu or we coudl take a blue pill and learn how media outlets incessantly fail to provide useful information about various foods

    ReplyDelete
  7. I'm on here as anonymous but it's Theo. I am scared for Zack. His last post talked about attempting suicide again and this time now in his house so people couldn't try to stop him.

    In desperation I contacted Suzanne Higgs on her blog to let her know what was going on and see if something terrible can be prevented and I have heard nothing back from her.

    Does anyone else know anything?!

    ReplyDelete
  8. I meant not in his house, sorry!

    ReplyDelete
  9. Hey Joseph.

    You haven't updated in a while however I do want to watch your blog, I've seen your posts on LB/BR and I hope to contribute more to your blog.

    Bard.

    ReplyDelete
  10. @NightStorm: Thanks. Sometimes I update fairly often. Sometimes I'm just spending my spare time in other activities.

    I encourage readers to subscribe to the RSS feed.

    ReplyDelete
  11. Hi!

    Engrossing blog. Would love your thoughts on my findings and thoughts in my blog Bob Versus... Autism:

    http://bobversus.com/archives/501

    Have a Kim (Peek, get it?)

    ReplyDelete
  12. Hi, enjoy your blog, thanks.

    I recently got dragged to a therapy "seminar" for an autism treatment provider. PACE Place. One thing mentioned was Polyvagal Theory, by Stephen Porges.

    Any one heard about these? I'd appreciate it.

    ReplyDelete