Saturday, April 17, 2010

MJ Reproduces A Result of Mine


The anti-vax movement peaked in 2002, maybe in 2003. There are several different lines of evidence that point in this direction. I recently presented just two of them: Google News Archive articles matching "autism vaccines" relative to "autism" articles, and VAERS report submissions. Additionally, Sullivan over at LB/RB has put forth a graph of cases before the US vaccine court.

It's not surprising that such easy-to-confirm observations would hit a nerve with some people. Commenter MJ took issue with my methodology, at first claiming that as more autism articles are written, the word "vaccines" would tend to become rare in them, and later claiming that Google News Archive does not have the right bias for this type of analysis. None of this made any sense to me, and you can read the exchange in comments.

Then MJ wrote a post in response to my analysis where, evidently, MJ has come up with a reproduction (not repetition) of my prior result. You can see MJ's graph (which I copied with "fair use" in mind) on the right.

MJ takes comfort in the fact that newspaper articles matching the word "neurodiversity" are quite uncommon, relatively speaking. But see – and I doubt MJ doesn't realize this – that's a comparison of apples to oranges. Articles matching "autism vaccines" are about a public health issue, one that is bound to interest all kinds of readers. They are articles about court cases, studies, etc. Articles on neurodiversity are about an ideology, which reporters might not cover simply because they don't see a payoff in covering it.

If you're going to compare them, it obviously only makes sense to compare trends, not absolute article counts. MJ's scaling obviously doesn't allow us to see a trend, so I've produced the following graph.



It's an entirely different pattern. (Note that since article counts are relatively small, there's bound to be more noise in these series.)

One More Thing

Regarding VAERS, MJ says:
As for VAERS, it wasn't meant to do this sort of tracking nor is it an accurate measure of all children who had a reaction to a vaccine - especially for controversial relationships like autism.

I didn't look at a measure of "reactions" – and it's not clear if reports with "autism" as a symptom are even valid. I looked at submission counts. These are clearly a valid proxy of new parents recruited into either an anti-vax mindset or vaccine litigation.

9 comments:

  1. I think it's been well-established that the Jabberwock (MJ) is a "blinker-thinker". He only knows what he believes, rather than vice-versa.

    "Don't believe everything you think!"

    ReplyDelete
  2. Dear Joseph,

    Thanks for this posts and the previous ones. I don't naturally think in quantitative ways, so I so appreciate those, like you, who lead me to see things I would otherwise not.

    Being more of a qualitative thinker, I wonder if the designation "anti-vax" is quite appropriate. This is my reasoning: prior to 2002, there was an movement that proclaimed that vaccinations were (a)ineffective in reducing common infection diseases of childhood (CIDC) (b)vaccinations against CIDC posed more health risks than the actual CIDC.

    What changed after 2000 was, rather than vaccines being generally "bad", was that vaccines were somehow implicated in autism causation -- either "autism = mercury poisoning" and/or the messy idea that somehow, the measles-mumps-rubella (MMR) vaccine was somehow causal in autism.

    I don't have the skills to do what you have done, using the terms "autism mmr" or "autism vaccine injury" but I do wonder what the outcome would be.

    ReplyDelete
  3. Thanks, Liz. Articles that match autism "vaccine injury" generally follow a similar pattern, with a peak in 2003, and a boost in 2008.

    Matches for autism mmr and autism thimerosal have even more obvious peaks: MMR in 2002, and thimerosal apparently in 2004.

    Matches for "andrew wakefield" peak in 2004, with a boost in 2010 (for obvious reasons.)

    Matches for handley autism peak in 2005. You might recall that's when JB Handley went on TV, all sure of himself. He proclaimed autism is exactly the same as mercury poisoning, and that autistic children would get cured with chelation in 2 years time.

    ReplyDelete
  4. Something else that's interesting: If you look at matches for "vaccine injury" (in quotes) between 1980 and 2010, it appears that a prior peak in 1990 was pretty close to the more recent 2002 peak.

    The 1990 articles are mostly about the VICP.

    ReplyDelete
  5. I think you are missing the point of what I am saying, so let me make this simple.

    Your chart shows nothing meaningful. You are charting the percentage of articles per year that have the word "vaccines" and comparing it to the total number of articles about autism. Yet, it would be expected that as autism became more mainstream there would be more and more topics discussed, such as treatment opinions, life stories, genetics, etc. Therefore I would expect the percentage of articles dealing with one subject to drop.

    Or even simpler terms, as a topic matures and broadens, I would expect one part to account for less of the whole, and the topic of autism has certainly broadened from 1997. And this is what your chart shows - that the number of articles concerning autism and vaccines is dropping as percent of the whole.

    If you look at the counts, there were 1840 articles about autism in 1997 and 25,550 articles in 2009. If all of the additional articles were about vaccines, the world would be very bored by now. Interest in autism is expanding beyond the vaccine story, and that is a good thing for all involved.

    Google news archive is not representative and many news articles never make it into the index. Furthermore, sources have been added and removed over the years which could significantly change your basic population of sources. Your analysis can only be as valid as your data source, and in this case, the data source isn't good.

    Also, I never said it was "biased", you did.

    But, if you want to see if a specific topic is becoming more or less common, you look at the trend for that topic, not the trend of a percentage that relates to another rapidly changing item. If you look at the relative year over year change (or some other way of looking for trends) in the number of stories about autism and vaccines, you will see that the topic grew 10 of the past 13 years and is on track to grow again this year.

    My scaling on my charts was not bogus, it is consistent for both items. The reason you can't see the ND trend is because it isn't there - the numbers are basically noise. I also provides a chart will all of the absolute numbers as well. If you want to talk about scaling, your chart above with 1 per 1,000 and 1 per 100,000 is a classic example of abuse of scale.

    (BTW, you are free to include any chart or image I produce as long as you reference the original source).

    But I think you missed the overall point. Your analysis is meaningless and doesn't show anything. I wrote what I did as a tongue in cheek response to your post, knowing full well that the results are bogus (which I even said), to point out the absurdity of what you were saying. I guess you missed that part.

    ReplyDelete
  6. @Clay - which is it, do I have ilk and look at the world through sludge colored glassed or am I a blinker-thinker? Or perhaps it was what someone else suggested and I have reading comprehension problems. I can't seem to keep the insults straight.

    I think what is most telling in your response is that you don't refute what I say or talk about the ideas but rather go straight for the personal insults.

    ReplyDelete
  7. You are charting the percentage of articles per year that have the word "vaccines" and comparing it to the total number of articles about autism. Yet, it would be expected that as autism became more mainstream there would be more and more topics discussed, such as treatment opinions, life stories, genetics, etc. Therefore I would expect the percentage of articles dealing with one subject to drop.

    Again, MJ, that makes no sense. You already looked at "neurodiversity" in this regard, and I suppose you might have looked at other trends, so you must have figured out your hypothesized effect does not exist.

    In any case, it's a simple matter to control for the total number of articles instead. This would measure the public's exposure to anti-vaccination from the MSM, as opposed to what I measured (the media's interest in the topic.) You can simply count the number of articles that match, say, "a". You will find a roughly similar pattern. As a matter of fact, you'll find a greater decline of anti-vax coverage in recent years with this method.

    You can search "vaccine injury" (in quotes) instead, and control it for the total number of articles. Again, you find the same pattern.

    However you slice it, you find the same result.

    ReplyDelete
  8. Joseph,

    Here is your reply -

    http://autismjabberwocky.blogspot.com/2010/04/analyzing-medias-interest-aka-josephs.html

    I started typing it as a comment and it got way to long, so I made a post out of it, complete with pretty pictures.

    Enjoy.

    ReplyDelete
  9. @MJ: First of all, do you realize that the word "and" is meaningless to Google? Google doesn't parse boolean expressions. It's sufficient to search for "autism vaccines" and that will only match articles with both words. If you include the word "and" in there it will simply narrow it down for articles that also contain the word "and". The reason you're using that search phrase is because you found it gives results that you see as slightly better.

    That's not important either way. I thought I'd just point it out.

    I general, any criticism of the form "the data just can't be any good" is vague and lazy, in my view. It's all too easy to dismiss any finding that way. Plus it's a simple matter to check Google Scholar to determine if and how Google News Archive is used in research.

    But let's see what else you had to say.

    The thrust of your argument is that the number of articles matching "autism and vaccines" has grown in terms of absolute counts. This argument is invalid, which is easy to show, as follows.

    The number of articles matching "autism vaccines" in 2002 is 1450. The number of articles matching the same in 2008 was 2130. Fair enough, but let's compare matches for the word "the": That would be 615,000 articles in 2002 vs. 1,290,000 articles in 2008.

    In other words, articles matching "autism vaccines" have dropped from 23.6 in 10,000 total articles in 2002 to 16.5 in 10,000 articles in 2008.

    Clearly, the number of articles that Google indexes varies from year to year, and it grows in general. You're taking advantage of this fact by failing to control for it.

    So your statement

    If we were just measuring interest by number of articles, there is clearly an increase in interest over time

    is just wrong. It assumes that Google Archive's coverage of the media is equivalent from year to year. It obviously isn't.

    Moving on, you show a graph of increase in interest from year to year. In this graph we see that autism interest growth clearly outpaces "autism vaccines" interest growth almost all years after 2002. 2008 is an outlier, because of two events that occurred that year, as I've noted. You give the graph an odd interpretation, though.

    (That's not really a good measure of "increase in interest" either way, because again, it fails to control for growth in the total number of articles, but that's fine in this case.)

    ReplyDelete