Friday, May 4, 2012

The path from science to alarmism: How science gets twisted before it gets to you



Source.


Today's post is long. It's long because it involves the winding path that science can take from ignition to exploding into the public view... and how the twists and turns in that path can result in a skewed representation and understanding of the science. Read the whole thing. It focuses on an example that involves autism--which seems to pop up in skewed representations every day--but certainly this path from science to you, the consumer, happens with scientific information in general. The author is Jess, who blogged this originally at "Don't Mind the Mess" and graciously gave us permission to reproduce it here. Jess, an attorney with a B.S. in biochemistry, parent of an autistic child and brand new baby, and self-described "Twitter fiend," tweets as @JessicaEsquire
---------------------------------------------------------------
I am putting my foot down.
As the parent of an autistic child I hear a lot about vaccines and about half a million other things that people think cause autism.
I’m hyperaware of the attention autism gets in the media. So I know about the CDC’s new stats on autism rates. I know about the debate on whether the increase in autism is due to more awareness and diagnosis or more actual occurrences. (Personally, I find the former to be a serious factor, though who’s to say how much.) And I see all the articles that come out week after week about the millions of things that are linked to autism.
There’s a recurring problem here. Valuable research is done. Research is disseminated. Information is reported. Articles are read. Findings are spread. What starts in a lab ends up in a Facebook status. What starts as truth ends up as mistruth in something like a child’s game of telephone. Along the way, piece by piece, truth fades away in favor of headlines and pageviews and gossip.
It’s getting just plain stupid. I’m starting to suspect these articles have nothing to do with serious research but with a search for traffic and hype, an attempt to ride the wave of a trendy topic as concerned parents read every horror story they can find.
A particularly egregious one came up recently. This one doesn’t just cite some random correlation. This one is just plain making things up. The problems here just pile one on top of the other. So let’s consider it piece by piece, a case study in how real research becomes misinformation.

Part One: Research

It starts with scientists. It starts with research. They write up their findings and publish them in a peer-reviewed scientific journal. In this case there are several papers published over a few years about chemicals and their link to brain development. They cover a wide variety of issues and present a wide variety of conclusions. All of them suggest further study.
Maybe they have bad methodology or use statistics incorrectly. Only a few people would ever know the difference. That’s not my concern today. Bad science is one thing, but bad information on good science is another. So let’s assume we have good, solid science in this research.

Part Two: The Conference

Scientists and researchers with similar interests get together and discuss their findings. It’s not that difference from any other conference. There are panels and presentations.

Part Three: The Op-Ed

Next, a group that works on environmental hazards for children publishes a paper. Not a research study but an op-ed in a peer-reviewed journal. In this op-ed they review the conference from Part Two and encourage the study of environmental factors and their relationship to neurodevelopment disorders. Autism is one of many neuro-ish disorders and is mentioned by name in the piece and its title. It’s unclear to me why they zero in on autism. They have a couple vague pieces of evidence that are autism-specific, but the vast majority of what they’re looking at has never been demonstrated to have any kind of relationship to autism, not even a correlation.
Problem #1 is the unnecessary autism name-checking. Problem #2 is much worse, it’s the list of 10 chemicals they suggest for future study. The list itself isn’t a bad idea, I guess. They’re suggesting places for potential research, which certainly needs to be done. But it does reek a little bit of the kind of thing magazines do, you know what I mean, 10 Ways To Get Your Guy All Fired Up! and such. Still, it’s their prerogative.
So let’s examine their evidence for these suggestions. They cite at least one paper for each of these chemicals. I checked them all. The vast majority of them have never shown any connection to Autism (or even ADHD, another diagnosis they name-check). In fact, many of them show that with exposure to these chemicals, the outcome differentials between exposure and non-exposure is 5 IQ points.
FIVE IQ POINTS. Statistically significant? Perhaps. Practically important for a parent? No.
IQ itself is a strange and vague thing. And 5 points isn’t going to move your super-genius down to the level of an average person They’d still be a super-genius. And adding 5 points to someone with severe deficits isn’t going to make them average, either. It’s hard to imagine what difference you’d see between two people whose IQ’s are 5 points apart.
Such statistical differences may well be a sign to warrant further study. And they may be a sign that these chemicals affect neurological development. But it’s getting a bit ahead of ourselves to say they are suspected of being tied to autism. Many of these papers are in areas of research that are just beginning. Many of them involve homogeneous groups (for example, all the participants are Mexican-American migrant workers) which makes issues of genetics and heredity very difficult to account for. Many involve parents self-reporting by filling out surveys rather than having the children examined by professionals.
Let’s be fair. These are the very beginnings of research. You’ll need to do all sorts of rigorous testing and consideration to make real connections. Of course more research is needed. And it’s important that we keep that in mind as we move forward.
(Though, of course, no one else will.)

Part Four: The Press Release

The op-ed is about publicity so it’s the beginning of the problem. But it gets worse. A press release comes out with the list of ten chemicals and already the twisting starts. These are chemicals suggested for further research, but suddenly they’re a ”List of the Top Ten Toxic Chemicals Suspected to Cause Autism and Learning Disabilities.” This, unsurprisingly, is the headline you’ll see all over the internet when news organizations report on the press release. Already it’s turned from suggestions for research into a watchlist.
It gets worse. The press release has this second headline:
The editorial was published alongside four other papers — each suggesting a link between toxic chemicals and autism.
No, actually that’s not at all accurate.
Let’s start with the first paper, which examines the possibility of a connection between maternal smoking and autism. What’s their conclusion?
The primary analyses indicated a slightly inverse association with all ASDs[.]
What does that mean? Among the autistic kids vs. regular kids, there was actually LESS maternal smoking in the autism group. The paper does point out that when it comes to “subgroups,” for instance high-functioning ASD or Asperger’s, there may be a possibly positive relationship. But there are so many caveats I can’t even get to them all. Let’s just take this one:
The ASD subgroup variables were imperfect, relying on the child’s access to evaluation services and the documentation by a myriad of community providers, rather than direct clinical observation.
This means that when they’re saying some groups of ASD kids may have this relationship, they didn’t actually classify these kids. They never saw these kids. They’re relying on data collected by other people. Not even by a consistent set of people. It comes from 11 different states and who knows how many providers. Who’s to say how accurate any of it is. And who’s to say whether these kids are correctly classified at their particular place on the spectrum.
So take all that with a whole jar full of salt and you’re still looking at, overall, no connection with smoking. If anything, the data would indicate smoking has LESS autism rather than more.
After this there are 2 papers on the same chemical. One of them does not contain the word “autism” anywhere. (One of its references has it, but nowhere does it appear in the text of their paper.) The second paper is better. It focuses on the chemical’s effects in particular processes which have been linked to autism. This is very micro-scale science, there are no people involved, just cells and chemicals. It’s important research, but there’s a long stretch between cellular interactions and a person’s diagnosis. It didn’t involve any analysis with autistic individuals. This is certainly the most useful paper of the bunch by a long shot, but it still just sets the stage for further research.
The fourth paper is a review. That means it asserts no new information but summarizes the research on a particular issue, specifically pesticides and autism. Technically I suppose it does assert a link, but none of this is new information.
So I think we’ve pretty much destroyed the headline in that press release. There were not 4 articles suggesting a connection between chemicals and autism.
Is it likely that the writers who take this press release and write articles on it are going to read the papers it cites? Are they going to realize that what they’re saying isn’t actually true? They should. Of course they should. But they don’t.
This list has chemicals suspected of being tied to neurological development. And we should just leave it at that. It’s not that they shouldn’t be studied. They should. But we shouldn’t be throwing out buzzwords like ADHD and Autism when the research doesn’t show any firm data.

Part Five: News Articles

This is a process, though. First research, then op-ed, then press release and finally news articles. So what’s the headline of our news article? “Top 10 Chemicals Most Likely to Cause Autism and Learning Disabilities.” Guilty of serious fearmongering, no? A more accurate title may be: Researchers propose list of chemicals potentially tied to neurological development for further study. But I doubt anyone’s going to write that.
The article itself, to be fair, is full of caveats. The reasons for the increase in autism are “controversial.” There is a “gap in the science.”  But then you get a sentence like this:
But clearly, there is more to the story than simply genetics, as the increases are far too rapid to be of purely genetic origin.
Clearly? Clearly says who? What source says it’s too rapid? The author certainly isn’t a reliable source. She is Robyn O’Brien, a writer for Prevention who posted this article. Her scientific credentials are nonexistent. She is a former financial analyst who now writes about the food industry. She has an MBA, and her undergraduate was in French and Spanish.
Full disclosure: I have a B.S. in Biochemistry, but I feel I’m unqualified to write this article. I’d much rather it be written by someone with a PhD. I’m married to a PhD, which has given me a lot more exposure to science since leaving school, but I fully acknowledge that I shouldn’t be the one doing this. I know how to read a scientific article and examine its conclusions, but I certainly am not someone who can tell you if their methods and analysis are correct.
But I’m talking because there aren’t enough people talking about it. Because the PhD’s aren’t generally science writers. They are scientists. They write about their research in journals, not in the newspaper. And certainly not on a blog for a healthy living magazine.
The author goes on to restate the inaccurate subheadline of the press release verbatim.
In the end she suggests things like buying organic produce, opening your windows and buying BPA-free products.
This is part 5 of our process, but it’s where many of us start. Many of us will only read this article and not the press release or the op-ed or the research papers. Most of us aren’t qualified to do so, all we have is this article. Well, we have that and what other people tell us. Which leads us to our next step.

Part Six: Readers

The article is frustrating, but I can only get so mad. She is saying what the scientists told her to say. She has even included some cautionary language. The problem is that when writing for laymen, you have to be careful.
And with AUTISM? You have to be really careful. Just for you I’m going to venture into the comments to this article to show you how people have responded.
–How about we quit injecting our kids with aluminum, formaldehyde and the rest of the toxic stew that they call vaccines — we bypass every natural defense our bodies have (skin, saliva, stomach acid) to put these things directly in the blood stream.
–Thank you Robyn for always providing sound information to continue guiding our decisions.
–What about heavy metals like Arsenic that are trapped in soils that our “organic” brown rice is growing in to be made into brown rice syrup to sweeten organic foods and baby formula? Not to mention the reports coming in regarding the radiation and contamination from Fukushimi that has reached the west coast an is spreading across this country in the produce and even the pollen…
–Unvaccinated children are some of the healthiest little people on the planet. As far as the Autism link, who really knows but why risk it.
–Thank you for this information. It confirms to me that we should keep doing what we are doing. It also helps me to enforce our no shoes policy in our home. Some people are so disrespectful and just don’t take them off and I hate to sound like a nag and ask even though they already know its what we prefer.
Thankfully there are some people in there who take the writer to task, but how is a reader to trust any one commenter over another? You have no way of knowing from a comment what someone’s experiences or qualifications are.
There’s a reason we need responsible scientific reporting. I’m all for the open dissemination of information, but I’m also aware of what happens when people read something they don’t understand.
autism FB The Whole Truth About Autism
I encountered this FB conversation the other day. Usually I overlook such things but I could not help myself. I jumped in. I tried hard to be polite and present facts. When all that was over, no one was convinced. The response?
autism FB 2 The Whole Truth About Autism
Enough articles on vaccines and people are scared even without evidence. Enough headlines and people don’t bother reading articles. It doesn’t matter how much is retracted or debunked, the damage is done.
We need responsible science reporting. We need responsible reporting, period. I’ve seen plenty of lazy articles on Supreme Court opinions that lead me to read the opinion myself only to realize that they’ve stated the conclusions all wrong.
I don’t want to go on all day, but I do feel like it’s important for us to put our foot down and demand better.
We aren’t all scientists. But we can ask for science writers with the appropriate qualifications. We can ask for links and citations in their articles. (I spent quite some time tracking everything down for this post, and luckily I’m relatively familiar with looking up scientific articles online.) We can ask for articles that show failed connections. It doesn’t all have to be “Autism linked to X” there’s plenty of “Autism not linked to Y” that happens in these studies but you never see that, do you?
As for us laymen, we have to find our own trusted experts. Ask your pediatrician. And if your pediatrician’s not qualified (most of them are MD’s but not PhD’s) ask them if they have a trusted source. Track down specialists in Autism with PhD’s and ask them what they think of the research. Find reliable books and articles and spread them to your friends. We can’t necessarily do a lot, but we can do our part to stop the spread of misinformation and demand better.


These views are the opinion of the author and do not necessarily either reflect or disagree with those of the DXS editorial team.
---------------------------------------------------------------
We also suggest

9 comments:

  1. Here here. That's pretty much why I get my news at http://www.theonion.com/

    ReplyDelete
  2. Thank you SO much for this article! Really.

    ReplyDelete
  3. You putting your foot down? Impressive. Are you huffing and puffing too? Snorting in righteous indigination?

    The causes of autism disorders are unknown. Genetic research despite the receiving the overwhelming lion's share of autism research dollars has provided no answers. Environmental autism research is important. Your temper tantrum is irrelevant. I do not share the views of Bob, Stephen and Emma posted above. Your comment is nonsense.

    ReplyDelete
    Replies
    1. This post from our contributor, Jess, an attorney like you, is actually an excellent parsing of the path that science can take from being a discussion about possibilities or ideas to becoming entrenched as a top-10 list in the public mind. Your mockery does little to encourage people to read what you say, but I did. Your last paragraph doesn't say anything that is on point with what the post says. For the purposes of assisting in staying on point, her core argument is, "Bad science is one thing, but bad information on good science is another."

      To say "I am putting my foot down" is not an idiom that reflects a toddler's stamping her foot. Rather--and any dictionary explaining idioms can tell you this--it is an assertion of firmness and taking a stand or bracing to stop something. You seem to have conflated the idioms of stamping a foot and putting a foot down.

      If you have comment on Jess's description of how a scientific possibility gets distorted from inception to dissemination to the public, that comment is welcome. What she describes here goes far, far beyond the presentation of science-related news about autism, but Jess has also written here an apt presentation of the pitfalls and cynicism involved in using autism and ADHD as a marketing tool. There was very little in that editorial's citations that involved autism at all, and her commentary here is enlightening on that point and in tracing the path of how an editorial reflecting a scientific meeting consensus can become etched firmly into the public mind as incontrovertible evidence linking "Hot Scientific Topic of the Day" to "Hot Scientific Bogeyman of the Day."

      Delete
  4. You're only just figuring this out now?!? I thank you for the article and adding another to the list of so-called scientific papers. What would be of great benefit is to determine EXACTLY what SCIENCE is and what it is not. Then, and only then can we understand what is at work here. Science, in the name of science etc. has been pushed to the edge of becoming its own equivalence to religion. Both may be ways of knowing or understanding but each is quite different. Both as subject to human nature and the subsequent effects of politics, misuse and abuse, drives to gain power and influence thought...to provoke people into or away from action. Science is a tool to be used. So is a gun. Each can have a good side and a bad side. The difference is that in most places guns are regulated and people require training on HOW TO USE them.

    ReplyDelete
    Replies
    1. I don't think that Jess expressed having only "just" figured this out now; however, this particular treatment of an editorial as new and exciting findings and using autism as marketing tool when it was only peripherally relevant were among the problems she highlighted here.

      Delete
  5. It seems the last ten years, autism is a buzzword in research papers that helps funding, to the tune of a billion dollars. It also is a gravy train for geneticists that just ran out.

    The kids? What kids?

    ReplyDelete
  6. Thank you for this article. I came to it by way of 3QD and it was well worth the reading. As a layperson who spends a fair amount of time reading up on science (as close to the source as I can get with an autodidact's education and "some" college), I am often frustrated by television/online news that seemingly tries to make everything into a Twitter-length statement of thrilling-chilling excitement, or else tries so hard to be "fair and balanced" that celebrity woo is given an equal billing with the research facilities whose dedicated, educated, well-trained staffs are simply trying to do the job that has to be done for any genuine results to bear fruit.

    Frankly, the fact that some of the comments here already reflect the knee-jerk pop cynicism and quasi-trolling rampant on the Web simply points to the other side of the coin. That is, even when someone attempts to untangle the spaghetti (and explain how its made and how it gets tangled), some people will only fume about how you've failed to give them what they want, which is a bowl of spaghetti from which they can judge all of Italy.

    ReplyDelete