Today's post is long. It's long because it involves the winding path that science can take from ignition to exploding into the public view... and how the twists and turns in that path can result in a skewed representation and understanding of the science. Read the whole thing. It focuses on an example that involves autism--which seems to pop up in skewed representations every day--but certainly this path from science to you, the consumer, happens with scientific information in general. The author is Jess, who blogged this originally at "Don't Mind the Mess" and graciously gave us permission to reproduce it here. Jess, an attorney with a B.S. in biochemistry, parent of an autistic child and brand new baby, and self-described "Twitter fiend," tweets as @JessicaEsquire.
As the parent of an autistic child I hear a lot about vaccines and about half a million other things that people think cause autism.
I’m hyperaware of the attention autism gets in the media. So I know about the CDC’s new stats on autism rates. I know about the debate on whether the increase in autism is due to more awareness and diagnosis or more actual occurrences. (Personally, I find the former to be a serious factor, though who’s to say how much.) And I see all the articles that come out week after week about the millions of things that are linked to autism.
There’s a recurring problem here. Valuable research is done. Research is disseminated. Information is reported. Articles are read. Findings are spread. What starts in a lab ends up in a Facebook status. What starts as truth ends up as mistruth in something like a child’s game of telephone. Along the way, piece by piece, truth fades away in favor of headlines and pageviews and gossip.
It’s getting just plain stupid. I’m starting to suspect these articles have nothing to do with serious research but with a search for traffic and hype, an attempt to ride the wave of a trendy topic as concerned parents read every horror story they can find.
A particularly egregious one came up recently. This one doesn’t just cite some random correlation. This one is just plain making things up. The problems here just pile one on top of the other. So let’s consider it piece by piece, a case study in how real research becomes misinformation.
Part One: Research
It starts with scientists. It starts with research. They write up their findings and publish them in a peer-reviewed scientific journal. In this case there are several papers published over a few years about chemicals and their link to brain development. They cover a wide variety of issues and present a wide variety of conclusions. All of them suggest further study.
Maybe they have bad methodology or use statistics incorrectly. Only a few people would ever know the difference. That’s not my concern today. Bad science is one thing, but bad information on good science is another. So let’s assume we have good, solid science in this research.
Part Two: The Conference
Scientists and researchers with similar interests get together and discuss their findings. It’s not that difference from any other conference. There are panels and presentations.
Part Three: The Op-Ed
Next, a group that works on environmental hazards for children publishes a paper. Not a research study but an op-ed in a peer-reviewed journal. In this op-ed they review the conference from Part Two and encourage the study of environmental factors and their relationship to neurodevelopment disorders. Autism is one of many neuro-ish disorders and is mentioned by name in the piece and its title. It’s unclear to me why they zero in on autism. They have a couple vague pieces of evidence that are autism-specific, but the vast majority of what they’re looking at has never been demonstrated to have any kind of relationship to autism, not even a correlation.
Problem #1 is the unnecessary autism name-checking. Problem #2 is much worse, it’s the list of 10 chemicals they suggest for future study. The list itself isn’t a bad idea, I guess. They’re suggesting places for potential research, which certainly needs to be done. But it does reek a little bit of the kind of thing magazines do, you know what I mean, 10 Ways To Get Your Guy All Fired Up! and such. Still, it’s their prerogative.
So let’s examine their evidence for these suggestions. They cite at least one paper for each of these chemicals. I checked them all. The vast majority of them have never shown any connection to Autism (or even ADHD, another diagnosis they name-check). In fact, many of them show that with exposure to these chemicals, the outcome differentials between exposure and non-exposure is 5 IQ points.
FIVE IQ POINTS. Statistically significant? Perhaps. Practically important for a parent? No.
IQ itself is a strange and vague thing. And 5 points isn’t going to move your super-genius down to the level of an average person They’d still be a super-genius. And adding 5 points to someone with severe deficits isn’t going to make them average, either. It’s hard to imagine what difference you’d see between two people whose IQ’s are 5 points apart.
Such statistical differences may well be a sign to warrant further study. And they may be a sign that these chemicals affect neurological development. But it’s getting a bit ahead of ourselves to say they are suspected of being tied to autism. Many of these papers are in areas of research that are just beginning. Many of them involve homogeneous groups (for example, all the participants are Mexican-American migrant workers) which makes issues of genetics and heredity very difficult to account for. Many involve parents self-reporting by filling out surveys rather than having the children examined by professionals.
Let’s be fair. These are the very beginnings of research. You’ll need to do all sorts of rigorous testing and consideration to make real connections. Of course more research is needed. And it’s important that we keep that in mind as we move forward.
(Though, of course, no one else will.)
Part Four: The Press Release
The op-ed is about publicity so it’s the beginning of the problem. But it gets worse.A press release comes out with the list of ten chemicals and already the twisting starts. These are chemicals suggested for further research, but suddenly they’re a ”List of the Top Ten Toxic Chemicals Suspected to Cause Autism and Learning Disabilities.” This, unsurprisingly, is the headline you’ll see all over the internet when news organizations report on the press release. Already it’s turned from suggestions for research into a watchlist.
It gets worse. The press release has this second headline:
The editorial was published alongside four other papers — each suggesting a link between toxic chemicals and autism.
No, actually that’s not at all accurate.
Let’s start with the first paper, which examines the possibility of a connection between maternal smoking and autism. What’s their conclusion?
The primary analyses indicated a slightly inverse association with all ASDs[.]
What does that mean? Among the autistic kids vs. regular kids, there was actually LESS maternal smoking in the autism group. The paper does point out that when it comes to “subgroups,” for instance high-functioning ASD or Asperger’s, there may be a possibly positive relationship. But there are so many caveats I can’t even get to them all. Let’s just take this one:
The ASD subgroup variables were imperfect, relying on the child’s access to evaluation services and the documentation by a myriad of community providers, rather than direct clinical observation.
This means that when they’re saying some groups of ASD kids may have this relationship, they didn’t actually classify these kids. They never saw these kids. They’re relying on data collected by other people. Not even by a consistent set of people. It comes from 11 different states and who knows how many providers. Who’s to say how accurate any of it is. And who’s to say whether these kids are correctly classified at their particular place on the spectrum.
So take all that with a whole jar full of salt and you’re still looking at, overall, no connection with smoking. If anything, the data would indicate smoking has LESS autism rather than more.
After this there are 2 papers on the same chemical. One of them does not contain the word “autism” anywhere. (One of its references has it, but nowhere does it appear in the text of their paper.) The second paper is better. It focuses on the chemical’s effects in particular processes which have been linked to autism. This is very micro-scale science, there are no people involved, just cells and chemicals. It’s important research, but there’s a long stretch between cellular interactions and a person’s diagnosis. It didn’t involve any analysis with autistic individuals. This is certainly the most useful paper of the bunch by a long shot, but it still just sets the stage for further research.
The fourth paper is a review. That means it asserts no new information but summarizes the research on a particular issue, specifically pesticides and autism. Technically I suppose it does assert a link, but none of this is new information.
So I think we’ve pretty much destroyed the headline in that press release. There were not 4 articles suggesting a connection between chemicals and autism.
Is it likely that the writers who take this press release and write articles on it are going to read the papers it cites? Are they going to realize that what they’re saying isn’t actually true? They should. Of course they should. But they don’t.
This list has chemicals suspected of being tied to neurological development. And we should just leave it at that. It’s not that they shouldn’t be studied. They should. But we shouldn’t be throwing out buzzwords like ADHD and Autism when the research doesn’t show any firm data.
Part Five: News Articles
This is a process, though. First research, then op-ed, then press release and finally news articles. So what’s the headline of our news article? “Top 10 Chemicals Most Likely to Cause Autism and Learning Disabilities.” Guilty of serious fearmongering, no? A more accurate title may be: Researchers propose list of chemicals potentially tied to neurological development for further study. But I doubt anyone’s going to write that.
The article itself, to be fair, is full of caveats. The reasons for the increase in autism are “controversial.” There is a “gap in the science.” But then you get a sentence like this:
But clearly, there is more to the story than simply genetics, as the increases are far too rapid to be of purely genetic origin.
Clearly? Clearly says who? What source says it’s too rapid? The author certainly isn’t a reliable source. She is Robyn O’Brien, a writer for Prevention who posted this article. Her scientific credentials are nonexistent. She is a former financial analyst who now writes about the food industry. She has an MBA, and her undergraduate was in French and Spanish.
Full disclosure: I have a B.S. in Biochemistry, but I feel I’m unqualified to write this article. I’d much rather it be written by someone with a PhD. I’m married to a PhD, which has given me a lot more exposure to science since leaving school, but I fully acknowledge that I shouldn’t be the one doing this. I know how to read a scientific article and examine its conclusions, but I certainly am not someone who can tell you if their methods and analysis are correct.
But I’m talking because there aren’t enough people talking about it. Because the PhD’s aren’t generally science writers. They are scientists. They write about their research in journals, not in the newspaper. And certainly not on a blog for a healthy living magazine.
The author goes on to restate the inaccurate subheadline of the press release verbatim.
In the end she suggests things like buying organic produce, opening your windows and buying BPA-free products.
This is part 5 of our process, but it’s where many of us start. Many of us will only read this article and not the press release or the op-ed or the research papers. Most of us aren’t qualified to do so, all we have is this article. Well, we have that and what other people tell us. Which leads us to our next step.
Part Six: Readers
The article is frustrating, but I can only get so mad. She is saying what the scientists told her to say. She has even included some cautionary language. The problem is that when writing for laymen, you have to be careful.
And with AUTISM? You have to be really careful. Just for you I’m going to venture into the comments to this article to show you how people have responded.
–How about we quit injecting our kids with aluminum, formaldehyde and the rest of the toxic stew that they call vaccines — we bypass every natural defense our bodies have (skin, saliva, stomach acid) to put these things directly in the blood stream.
–Thank you Robyn for always providing sound information to continue guiding our decisions.
–What about heavy metals like Arsenic that are trapped in soils that our “organic” brown rice is growing in to be made into brown rice syrup to sweeten organic foods and baby formula? Not to mention the reports coming in regarding the radiation and contamination from Fukushimi that has reached the west coast an is spreading across this country in the produce and even the pollen…
–Unvaccinated children are some of the healthiest little people on the planet. As far as the Autism link, who really knows but why risk it.
–Thank you for this information. It confirms to me that we should keep doing what we are doing. It also helps me to enforce our no shoes policy in our home. Some people are so disrespectful and just don’t take them off and I hate to sound like a nag and ask even though they already know its what we prefer.
Thankfully there are some people in there who take the writer to task, but how is a reader to trust any one commenter over another? You have no way of knowing from a comment what someone’s experiences or qualifications are.
There’s a reason we need responsible scientific reporting. I’m all for the open dissemination of information, but I’m also aware of what happens when people read something they don’t understand.
I encountered this FB conversation the other day. Usually I overlook such things but I could not help myself. I jumped in. I tried hard to be polite and present facts. When all that was over, no one was convinced. The response?
Enough articles on vaccines and people are scared even without evidence. Enough headlines and people don’t bother reading articles. It doesn’t matter how much is retracted or debunked, the damage is done.
We need responsible science reporting. We need responsible reporting, period. I’ve seen plenty of lazy articles on Supreme Court opinions that lead me to read the opinion myself only to realize that they’ve stated the conclusions all wrong.
I don’t want to go on all day, but I do feel like it’s important for us to put our foot down and demand better.
We aren’t all scientists. But we can ask for science writers with the appropriate qualifications. We can ask for links and citations in their articles. (I spent quite some time tracking everything down for this post, and luckily I’m relatively familiar with looking up scientific articles online.) We can ask for articles that show failed connections. It doesn’t all have to be “Autism linked to X” there’s plenty of “Autism not linked to Y” that happens in these studies but you never see that, do you?
As for us laymen, we have to find our own trusted experts. Ask your pediatrician. And if your pediatrician’s not qualified (most of them are MD’s but not PhD’s) ask them if they have a trusted source. Track down specialists in Autism with PhD’s and ask them what they think of the research. Find reliable books and articles and spread them to your friends. We can’t necessarily do a lot, but we can do our part to stop the spread of misinformation and demand better.
These views are the opinion of the author and do not necessarily either reflect or disagree with those of the DXS editorial team.