Christie Aschwanden is the author of GOOD TO GO: What the Athlete in All of Us Can Learn from the Strange Science of Recovery and the lead science writer at FiveThirtyEight. Her new podcast, Emerging Form launches in mid-February. Find her on Twitter @CragCrest.
+ Full Transcript
(Background music plays)
Rosemary Pennington: Scientific information is complicated. Discussions of methodology, statistical analyses and the wider implications of findings can all be difficult for the layperson to navigate, which is where science reporters enter the picture. Those who do their job well can help an audience understand why a study or a set of findings is important, in a way that can make the data accessible, helping audiences better understand the world around them. Science journalism is the focus of this episode of Stats and Stories where we explore the statistics behind the stories and the stories behind the statistics. I'm Rosemary Pennington. Stats and Stories is a production of Miami University's departments of Statistics and media, journalism, and film as well as the American Statistical Association. Joining me in the studio, our regular panelist John Bailer, chair of Miami’s Statistic department and Richard Campbell, of Media Journalism and Film. Our guest today is Five Thirty Eight science reporter Christie Aschwanden. Christie thank you so much for being here!
Christie Aschwanden: Thanks so much for having me!
Rosemary Pennington: Just to get the conversation started, how is it that you became a science reporter?
Christie Aschwanden: Oh yeah well I didn't set out to become one, that's for sure. I started out actually in high school, I wanted to be a doctor and I think that was kind of a failure of imagination actually because I got really interested in science and I took this physiology and anatomy class that was really interesting to me and so anyway then I went and shadowed some doctors and realized that being a doctor meant hanging out with sick people all the time, which wasn't exactly what I wanted to do. It sounds really cruel and I don't mean it like that but I was really sort of interested in a geeky side of science you know rather you know sort of the interesting science regarding medicine rather than the practice of it. Anyway and then I went to college I majored in biology and thought I was going to become a scientist but the problem is that I could not find that one thing to specialize in and you know when you get your PhD, you become the world's expert on one tiny little thing and it turns out that I'm a generalist. I'm interested in a lot of different things and so I kept sort of stalling to apply to graduate school because I wasn't sure what I wanted to study and somewhere along the way, I discovered that there was this thing called science writing that I could do, that I could actually get paid money to look at interesting studies, talk to cool scientists and write about it and so that was sort of what set me on my way.
Richard Campbell: So what are the challenges of being a generalist rather than a specialist? Because we live in a world especially in science, where people are specialists and you've chosen a different path, but that's what journalists do generally they're generalists trying to report on specialists. So what does that take for you?
Christie Aschwanden: Yeah that's right. I think you know, along the way I have developed some specialties. I mean once I've written about something you know a bunch of times, I start to develop you know, some kind of base knowledge about it. But I think I actually serve my readers better when I don't know too much about a subject. You know I’m able to ask the questions that my readers are going to be asking. And in fact, I will often, when I'm doing an interview with an expert or a scientist, I will often sort of hold back and oftentimes I will ask questions that I know the answers to, just because I want to be able to get those quotes and also I want to be sure that the person is talking sort of at the level that I want to be discussing things with my readers.
John Bailer: So what's been the hardest science story that you've covered so far?
Christie Aschwanden: Oh that's a really great question. I don't think anyone's ever asked me that before. You know I think I would say in terms of something that's been an ongoing subject that I've covered for many years, I would say a cancer screening, just because it's something that the evidence really is contradictory to what most people think and so that's been really difficult to sort of get across and you know, oftentimes sort of having an audience that's hostile to the message that I’m sending.
John Bailer: Yeah I'm really struck by the challenge of trying to tell these stories in health research, you know when feelings run so high.
Christie Aschwanden: Right. Yeah people get very emotional and it's really difficult I mean I think the other thing that's very difficult about writing about cancer screening is that you have the story that many people tell themselves, which is that a cancer screening test saved my life, when in fact a lot of these people, and you know, one of the issues is we can't tell on an individual basis which ones, but a lot of these people are actually victims of over diagnosis, so they're being treated for cancers that would have never harmed them and so what's actually harmed, feels to them like this great triumph of medicine.
Rosemary Pennington: You know one of the criticisms that gets leveled at science reporting, and health reporting in particular, is sort of the boomeranging nature of some particular, again, health reporting where one week a study says one thing and one week a study says something completely contradictory. How have you as a science reporter tried to navigate through that space?
Christie Aschwanden: That's a great question. I would say that my approach to this has evolved a lot since I started, you know. When I first started off in this profession, you know, I did a lot of these studies…I'm sorry a lot of these stories, that were based on a single study, you know, Oh! New study coming out in science, this is exciting, let's write about it. But over the years I've come to realize that you know science is a process, it's not, you know, one answer and so I'm pretty hesitant these days to write the single study stories and in fact at Five Thirty Eight, almost as a policy, we don't do stories that are just about a single study. I won't say that we never do, but if we're going to do that story, it will always be putting that study into a larger context and so I think that you know a lot of it is just recognizing that uncertainty is inherent in science and science is sort of…Brian Nosek, who is the co-director of the Center for Open Science told me once this thing that I think is really true and that is that science is sort of a process of uncertainty reduction and I think that's a really good way of thinking about it and so, you know you never completely eliminate that uncertainty but you're just sort of, you know taking little parts of it away and so you need to be careful and I try to be very careful not to present anything as sort of like settled or the final answer.
Richard Campbell: Can you talk specifically about how you do that? Because I think a lot of the general public thinks that science and data are certain.
Rosemary Pennington: Yeah.
Richard Campbell: That when you see number…
Christie Aschwanden: Oh yeah!
Richard Campbell: When you see these numbers, this is the actual truth. So you just talked about how you're careful about that. Can you talk a little bit more about that?
Christie Aschwanden: Yeah. I think it's really important to put things in context. So, any particular study is very good at answering the exact specific question that it's asking. So you know in this group of study subjects, subjected to these very specific conditions, this is what happened. But the thing is, that's never exactly what we're actually hoping to answer, you know. We don't care like, did this drug work for the 100 people in that study? You want to know will it work for me and it's always a little bit of a leap to go from one to the other, you know. We can accumulate evidence, we can be more or less certain about things. So really putting things into context and I think it's really important to also note the caveats and be sure to say, you know, what the uncertainties are.
John Bailer: So I think you hit something and I always find a challenge, when I'm looking at stories, is the idea of how these uncertainties are communicated. When you mentioned the idea of caveats, but I think a lot of us who are looking for the definitive statement about what this means, what this result implies, without the idea that there is uncertainty associated with it. So what suggestions do you have for communicating that uncertainty?
Christie Aschwanden: It’s really hard and I’ll just say that even when do you do it, often times people don't see it or they don't listen or they don't, you know, they sort of glance over. And there's also…I don't have this problem with my editors at Five Thirty Eight, but you know there are a lot of editors that really want things to be serviceable, they want - what's the bottom line - tell people what to do and yeah it's not to say that just because there's uncertainty, doesn't mean that we can't give advice and we can't, you know, at the end of the day, you need to decide whether to take the drug or not, whether to eat the food or not, you know, whether to do the thing you know, to make these decisions. And so we can make those decisions, you know, based on our best knowledge at the time. And I think it just a matter of understanding that, you know, just because the state of the evidence right now tells us that this is the right thing to do, you have to be open to new research and new evidence that may suggest that actually we've got it wrong and we need to do something else. But you know, you take what you know at the time, you put that together. So I don't think that it's a matter of just saying like, we don't know anything and no one knows anything, or you can't make a decision. It's more like you know, here's what we know right now and here's what we can do with that and here are the uncertainties, you know. And there are a lot of things that we're pretty damn certain about. You know, smoking causes cancer, we know that. At the same time, not every single person who smokes will get cancer. You know there's still some uncertainty baked in there. At the same time we can be pretty certain about these things.
John Bailer: You know, your comments make me think a lot about the debates that we hear recently about reproducibility in science, and you know, I thought that was interesting that you talked about kind of almost a policy of not reporting a single study results these days. You know, what's the message that we tend to communicate to the public, the idea that there are challenges in thinking about reproducing scientific results and that this is just part of the uncertainty of process? How do we avoid that kind of definitive statement being made from a single study and put in the context that it may not be reproduced in some future work?
Christie Aschwanden: Yeah I think you know the first study that finds something is interesting, that's always going to be less definitive, you know, than having multiple studies. So in a lot of ways, the study that kind of comes on top of a bunch of other studies is actually more interesting because you're getting closer to certainty, right? Than you are on that first study. So again, I think the big challenge here, and this is something I really still struggle with, to be honest, is helping readers understand that uncertainty is baked into science, that it's actually a feature of it. It's not a shortcoming and one issue that we have and I wrote something about this, is that you know, there are people who seek to undermine scientific findings that are inconvenient to them or somehow against their best interests by, sort of pointing out this uncertainty and saying oh, we are not sure about this thing and therefore the whole enterprise is wrong and I think we need to be careful that you know if you don't understand that uncertainty is a part of the process, and that its inherent, and that you know, every study is going to leave some unanswered questions, then you become sort of open to this kind of manipulation, where people think well, because we can't say with 100 percent certainty, we don't really know anything and it's all bonk and we need more time to learn more things it’s like you know at some point, we know enough to make decisions and make the best decisions that we can right now. But this uncertainty, sort of fanning the flames of uncertainty, can be used to delay action and that's not a good thing.
John Bailer: You made a comment at a conference where I was attending a session, you gave, that statistics gives answers in probabilities and the human brain thinks in stories. And so what advice do you have for turning these probabilities, and also that are occasionally uncertainties, as you've just been describing into stories that will connect to someone thinking about this?
Christie Aschwanden: Yeah I think you know, finding stories, finding ways to turn those statistics into stories is really important. Now, there you have to be really careful that you don't start you know just so stories are these stories that are created to like explain the data you know and so they perfectly explain the data. So it's very easy to fall into this trap of thinking, this is exactly how it is. So you have to be careful, but I think metaphors are really helpful, finding something in everyday life. I mean we deal with uncertainty in everyday life all the time and people are generally fairly comfortable with it and so if you can find an example from everyday life that could be something that can be helpful, you know, that this is akin to you know, making a decision like this that you do in your normal life.
(Background music plays)
Rosemary Pennington: You're listening to Stats and Stories where we discuss the statistics behind the stories and the stories behind the statistics. I'm Rosemary Pennington with Miami University statistics department chair John Bailer and media, journalism, and films, Richard Campbell. Today we're talking science reporting with Five Thirty Eight’s Christie Aschwanden. Christie, so I'm going to ask you a sort of different question and this is partly on my mind because of a Facebook conversation I was having with some other academics where they seemed very suspicious of talking to reporters and I, in my past life, also did science and medical reporting and getting scientists to talk can be really difficult. There seems to be some suspicion on the part of researchers about reporters possibly misconstruing information. I will admit that as an academic now I don't like to give interviews ever, (Collective laughter) which feels bad as a former journalist. So how do you, number one, navigate sort of those interactions with researchers or scientists that may be a little wary, and sort of what advice might you have for journalists who want to you know thoughtfully engage with researchers and scientists about how to sort of create a fruitful environment for that?
Christie Aschwanden: Yeah, that could be hard to do. It's pretty unusual, but once in a while I will come across someone who's really hesitant to talk, lots of times just sending them examples of my other work will help you know say here's another…I'll send them some stories I've written so that they get a sense of the kind of writing that I do and that often sort of bridges the gap where they feel like you know they read some of my previous work they'll feel like they can trust me but I guess I would just say you know I do say this to scientists you know I'm trying to cover the science and explain it in the best possible way and if I really want to get it right and you know I need your help on this. If you choose not to help me here like the chances of me getting it wrong really go up and so if you're concerned that I'm going to get something wrong tell me like explain to me that thing that you're worried I'll get wrong so I can get it right.
Richard Campbell: Following along those same lines, I'm interested in, there was a point where you wanted to be a doctor, you were interested in science, your undergrad degree was in science, but at some point you decided you wanted to write. So what was…that doesn't happen to a lot of people interested in science necessarily, so what…
Christie Aschwanden: Writing is really hard, yeah.
(Voices overlap)
Richard Campbell: Yes it is, science is hard too! What was the trigger for you? What turned you toward being a generalist and a writer?
Christie Aschwanden: Good question. So I was working, before I made the transition to writing, I had been working at this biotech company and I was kind of bored of my job and so sometimes I would escape and we had a library and the company library they had a New Scientist magazine which I had never seen before and I quickly became addicted and these stories were so cool. Anyway, the company hired a guy who was in charge of like sort of public relations, writing press releases and things like that and he started a company newsletter and so just by happenstance, again I think it was just because I was sort of bored at my job, I started writing a column for it and I would sort of poke fun at the C.E.O. and I remember at the time we were trying to get a…lot of us that rode our bikes to work and we're trying to get like a bike shed where we could store our bikes during the day. Anyway. I was writing columns on that, and the guy who ran the newsletter said hey, you know you're a really good writer. Have you ever considered science writing? And I was sort of like, what's that? And then he said well you know I've noticed you're really addicted to New Scientist magazine. You know, you could write for them. I was like woah, wow! That sounds really cool and in fact the very first thing I ever published was a New Scientist, true story. So I ended up, he told me about...he tuned me into the National Association of science writers which I have been a member of, my entire career. Anyway, but I ended up going to this program at U.C. Santa Cruz science communication program that really specializes in turning scientists into journalists and into writers and so spent 9 months there and that was that was sort of it, never turned back!
Richard Campbell: Very good! You know you talk about mistake, a lot of your writing is about mistakes in data, mistakes in statistics. So talk about some mistakes you see and that journalists make when they write about stuff. You've been doing it a long time, you would know, you would recognize the mistakes that journalists make, and fairly common ones I suspect. Can you talk about that?
Christie Aschwanden: Yeah, I think one common one is that people don't read papers, they just read the abstracts of the abstracts. I mean there's actually data on this. The abstracts can be pretty misleading and they're often times conclusions that are sort of put up in the abstract that aren’t actually supported by the data. You really have to read, like I tell everyone the first thing you should do is read the methods because if you don't know what they did, you don't know what you can conclude, because whatever conclusions you can draw on the generalizability of the study will be very dependent on the methodology and how they did it, what they were actually doing. So you really need to know that. So that's a really common one that I see, and then another one I think is just sort of not being skeptical enough and I don't mean like…look, I'm not playing gotcha journalism or anything like that but I think that there is often you know, this idea that science is so cool, which I totally agree with by the way, but you know, a political reporter doesn't go in, you know trying to promote politics or trying to promote a party or whatever. And I think as science journalists, we need to apply the same sort of skepticism, you know my job is to cover science with warts and all and you know, they're often, you know, science is done by human beings who are flawed like the rest of us and so it's important that you know we're not cheerleaders.
John Bailer: I love the comment you made in an interview, where you were saying, what kind of evidence might overturn what you were thinking about this now. So that ties to that healthy skepticism that you mentioned, the idea that you know, you could refute this assertion that's being made in research. Is there a particular story when you've found that kind of data that completely changed your perspective on the story?
Christie Aschwanden: I have a recent example of this actually. So at Five Thirty Eight, we did a project on Sex-Ed and it was actually, we had intended it to be bigger than it was. We ended up just I think there were just a few stories and we did a chat about it. But I went in, so my part of this project was to go talk about the evidence on Sex-Ed, sort of what works and what doesn't and of course, you know, the first problem that you run into, which is true of any kind of science is, well, how do you define “working”, like what is Sex-Ed trying to do and that's where it's sort of like, Oh my God, this is going to be much harder you know this is not going to be an easy thing to do. Anyway, but I had known sort of going into it and had heard about research basically showing that abstinence only programs weren’t effective and specifically at reducing teen pregnancy and STDs and things like that but I sort of had this impression that there were other programs that did work and were really you know shown to help with these things. And so what I found when I went in and looked at the evidence was that it's actually really hard to measure whether these things are “working”, like just that the research problem of studying this is extremely hard and so although it was true that abstinence only programs didn't seem to make any difference, like they really weren't making a dent in these things that people care about, like teen pregnancy and you know the onset of sex and things like this, these other programs weren’t much better and the things that “worked” still had pretty modest effects and so you know what I really came away from this thinking is like wow this is a really tricky problem and like maybe you know a school based program isn’t going to be the be-all and end-all of you know, how we do these things that people are trying to do with these programs, and then you also have these problems with you know every school is different, every population is different, so it's just a much more difficult and tricky problem than I think I understood going into it.
Rosemary Pennington: Christie, you are credited, that is the word I was looking for, you are credited with creating sort of a Bechdel test for science writing. Could you explain what exactly that is and why you felt compelled to create this thing?
Christie Aschwanden: Sure, so it's called the Finkbeiner test and it's named after my colleague, Ann Finkbeiner, and it's all started it's actually all her to be honest I just was the one who sort of said hey we should really make this a thing. So Ann, yeah, Ann is a brilliant writer and she writes a lot about astronomy and she had had an assignment…well first, I guess I should back up and say that Ann and I both blog at this group science blog called Last Word on Nothing, that’s how I know her, yeah, and so anyway she had written a post for Last Word on Nothing about how…she had been assigned to write about a female astronomer and she wrote this whole post about all the things she was not going to write about. She was not going to write about you know, how she deals with her child care, she was not going to write about you know, how she nurtures all her students, she was not going to write about, let's see, this is terrible…I'm forgetting all the different parts but she was not going to write about how she was taken aback at how competitive the field was and basically, she was sort of saying, I'm not going to write about her in a gendered way you know, I'm done you know, making this like, isn't this amazing, how this person is both an astronomer and oh my God she's also female, can you believe it? And I thought it was brilliant and I thought you know these are really good rules, we should just make this into a thing and so I took what she had written and basically laid that out into rules and published it and it sort of took off and right around this time, the New York Times did this pretty famous obituary that mentioned you know she made a mean beef stroganoff in the lead which was obviously you know this was a brilliant sort of and so anyway that sort of helped it get some attention. But I think the takeaway here is just really thinking about the ways that we as journalists can reinforce you know, unhelpful gender stereotypes in the way that we write about people and I think this goes not just for women but also for men. There's actually an academic in the U.K. who did I think it was a master's thesis using the…I think binary test, sort of based on that and she had looked at stories in the U.K. But the interesting thing was she also did the Finkbeiner test for men, so sort of flipped it around and she found that things were just as gendered for the men you know where they were like described as really strong and all of these other stereotypes that aren't necessarily positive and so I think it's something that we should think about you know regardless of the gender or those sort of identity groups that a person might belong to.
John Bailer: You know, you’ve talked a lot about the idea of bias and objectivity being encouraged in science and journalism and I'm curious how you would put the idea of discussions of bias and objectivity into a context that when we hear a lot of allegations of fake news or descriptions of reporting as fake news, so how does the bias and objectivity run counter to that?
Christie Aschwanden: Yeah, so I think one of the important issues here is just to realize, I mean, we are all sort of vulnerable to motivated reasoning, like so, if you really want something to be true, like you're very susceptible to believing things that are made up to get you on board and vice versa, if you're also sort of primed to not like someone or not like something or feel certain way and so I think that we have to be really careful about this and always be asking, how do I know that this is true? What are the sources, and things like this I mean I think in terms of getting my audience to believe that the things that I'm writing are true and are not “fake news”, it's hard. I mean if someone is primed to think that global warming is a hoax I probably am not going to be able to convince them because the things that I'm holding up as evidence may not be received as evidence on their part you know if they are feeling that their talking heads, talking points are stronger evidence than my scientists’ data, there's not much I can do about that. But I think that we just have to and you know it's hard it feels these days like truth and numbers and all of this feel a little bit impotent in these times. But I think you know in the end the truth does win and reality is reality, whether we choose to believe it or not. You just have to keep going.
Rosemary Pennington: Well Christie, I hate to leave it there but that's all the time we have for this episode of Stats and Stories. Thank you so much for being here.
Christie Aschwanden: Thanks so much for having me, it's been a pleasure.
Rosemary Pennington: Stats and Stories is a partnership between Miami University's departments of Statistics and Media Journalism and Film as well as the American Statistical Association. You can follow us on Twitter, Apple podcast or other places you can find podcasts. If you'd like to share your thoughts on the program, send your email to statsandstories@miamioh.edu or check us out at statsandstories.net and be sure to listen for future editions of Stats and Stories, where we discuss the statistics behind the stories and the stories behind statistics.
(Background music plays)