Regina Nuzzo (@ReginaNuzzo) is a science writer and professor of statistics at Gallaudet University. Her writings on science, medicine, health, statistics, and the scientific research process have been published in a variety of outlets, including Scientific American, Nature, ESPN, Science News, Reader's Digest, New Scientist, and the Los Angeles Times. Her p-values feature for Naturewon the 2014 ASA Award for Excellence in Statistical Reporting.
+ Full Transcript
Bob Long: You’ve probably had the experience of sitting down to read an article about a new study that really seems relevant to you, but what if you found out the reporter misinterpreted the data that was the basis for that story, or worse yet, what if the failure to understand biases caused the scientist to produce a study that was flawed. Our guest today on Stats and Stories has written that in today’s environment our talent for jumping to conclusions makes it all too easy to find false patterns, or to ignore alternative explanations for a result, or to accept reasonable outcomes without any question. I’m Bob Long we welcome you to another edition of Stats and Stories; it’s a program where we look at the statistics behind the stories and the stories behind the statistics. Our discussion today, about science journalism, is going to focus on why the media are uncomfortable with uncertainty and what we can do to fix that through statistics. Before we talk with our special guest, Stats and Stories reporter Reis Thebault tells us about the website Sense About Science USA that helps journalists connect with statisticians to improve the accuracy of their stories about science.
Reis Thebault: Journalists, for a long while, have had a dirty little secret: we’re not that great at math. We’re right brain-type people.
But, as big data and computer algorithms become essential tools in investigative reporting, it’s now more important than ever for journalists to get their numbers straight.
Neda Afsarmanesh is the deputy director of Sense About Science, a nonprofit organization dedicated to helping the journalists who are treading through the often murky waters of science and research reporting.
She says, when it comes to databases, it’s not enough to simply memorize a couple Excel functions.
Neda Afsarmanesh: “So you definitely need to have a savviness about how you’re analyzing them or what questions you’re asking from the people who created the datasets. It’s not just about knowing the statistics that you can use, but also it’s about understanding the data, how the data was collected, and what’re the questions you need to ask scientists about that.”
Reis Thebault: But Neda says the public also has some work to do. We're in the age of Twitter, where all news must be breaking and each story needs to be bigger and bolder than the one before it. She says readers need to curb their expectations.
Neda Afsarmanesh: “It’s also how we’ve been treated to understand science, that every story needs to be on the front page whereas much of science builds on itself and slowly builds up to a conclusion. I think as readers, as viewers, as the general public, we’ve grown accustomed to these stories with the types of headlines we’re talking about and I think that’s something that needs to change.”
Reis Thebault: Neda says that this stuff is so important because science and statistics underpins our public policy and, really, our entire lives.
Neda Afsarmanesh: “Much of the personal and societal choices we make are based on the news we consume and that’s where we get most of our information and if that’s not accurate or if that’s not a good overview of actually what’s going on in the world and what we should be concerned about and what we should care about, then we aren’t really as informed as we think we are. Society is based on evidence and how you look at evidence and how you analyze it and you need statistics for that as well. You know, scientific process, evidence, statistics, these all go hand in hand and influence our lives.”
Thebault: For Stats and Stories, I’m Reis Thebault
Long: Joining me for Stats and Stories are our regular panelists, Miami University Statistics Department chair John Bailer, and Media Journalism and Film Chair Richard Campbell. And our special guest today is freelance science writer and Gallaudet University professor, Regina Nuzzo. She has a Ph.D. from Stanford, a graduate degree in science journalism from University of California-Santa Cruz and her writings have been published on science, medicine, health, statistics, and the process of scientific research in publications such as Scientific American, Nature, ESPN, and the Los Angeles Times and she also teaches statistics in American Sign Language. Welcome to our show, Regina.
Regina Nuzzo: Thanks so much.
Long: I wanted to start off though, since I’m a former journalist myself, with some comments I read that you had about how the media are uncomfortable with uncertainty. It strikes me that in 2016, that’s probably more true than ever before, but I wanted to have you talk a little bit about why you think, in general, the media is uncomfortable with uncertainty and what statistics can really do to help that situation.
Nuzzo: I think that’s a great question, and to be fair it’s not just journalists that have the problems with uncertainty, not just the media. I feel like our brains are hardwired to give us black and white answers, we’re biased towards binary sorts of things, and nature is more in shades of grade, so we are already wanting, trending towards more certainty than the world can offer. Then you add the pressures that the media have; we have limited space, we’re not really sure who our audience is, we have to make sure that we get in and get out very quickly, reader attention span is way down, we don’t have the time, the room, and we can’t necessarily trust that our readers are going to sit around and wait for the caveat, and say ‘maybe, maybe not’ they want to know the answer, they want to know, especially with news you can use, which is often a lot of science, and health and medicine journalism, ‘how can I apply this to my life today?’ And for you to say ‘it might be true, it might not be true there’s an 80% chance, here are the reasons why and why not.’ Who’s going to sit around for that?
Long: John Bailer, we’ll go to you for the next question.
Bailer: Thanks Bob, so Regina, how did you get into this science journalism game after starting out a career, first as an engineer and then as a statistician?
Nuzzo: You know, writing was always my secret passion, and I would run around in graduate school with a New York review of books, stuffed into my folder, and I would read that on the sly instead of reading my stats journal, so that was always my secret, guilty vice. I didn’t realize that you could actually make a living doing this, and I told my advisor in my post-doc at McGill University, that I wanted to write about statistics for a general audience, and he said ‘that’s not really rewarded in academia, so don’t do that.’ So I decided, ok, that’s fine, I’m getting out of academia, and went and learned how to be a journalist. I knew that I wanted to write, I wanted to take the message out to a bigger audience, because they want to know, they deserve to know, and I think that they’re capable of understanding these sorts of issues, and it’s vitally important in today’s society. So I went and got a degree in science journalism, fell in love immediately, immediately. So I wrote about science journalism for a lot of different things, I ended up writing a lot about sex, and health, and medicine at a variety of outlets and then I decided, ok, let’s try merging this with statistics, I have this background in statistics and I enjoy writing about science, let’s see if I can put them together in unusual ways.
Long: Richard Campbell, we’ll go to you for the next question.
Campbell: One of the things in journalism that I talk about is that a good journalist’s job is to translate stuff into common sense, and you have both perspectives, which is very unusual. So, I often have academic friends who complain about how bad journalism is, and I also have journalist friends who complain about why they can’t understand what academics are talking about. So, talk a little bit about how you’ve kind of negotiated those two worlds.
Nuzzo: That’s a great question, and I think that if more people understood the pressures of the other side then they wouldn’t be so quick to complain. Journalists are under a lot of time pressure, a lot of pressure from their editors, for their audience. Academics are under their own pressures, so with that, I also think, at the same time, that statisticians and journalists are really long lost cousins, you know, siblings separated at birth, twins separated at birth. I realized the more that I learned about science journalism, since I came to that late after statistics, that this is really the same sort of thing as statistics. So, statisticians they find… journalists, they find the story among the facts, right? It’s no good just to report the facts; you have to have the narrative in there, and statisticians, we find the signal among the noise. So, you’re taking all these data points and creating something new out of that. You know, in journalism, you have the adage the journalism professors say ‘If your Grandma says she loves you, be sure to check it out.’ Check out your facts, highly skeptical people, right?
Campbell: Yes.
Nuzzo: Same things for statisticians, I think that statisticians would translate it to ‘what’s the strength of evidence that Grandma loves you?’ But it’s the same thing…show me something. And the last reason I think that they would really make a good team together…journalists are very, very sardonic people. They have, what I would say is a good B.S detector, right? They can spot something a mile away. Statisticians have the same thing; they have what I call a bad science detector, so they’re able to see that, so I think that they make great partners in that way, more so than you might think.
Campbell: So this is why we get along John.
Bailer: Yeah, I thought there was a reason we were still talking to each other.
Campbell: She just explained it.
Long: You know the one thing…the one point you made right at the very beginning that I find interesting…one of the other problems and frustrations for journalists and scientists today is also the audience, because the audience wants, you touched on this, they want everything immediately, does that pose a real problem too, because, you know, you want to give the honest to goodness truth but they don’t seem to have enough patience sometimes to allow you to do that.
Nuzzo: It’s true, you know sometimes I think they might have more patience than we give them credit for and I think that the editors are conservative creatures, they need to be able to put these things out and they’re unwilling to take risks and I think there’s this natural tension between a journalist and an editor. I think it’s the journalist’s job to push, to include more caveats and to include a little more and the editor’s job to push back and say ‘no, no, no remember your audiences, keep it simple.’ But I think that increasingly, today’s audience is very savvy about uncertain and they want the facts behind it. Perhaps in the days where you would just pull open a newspaper and that was it you needed to be able to get through it quickly, but today people are used to clicking on hyperlinks and we have Wikipedia, people will get lost in Wikipedia, so they want that behind it and I think that they’re capable of slightly more uncertainty than we give them credit for, I think that there’s a lot that journalists can do to enhance that and I have some ideas that I have suggested to various people, but that’s my quick answer to that.
Long: Richard Campbell
Campbell: You just mentioned Wikipedia, and in preparing for this and listening to your talk today, I said, ‘well I’m just going to go on google and ask what’s a p-value?’, which was the subject, so this was the first thing that came up, this was on Wikipedia. “In frequentist statistics, the p-value is a function of the observed sample results (a test statistic) relative to a statistical model, which measures how extreme the observation is.” So, that didn’t help me at all.
Nuzzo: No, why would it.
Campbell: That’s right, so help me with this, and tell me a story that’s going to help our audience understand what a p-value is and how important it is.
Nuzzo: Well, this is my job to explain a p-value right here right now.
Campbell: Oh we have two of you that can…
Nuzzo: There we go, you know I like to say, and this is not a technical definition I would like to stress, that a p-value is a good index of surprise. How surprised or how skeptical should I be about my findings? That does not mean that it’s…a p-value can be a zero, a number between 0 and 1, that does not mean that it’s linear scale, that it nicely matches up, that .05 actually means anything 5% in a way that would make a difference to you. Smaller numbers mean you should be more skeptical, more surprised by the results, and I think that’s the simplest way of explaining it, when we start getting a little more technical and say ‘well what does that .05 really mean?’ I think that’s when we lose they lay audience, but John might have a different opinion of that, and you might have practiced something better than I have.
Bailer: You know I think that you have to couch it in the context of testing, and the standard type…when I try to talk about hypothesis testing, and Regina you may do the same, is think about it from the context of jury trials. That you start out with a belief about the state of what occurs for this particular person, this defendant, and the jury is told to believe that they are innocent; the alternative is that they’re guilty. So you have competing hypotheses about this, data is being collected to prosecute the hypothesis is innocent. If the data is sufficient, that leads you to reject that hypothesis and then declare guilt, if the data is insufficient you would say not guilty, there’s insufficient evidence to conclude guilt. In essence I think of a p-value as…I really like the index of surprise, I’ve never thought of that before but now I’m going to steal that and use it…
Nuzzo: Go right ahead.
Bailer: But I think that the idea is…there’s information that’s presented, it’s being processed by some data prosecutor to say ‘I don’t believe this hypothesis’ and this is just the way that’s quantified.
Nuzzo: I agree
Campbell: And one of the things that seems to make a little sense to me…like evidence, I’m comfortable with that idea, because that’s one of the things that I think statisticians and journalists have in common that good journalism is based on evidence. You know, have you looked at the data, have you gone the documents, and what do they say?
Nuzzo: I agree
Long: You’re listening to Stats and Stories where we discuss the stas…I can’t even say it… we discuss that statistics behind the stories and the stories behind the statistics. I always do that. Our topic is why the media are uncomfortable with uncertainty and what we can do to fix that problem. I’m Bob Long, our regular panelists are Miami University Statistics Department chair John Bailer and Media Journalism and Film Chair, Richard Campbell, and our special guest today is freelance science writer Regina Nuzzo who’s also a professor at Gallaudet University in Washington D.C. where she teaches statistics in American Sign Language. Let’s go back to John Bailer for the next question.
Bailer: Hey Regina, I’m curious, how has being a statistician helped you be a better journalist, and the complement, how has being a journalist helped you be a better statistician.
Nuzzo: Statistician how has that helped me be a better journalist? I feel like I have a secret weapon. My stats Ph.D. or any sort of stats degree has given me a leg up over other journalists. When I write science journalism, I can read the primary literature and I don’t need to stick around the press release, I can go beyond the abstract, I know how to get right in and understand, because I think that statistics is scientific process, crystallized. And so I have this bird’s eye view of how science works and how we’re marshalling data, we’re collecting evidence and making an argument for one theory or one hypothesis in favor of another. So I think that helps me bring that Skepticism and a real critical eye in that. But it also means that I know how to evaluate a study, and I’ve done that in a few of my stories, I’ve spotted weaknesses in the study and…Weaknesses, that’s probably a strong word…I’ve found things that should be caveat, and should be mentioned that scientists reading an article would understand but they need to be mentioned in the article for the popular press. So, I’m able to go to other statisticians or include in there, well yes its’ significant but the effect size is very small. It lets me add an element I think that a lot of other journalists wouldn’t.
Bailer: So how about in the other direction. How has being a journalist helped you be a better statistician?
Nuzzo: Well, that’s interesting. I have a healthy understanding of my audience and of narrative. So, really it’s doing the exact same things. Taking these complicated ideas, making them simple, making a story, and really understanding how to reach my audience. So journalism has taught me, it’s ok to use anecdotes, it’s ok to really reach your audience and use humor and meet them wherever they’re at, and so that’s what I try to do as a stats instructor, use humor, talk about sex, whatever works. Sex works.
Long: I think I read somewhere that you said that statisticians, as a reporter, are your favorite people, favorite resource. Talk a little about why…because a lot of people don’t understand the importance of being able to call somebody up on the phone that maybe can help you understand something sometimes.
Nuzzo: This is one of the things I‘ve really been talking to statisticians about and encouraging journalists, I say ‘put a statistician on speed dial’ because they are hugely skeptical and they can get right to the heart of the matter, they can talk about why a study or why a finding…put it into perspective in real life. What are the chances it’s true? What does this mean to you? And really translate it. Statisticians, when you get a good one, they know how to communicate, they know how to get that, and I really think they’re really an underutilized resource, I’ve been trying to encourage my colleagues, make friends, take them out for a beer, and get to know them.
Long: Richard Campbell, we’ll go to you know for the next question.
Campbell: So, we’re in a political season right now, and every day if we’re watching the evening news or the morning news we’re inundated with polls, and they’re often polls that seem completely contradictory to a lay audience. One of the things that I want to ask you…because I know that The New York Times when it does poll data will often tell you how they got these results, and they’ll report things that statisticians can often understand, but they’ll report them in a way that a lay audience can understand. I’m actually bothered as a former journalist and as an academic, how little they talk about what these polls actually mean. Because they actually drive narratives…they drive big narratives about who’s winning, who’s losing, so these large narratives are very powerful and they actually can influence people to vote one way or the other, and yet it seems that they’re built on a lot of uncertainty.
Nuzzo: Yes, this is one of my concerns with data journalism, I think in general, so if you don’t mind me broadening that out from polls. So, I’m a huge fan of data journalism and this is where journalists are basically acting as data scientists, are going out there, sometimes collecting their own data or using publicly available data, but yet, some of them have not been trained as scientists, and they haven’t been trained necessarily in statistics, so they don’t understand those caveats and they don’t understand how our brain is primed to find any sort of patterns real or not in the data, seize upon them as if they were real, and concoct stories, 'just so' stories, are sometimes what statisticians call them. So after the Kipling stories, of ok I see something that happened and I concoct this story for why this must be so, and I think it’s a danger in journalism if they’re not kept in check in the same way to say this may be real, it might not be real.
Long: John Bailer.
Bailer: So, I’m curious, what’s been the most unusual story that you’ve covered? Or you can change the question just like academics like to do to answer the one that you like, what’s been the most fun story, perhaps, if you’d like to answer that one instead.
Nuzzo: That’s a very interesting question. At the time that I’m working on it, every story is my favorite story, so I can’t say that. Definitely the sex stories have been interesting, the things I’ve written about sex just because people get so excited. Pun intended…not intended. But as an educator I feel that…when I’ve written a lot of stories about sexual science, it was a chance to sneak in some actual science, because someone will read it start to finish. And I will give them all kind of statistics, all kinds of science in there and they’re going to keep reading because it’s about sex, and they want to learn about sex. So I guess it’s a spoonful of sugar idea. So, it’s fine, again, I have no shame in that way, so I don’t mind, whatever it takes to get them interested, and I don’t do it in a salacious sort of way but I don’t mind doing that. So I manage to teach people a lot about statistics and science and they think that they’re learning about sex.
Long: You’re listening to Stats and Stories and again our discussion today is focusing on why the media are uncomfortable with uncertainty and what we can do to fix that problem and our special guest today is freelance science writer and Gallaudet University professor, Regina Nuzzo. I’m Bob Long and our regular panelists are Miami University Media Journalism and Film chair Richard Campbell and Statistics Department chair, John Bailer. Something else…I was reading some of your articles that you’ve written and I noticed one problem that you’ve raised in academia is that it’s extremely competitive today, especially where you’re supposed to pile up publications and things like that if you’re a professor and you want statistically significant results…the impression I got was that sometime the researchers because they have a stake in things, that can really cause a problem and I wanted to have you talk a little bit about that issue.
Nuzzo: I think that it’s great, and it’s something that I alluded to before, our cognitive bias, it’s towards finding patterns in data. Picture our brain evolving on the African savanna, you need to spot patterns rather than overlook them, the opportunity costs. Now translate that today, to a scientist sitting down in front of a computer, seizing on random patterns, not so useful, it was very useful to evade lions, and gather berries…not so useful anymore. So there have been a number of interesting proposals for how we can combat our own worst tendencies in data analysis, so I don’t necessarily blame scientists for doing these sorts of things, it’s just human nature, that’s it.
Long: Yeah, I was going to say, part of the problem is that they’re not deliberately trying to deceive people, but there are just biases that they’re just not taking into account.
Nuzzo: And they’re deceiving themselves and they don’t even realize it, so I don’t think there’s any malicious intent, there is fraud out there, but I think for the most part it’s just eagerness, it’s excitement, they get very excited about what they’re doing and they want to see things, they think they found a pattern, they want to go for it. So we have to design ways to keep that in check.
Long: John Bailer.
Bailer: I’m curious as you look at some of the resources that are out there now, what do you think are some of the best news coverage that you would see that really captures the story with the caveats and the nuance? So what are…if you were going to recommend to people to read to see that, where might you tell them to look?
Nuzzo: Am I allowed to promote my own work?
Bailer: Absolutely! We will.
Nuzzo: Not because I necessarily think because it’s a stellar way of doing it, but just because it’s fresh on my mind, I have stories about it. So for example, is it ok if I talk about penises here? No? Alright, that might be a no…
Bailer: No, that’s fine
Nuzzo: How about if I talk about dating?
Bailer: That’s fine, whatever you’d like to talk about
Nuzzo: Are you sure?
Bailer: You’re the guest
Campbell: She immediately noticed that all the men in the room got nervous.
Nuzzo: I know, I see the red faces
Campbell: I think you should go for it
Nuzzo: The blushing. There was a scientific article that was published in Proceedings of the National Academy of Sciences a couple years ago, and they looked at women’s preferences in penis size, and they had a very legitimate scientific experiment for how they did this, and it was a nice guy’s Ph.D. dissertation, so the interesting thing was that, and some people overlooked this, it was a non-linear effect. So, he actually fit a quadratic regression to this, non-linear regression and so, most people reported this because they saw ‘bigger is better’ so it was, it was bigger and better, until big was too big. And most people did not report that. So, I think it’s important to have those sorts of things in there, and not just go to the immediately most interesting sort of thing, which is bigger and better and this fits all of our preconceived stereotypes, so that might be one example, I don’t know if it’s the best one I have a couple of others, but maybe I just want to talk about penises.
Bailer: So then a quick follow-up so, the New York Times as Richard had mentioned has their little box where they’ll often explain the sampling method that was used. Now, you’ve mentioned some online sources that you liked when we were having conversations, what are some of those that you might recommend for folks to investigate?
Nuzzo: I really like what FiveThirtyEight and what Vox does, and I think we talked a little about this; it’s a much savvier audience in some ways so they can go in and handle the uncertainty. Wonkblog at The Washington Post does a wonderful job, so they tend to be the online ones, not traditional print, and so that means they have the space and agility to get something out quickly and to link to other things. I love Wonkblog I think they do a wonderful job at presenting what’s behind these sorts of stories and presenting data visualization, so I really like that idea.
Long: Richard Campbell
Campbell: So, one of the challenges in journalism education is getting students interested in telling important stories and significant stories that have numbers and data behind them and so on. How would you try to influence a student interested in journalism to take statistics, most of our students do take statistics but get them interested in even in double majoring, because we require a double major and I’d like to see more of that. I mean one way is to tell them that they can study penis size, so, that’s something that I hadn’t thought of before.
Nuzzo: Who can argue with that? If that fails then the backup…I really think that data journalism is extremely hot, and it’s where their future is. Right now people are doing it on their own, they’re learning R, the programming language or so they can sit down, they can collect their own data, or data that’s available from the government, from the public, I mean you go online, data everywhere, and play with it. They’re getting in there and they’re playing with it, they’re writing stories, they’re doing data visualization, basically they’re acting as social scientists or statisticians and writing stories about it, huge in demand. So, guaranteed jobs right there, and just thing how powerful you would be if you had formal training behind that instead of having to learn it on your own or learning it on the job later, so much better to do it now. So that’s how I convince people, maybe is just saying ‘you’ll have great jobs after you graduate doing fun stuff.’
Long: John we probably have time for one more question, so I’ll turn to you.
Bailer: Thank you. Regina, what do you like best about what you do?
Nuzzo: That’s a great question. I’ve said that I have academic ADHD, which means I’m interested in lots of different things, and I think it’s perfect for a journalist and perfect for a statistician. So a statistician, you get to peak into other people’s studies and say ‘ooh, let me learn about bugs today’ and the science that you’ve done about that let me helps you analyze your data. Journalism is the same sort of thing, ‘here what are you doing?’ and you get to talk to people about their passions, and their excitement, and I think it’s the same between statistics and journalism, so the idea of being able to put them together? Very exciting, it seems ideal to me.
Long: Regina Nuzzo we want to thank you very much for sharing your insights with us on Stats and Stories today. Regina is a freelance science writer and a professor at Gallaudet University in Washington D.C. If you’d like to share your thoughts on our program you can send us an email at statsandstories@miamioh.edu be sure to listen for future editions of Stats and Stories where we discuss the statistics behind the stories and the stories behind the statistics.