Joel B. Greenhouse, Ph.D., is Professor of Statistics at Carnegie Mellon University, and Adjunct Professor of Psychiatry and Epidemiology at the University of Pittsburgh. He is an elected Fellow of the American Statistical Association, the American Association for the Advancement of Science, and an elected Member of the International Statistical Institute.
Professor Greenhouse is a recipient of Carnegie Mellon University's Robert E. Doherty Award for Sustained Contributions to Excellence in Education, the Ryan Teaching Award for Meritorious Teaching, and the College of Humanities and Social Sciences' E. Dunlop Smith Award for distinguished teaching and educational service. He was Associate Dean for Academic Affairs in CMUs College of Humanities and Social Sciences (1997-2002) and Vice-Chair of CMUs Faculty Senate (2015-2016).
Episode Description
When reporters cover mass shootings news outlets often struggle to find ways to cover the event that won’t inspire others to do the same thing. Something similar follows in the wake of a suicide. Journalists don’t always cover suicides in their communities but when they do one of the concerns is whether that coverage might lead to a spike in suicides after the story is out. The media’s influence on the actions of individuals is a chronic concern for researchers in a number of fields and is a focus of this episode of Stats and Stories with guest Joel Greenhouse
+Timestamps
Why did you study 13 Reasons Why? (1:23)
What data did you use in this? (2:20)
How did you choose the age ranges? (7:40)
Editing the episode (9:45)
What’re the limitations of this research (10:56)
Correlation/causation issues? (13:21)
Any outrageous stories about your work? (15:02)
Most surprising result of your work? (16:15)
Contagion with suicide (18:24)
How would you follow this study up? (20:23)
+Full Transcript
Rosemary Pennington: When reporters cover mass shootings news outlets often struggle to find ways to cover the event that won’t inspire others to do the same thing. Something similar follows in the wake of a suicide. Journalists don’t always cover suicides in their communities but when they do one of the concerns is whether that coverage might lead to a spike in suicides after the story is out. The media’s influence on the actions of individuals is a chronic concern for researchers in a number of fields and is a focus of this episode of Stats and Stories where we explore the statistics behind the stories and the stories behind the statistics. I’m Rosemary Pennington. Stats and Stories is a production of Miami University’s departments of Statistics and Media, Journalism and Film, as well as the American Statistical Association. Joining me are regular panelists John Bailer, Chair of Miami’s Statistics Department, and Richard Campbell former Chair of Media, Journalism, and Film. Our guest today is Joel Greenhouse. Greenhouse is a professor in the Department of Statistics at Carnegie-Mellon University. He’s an elected fellow at the American Statistical Association, the American Association for the Advancement of Science, and an elected member of the International Statistical Institute. Last year Greenhouse co-authored a study examining the association between suicide rates in the U.S. and the Netflix show 13 Reasons Why, a program centered upon the suicide of a teenage girl and recently released its final season. Joel, thank you so much for being here.
Greenhouse: Thank you, Rosemary, thanks for having me.
Pennington: So, there are a lot of you all on this paper on 13 Reasons Why, can you talk a little bit about what made you want to study this show and how you went about doing it?
Greenhouse: Sure, I’ve been involved with a collaborative group for a number of years which has been interested in psychiatric disorders, mental health, and in particular, we’ve been following suicide and especially suicide in young children and adolescents. And so the lead author, Jeff Bridge, is a psychiatric epidemiologist and once the show released, he was familiar with what many of these issues that you actually laid out in the introduction and was waiting for the right time when data were available to actually do this analysis.
John Bailer: Can you describe a little bit the data that you used and then some of the- at least at some sort of broad level like what kind of analyses you did?
Greenhouse: Sure. The data actually is freely available from the Center for Disease Control. There’s a database called Wonder which keeps track of different types of outcomes like injuries and causes of death, and the reason we were waiting until late Fall of 20- what was it? I guess 17-18, 2018 sorry- before doing this analysis was that there was a lag before they actually made available the monthly suicide rates for the previous year. So that’s what we were waiting for and that’s what the delay was. So, the basic dataset was monthly suicide rates broken down by gender and by age, those were the factors or features we were particularly interested in and then the – so those were the data. The approach was sort of a classic time series approach where we were following how the variations in suicide rates looked across time. And there are some interesting features in the data like this. So one looked for seasonal trends and in suicide kids, there is a seasonal trend; it’s high during the school year, drops during the summer and furthermore over a period of time for the last five-six years there’s been an increasing trend in suicides in young kids. So, we had two features of the data that we were needed to deal with. One was a trend and then seasonal variation.
Richard Campbell: So, can you talk a little bit about your findings because you broke this down I believe into three different age groups and the real finding was among the youngest group that 10-17 group.
Greenhouse: Yeah, so let me start by saying why we wanted to break it down into the three age groups. The background- the sort of suspicions about suicide contagion is that the relationship or the relating to the victim has a bigger impact on people of the same age and gender. So the woman- the protagonist in the story- was a 17-year-old woman, and so we wanted to break out the age range so that we could look at those that we thought that was at the highest of risk, but also looking at these other age ranges provided a potential sensitivity analysis to see whether or not there was maybe something else going on that might be more general affecting the population as a whole, or really was more tailored to this age group that we thought would be at risk. So the findings were, again, the approach was because we had this history of suicide rates by month going back to January 2013, we could model the data and make a prediction as to what we would have expected to see following the release of the series, and as you know in Netflix the series gets released completely so we were looking at the months following it but particularly we were interested- we saw- we wanted to look at the couple of months following the release of the series. And so, from the model, the statistical model we had a prediction of what we would have expected to see in April 2017, and then we had what we actually observed, and it wasn’t really – it’s not from a statistics perspective. We also wanted to take into account the variability of the prediction, and so what we found was that the observed rate in April 2017 was outside the constancy interval of what the model had predicted and that was consistent with a spike in the 10-17 age group. We did not see a spike in the other age groups; 18-29 or 30-64. So that was pretty much what we had anticipated, so that was one finding and then looking at the rest of the year from April 2017 to December 2017 we saw a general excess in suicides in this age group across that whole period relative to what would have been expected based on the model.
Campbell: And you actually – I don’t know what the time period was, but it was about 200 deaths more than what was normal before the airing of that show.
Greenhouse: Correct and the time period would have been from April 2017 through the end of December 2017.
Bailer: You know, one thing when I looked at the age intervals that you grouped, I thought it was interesting you said these were kind of developmental stages- that is one of the reasons for doing this but I was curious- and I was going all the way down the ten, you know just sort of my really naïve, ignorant perspective on this was I would have thought maybe a sensitivity analysis would have focused on 13-17, not the very youngest of the set. Can you help me think through this a little bit?
Greenhouse: Sure, so we actually had an earlier paper where we looked at suicide rates in young children as young as ten, and surprisingly that paper reported an increase in suicides in that age range. It was shocking.
Bailer: Oh, my goodness.
Greenhouse: So, I think- so based on that earlier work we thought we could use a wide window.
Campbell: Did you happen to see that second study that was done in the second season? There was an article that appeared in social science and medicine but it was just 18-29-year-olds and it kind of tracked in the second season they found that the number of viewers that quit watching the show experienced more thoughts of self-harming thoughts than the group of viewers who kept with the show the second season and I think one of the reasons they thought there was more discussion in the show of characters talking about suicide working through that verbally but also it was an 18-29-year-old group that was tested here it wasn’t 10-17 and just sort of common sensical, you can see that students first dealing with- kids first dealing with suicidal thoughts don’t know what to do, don’t know how to process them, don’t know how to talk about them. So, I don’t know if you’ve seen that study but it seems very kind of incomplete to me when I saw that there had been a study of the second season.
Greenhouse: You know as Rosemary sort of mentioned in her introduction, there’s actually been a lot of research on how the media should be presenting events like this for example suicides. And a lot of concern was expressed even before the release of the series. So unfortunately, and again I wasn’t involved with any of this, that wasn’t really followed, and I think in retrospect based on Netflix, really sort of a year later before they released the second series- actually went back and edited the suicide scene. And so I think in retrospect and reflecting on the results of this study and other studies like it, realize that that was probably a mistake in that it was a better way of presenting these events which were more consistent with the guides to media.
Pennington: Media scholars are constantly struggling with this issue of finding effects in relationships between content and behavior and so I wonder if you can talk a little bit about how your group- your colleagues and you approached this study thinking through- because it’s really hard- you know the hypodermic needle has been disproven over and over again but there certainly seem to be research that suggests there are relationships between content and behavior. So maybe you could talk through what you guys were thinking about as you were putting this study together to make sure you were keeping that stuff in mind.
Greenhouse: That’s a really good question and a really good point. That is, there’s a limitation to this type of research and it has a title, it’s called an ecological association. And psychological in that we don’t know among those kids who committed suicide whether they even watched the show, right? So, we don’t have that link. And this is a challenge that happens a lot in epidemiology and statistics and in other fields as well, the social sciences. So, the best we can do is show there’s an association, and then ask the question could there be other reasons for this association and look at that. So, for example when we looked at these other age ranges and didn’t find the effect that sort of supported the hypothesis that we had put forward. We also did another sensitivity analysis which was we looked at homicide rates over this period. And the thought was well if there was something secular going on then perhaps it would have affected- we would have seen a spike in homicide rates as well and we didn’t. So again, we haven’t shown causation in any way, but we’ve been trying to build up a body of evidence that makes a strong case for that association.
Pennington: You’re listening to Stats and Stories and today we’re talking with Carnegie-Mellon’s Joel Greenhouse.
Campbell: So, I wanted to follow up on that- something that I think a lot of journalists don’t do well which explains the connection between causation and association or correlation. And in the study you talk- I think that’s one of the first things you talk about in the limitations of the study. Is this something that you find when your work is covered- I know that CNN picked it up and probably others any frustrations you have with the way journalists make this- you know they’re going to tell their own stories to the public and that’s how you’re going to be seen mostly right? Through the interpretations that journalists make of your work. Can you talk a little bit about that?
Greenhouse: So my experience- not necessarily with only this particular study, but when I’ve actually had a one on one with journalists, the ones I’ve spoken to have been very sensitive to this issue and it really wasn’t hard – in fact usually they were ahead of me in terms of recognizing that we weren’t making a causation type claim. What happens after that I’m not sure, but I think this particular study got a lot of headlines. In fact, I just got a notice that it was ranked among the top hundred altimetric- I’m not exactly sure I know what that means; I’m not exactly a social media person. But it sounded like that was really good.
Pennington: It means a lot of people are seeing it.
Greenhouse: So, I really did get out there and it seems to have struck a chord for many reasons.
Bailer: So, when you looked at some of the ways this was covered was there a particular outrageous headline that really jumped out at you when you looked at some of the coverage of your work?
Greenhouse: No, I didn’t- I really didn’t cover that carefully, but the things that I did see in the local- I’m in Pittsburgh and the local Pittsburgh Press and a couple of things that were in the national headlines all seemed pretty good.
Bailer: Oh good.
Greenhouse: Yeah, I think this was so- much a startling headline anyway that you didn’t really have to go out of your way to sensationalize it.
Campbell: I thought it was good that Netflix took some steps to edit out- I didn’t know that. I remember how much this series was hyped when it first came out, and I remember watching part of the first episode and I said I’m not watching anymore of this. Just it wasn’t the kind of story that I was interested in but I was also kind of worried about it because they were talking about kids that were under 18 and they were going to be watching this show so this was not a surprise, what you found at all.
Bailer: So, in your analyses what was the most surprising result in the work that you did, that would have run counter to your expectations prior to doing the work?
Greenhouse: Well, I’m always surprised when the analysis works.
[Laughter]
Greenhouse: So, there was one aspect to the analysis which was really still a little bit of a puzzle; it’s kind of interesting. Which is the series was released in March- March 31st the last day in March, but we also- using our models saw a peak, an increase in suicides in March itself. So before this series was released, and it was actually statistically significant, and so that- again this is taking into account the seasonal variation and taking into account the trend and one thing we hypothesized was that starting at the beginning of March was there was a trailer for the series promoting it and [inaudible] in preparation for this interview I actually went and found on YouTube that trailer and it doesn’t show the suicide but it’s pretty graphic about what’s going on, what happened, it shows the woman, it shows these men, it sort of has ominous music, it’s very dark and you know this is not going to be fun. So, we hypothesized that perhaps this was a trigger and it’s important to remember that this story first appeared as a book which was on the New York Times Best Seller List maybe 10 years before the Netflix series and it was very popular. So, there were many many young people who had read it and knew the story and so whatever effect that might have had.
Bailer: You- this has come up a couple of times and I’m wondering if you can maybe just explain what this is this idea of contagion when it comes to suicide and I also know it concerns reporters when it comes to covering mass shootings, maybe you could take a moment to talk about what that is, and maybe given what you know from the scholarship what might be some of the best practices for journalists that are covering these things? Because you know there’s always this tension between what is the public needs to know and then the fear of, you know, are we giving people too much information?
Greenhouse: So, I’m not really an expert on contagion, but my understanding of it is really very much in the same usage as we think of infectious diseases. That there’s an index case if something happens and then it spreads through the population and then of course who is at risk, who is likely to get infected, are all questions of interest. My understanding about the presentation in the media about events such as suicide is that what you don’t want to do is sensationalize it to make the individual look like after they're dead that they are some sort of hero and they get all sorts of rewards, which you’d rather do in a more positive way in presenting this would be that there are options other than suicide. That there’s a way that you can get help and I think that’s what the critics were trying to suggest in terms of the Netflix series that they could take a different approach whereas instead of sensationalizing this but actually presenting- my understanding presents it in a much more graphic way than was even in the book that they presented in the series really sort of went against all of the recommendations based on all of the research and experience.
Bailer: If you were going to follow up this study- this ecological association study, you’re saying okay here’s a bunch of money for you to design and follow up this with a study what would that look like? What would you do now if you could study this? Or maybe in a different program.
Greenhouse: Yeah, that’s a good question. I think that just sticking with the Netflix study we were looking at data on completed suicides. We do know that men complete suicides at a higher rate than women. It would be really interesting to go back and look at, say, E.R. records and look for, you know, self-harm data during this period, and do a similar analysis to see if there was an increase in that. I think getting that sort of data along with the suicide rates if we saw a spike in April, again we’re building up a case that would be fairly compelling, I think.
Bailer: I did like this part of the sensitivity analysis; you also not only included the other responses control response being homicide, but also the other age groups where you thought there should be nothing going on, but then you also did some other sensitivity analysis both including and excluding certain months. I thought that was kind of nice to see. You’re trying to protect yourself from potholes. So how did you go through thinking about okay we need to worry about this one or we need to worry about that one and redo this analysis for this concern?
Greenhouse: Right, I think what you’re referring to is in particular this concern about the spike in March before the series was actually released. And so the way we tried to deal with that is in building the model that would that we would use to predict the occurrence of suicides in April, we included March in that model that was one analysis and we took March out of that model and then so made- built the model without that March 2017 data in there. So it would be less influential and so we did several different analyses by including it taking it out making predictions based on the model that even sort of backed up a little further and to see what might- and whether or not the results still held up.
Campbell: I’m wondering if- you know there was a big spike I think there was another study on how many people looked up how to commit suicide on Google, I mean it went like it was over a million hits more than usual and I wonder if that somehow coincided with the promotion of the series, that people knew it was coming, maybe they read the book but I was sort of shocked at the Google study- how many kids were online looking that up.
Greenhouse: There was another study that came out after ours by addressing the same question appeared in [inaudible] but they had sort of an additional analysis where they looked at Tweets and Instagram references to this Netflix series, this 13 Reasons Why starting in April, and they saw a huge spike in the number of references to it which then dropped in June, which sort of coincides with the drop in suicide rates. What I would like to have seen is what those rates were prior to April because that would have been really interesting to see because, again, it might actually have informed us of that March peak, because if we saw a lot of activity in March in social media that would have actually sort of supported our speculation that there was a lot of attention being paid to the series. Pennington: Well Joel that’s all the time we have for this episode of Stats and Stories. I think Richard could probably keep talking to you all day about this but thank you very much for being here.
Greenhouse: Thank you very much for having me and again thank you guys very much for this series because I think it contributes a lot to what we’re doing and how we understand it, so thank you very much.
Bailer: Thank you, Joel.
Pennington: Stats and Stories is a partnership between Miami University’s departments of Statistics and Media, Journalism and Film, and the American Statistical Association. You can follow us on Twitter, Apple podcasts, or other places where you find podcasts. If you’d like to share your thoughts on the program send your email to statsandstories@miamioh.edu or check us out at statsandstories.net, and be sure to listen for future editions of Stats and Stories where we explore the statistics behind the stories behind the statistics.