William C. Thompson is Professor Emeritus of Criminology, Law, and Society; Psychology and Social Behavior at the UCI School of Social Ecology interested in human factors associated with forensic science evidence, including contextual and cognitive bias in forensic analysis and the communication of scientific findings to lawyers and juries. He has written about the strengths and limitations of various types of forensic science evidence, particularly DNA evidence, and about the ability of lay juries to evaluate evidence.

Episode Description

+Timestamps

Rosemary Pennington
Just a reminder that Stats and Stories is running its data visualization contest to celebrate its 300th episode. You can grab data about the show to analyze and submit your entry at statsandstories.net/contest. Your entry has to be there by June 30.

Death happens in medical settings for all kinds of reasons. However, when a death is unexpected, it can leave loved ones grieving and investigators wondering whether it was a case of medical misconduct or medical murder. When investigators decide to bring a case to trial, they often rely on statistics to help them make their argument. The Royal statistical society released a report last year on stats in such cases. And that's the focus of this episode of Stats and Stories where we explore the statistics behind the stories and the stories behind the statistics. I'm Rosemary Pennington. Stats and Stories is a production of Miami University's departments of statistics and media journalism and film as well as the American Statistical Association. Joining me is regular panelist John Bailer, emeritus professor of statistics at Miami University. Our guest today is Bill Thompson. Thompson's a Professor Emeritus of Criminology, Law and Society, psychology and social behavior at the UCI School of Social Ecology. He's interested in human factors associated with forensic science evidence, including contextual and cognitive bias and forensic analysis, and the communication of scientific findings to lawyers and juries. He's written about strengths and limitations of various types of forensic evidence, particularly DNA evidence, and about the ability of lay juries to evaluate evidence. Thompson was one of the co authors of that 2022 Royal statistical society report on statistics and medical misconduct and murder cases. Bill, thank you so much for joining us today.

Bill Thompson
Happy to be with you,

John Bailer
Bill. Thanks. Thanks again for jumping on with us today. I'm curious what led the royal stat society to think we need to put together a panel and really dive into this?

Bill Thompson
Well, as I understand it, the Royal Statistical Society has been approached a number of times over the years by individuals who are concerned about the use or misuse of statistical evidence in these cases of alleged medical misconduct. In cases where say a nurse was accused of killing a number of patients and so on. Those cases often turn on statistical evidence or statistics play a prominent role. And there have been a number of allegations about misuse of statistics in such cases that found their way to the royal statistical society. So I think this was an issue that the society had been thinking about looking into the idea of, of preparing a report came out of a this there's a statistics and Law Committee of the Royal statistical society which I happen to be on at the time this issue came up and and also on the committee at the time was Richard Gill, who is a statistician and physicist at Leiden University, who had had some involvement in in a prominent European case involving a nurse who was convicted of serial murder. But then the case was re-examined and she was later exonerated. So Gill, who had a lot of expertise, was there, and another member of the committee was Julia Matera from University of Rome, who had had some involvement in an Italian case involving a nurse named Daniela Puglia, who had also been accused and convicted of serial murder. But under circumstances that raised a lot of concerns about misuse of statistical evidence. And so there were some people with expertise there. And then there were some people like myself who had never thought about the issue. There were some lawyers in social science types like myself, there was a Scottish lawyer, and there were some very serious well trained statisticians like Peter Green, from Bristol University, so that there we had the expertise and there we decided to attempt to put together a report on it. Partly I think it's just because it was fascinating to us. And there's so much to say about these cases, and we thought it would be useful to pull it all together in a report that might be useful to people who are involved in doing criminal investigations. So, in these cases, like prosecutors and police and hospital officials, even defense lawyers, and what do they need to know, to, to think about these cases sensibly? So we decided to see if we could pull together a report. And we got it to the point where it could then be reviewed by the Royal Statistical Society and then go through their process to adopt it as an official publication.

Rosemary Pennington
So what kinds of stats are we talking about?

Bill Thompson
Well, the stats come up in various ways, but one of the most prominent ones is simply a p value for, for looking at the, quote, statistical significance of some often have a relative risk ratio. So what will happen is that it will be observed that an unusual number of patients may have died, you know, so in the neonatal intensive care, or they always have certain deaths, but during a particular period, maybe the rate of deaths goes up. And everybody's very concerned and what's happened and then it's noticed that nurse Jones was always on duty, or seemingly was always on duty, when these deaths occurred, and and maybe there are other reasons to think nurse Jones may be, you know, there was the Gothic costume she wears, and the fact that she, at the Italian nurse, I guess, at one point dressed as the angel of death as a Halloween costume, and this caused concern, but there may be other issues that could issues of concern. And everybody you know, and it's there, I think some of these cases do appear to have certain elements of scapegoating. To them, it's almost like a witch hunt. Right? So things are going terribly wrong. And well, maybe it's witches who are responsible. So I think that there's that kind of dynamic here. Although having said that, I also want to say that it's, it's probably not good to compare these entirely to witch hunts, because we're pretty sure that there weren't actual witches. And we're pretty sure that there are actual serial killers out there who are medical guys, so, you know, doctors and nurses who have, for reasons that are often inexplicable, decided to kill people. So it's not, you know, so there's there there is a, there are actual cases where this happens. And we need to distinguish them from cases where coincidence or other factors may have created the appearance of guilt and somebody innocent. And it's not easy, it's not an easy challenge.

John Bailer
You know, you're describing this, and I find myself thinking a lot about disease clusters. You know, there's a small town with a lot of cases of a certain type of cancer, and oh, by the way, there just happens to be a plant in the community. And it may or may not be associated, but it's often directly attributed to it.

Bill Thompson
Right? And, and, yeah, no, I think that's exactly the analogy that we started with, you know, when, when you're when you statisticians are often brought in and asked, if these deaths are occurring randomly, at the rate at which we think death normally would occur? What's the probability that by chance, we would have a cluster of so many, during the periods when nurse Jones is on duty, you know, and it's possible to calculate a p value and often probability looks extremely, exceptionally low. And then that then becomes what appears to be a very solid, scientifically based piece of evidence, strongly supporting guilt. But, you know, it can be misleading in so many different ways. Um, some of it is, some of it is that people don't understand what a p value means, though. So there's a lot of the case when we looked at the evidence and how it came in, and how the lawyers argued, there are a lot of difficulties with fallacious interpretations of p values, and particularly transposing of the conditional probability, you know, a p values and you know, is says something about the conditional probability of getting certain evidence, certain evidentiary results under a hypothesis. Right. But so often, when that's discussed in court, it's transposed into the probability of the last hypothesis being true or not a true condition on the evidence, which is clearly fallacious. But lawyers and judges and jurors don't always, it's not always easy to see the fallacy. And so it's easy for them to go from whether it's, you know, from a statistic like there's only one chance in 10 billion to 10 million that we would see so many deaths on nurse Jones watch. By chance to, to say well, there's only one chance and 10 million that this many deaths occurred by chance, which is the very argument that was often made problematic. And so partly it's that it's partly misinterpretation of statistics and partly the cluster idea also that, you know, maybe there's like only one chance and 10 million, suppose we've computed that correctly, we may not have as far as I can go on in a minute. But assuming we've computed that correctly, that's one chance in 10 million for any particular nurse to have that many deaths at a particular time. But you know, how many nurses are there in the world? Right, you know, there really are, there might be, you know, there probably are easily 10 million nurses probably in the world somewhere. And so even if it's an extremely rare thing for that cluster of deaths to occur for any particular nurse, it's quite probable that somewhere, sometime there's going to be a nurse that has that kind of cluster occur. And, and so how do we, how do we then distinguish the nurse who's really killing people from the nurse who's just been implicated by coincidence? And again, I think that's what prompted the report was the sense that this is not easy. And it's easy for people to get what can easily be confused about these numbers. And therefore, we need to try to lay this all out for them in a sensible way. So we made the attempt, and I hope it has some benefits, I hope people will actually read it who need to read it.

Rosemary Pennington
In the report, you have these case studies where you walk through how stats played out in particular cases, do you want to walk us through one of those to kind of explain sort of the case what was at stake and sort of how stats were used?

Bill Thompson
Or I think the kind of prototypical case is this case of a woman known as Lucia, whose name was Lucia de Berk, they called her that should do today. But she was a Dutch pediatric nurse. And she worked in a hospital. And it's kind of the typical case, because she was working in a hospital and an unusual number of deaths were occurring. Apparently, there were other reasons that she had leukemia, although there was evidence that she actually was quite a good nurse. I mean, she was particularly energetic, she worked long shifts, she worked extra shifts and things like that, which might be part of the reason that an unusual number of deaths were occurring on her shifts is because she was doing a lot of extra shifts, and she was doing shifts at the tough times and taking on the tougher case and so on. But an unusual number of deaths occurred. The hospital was under a lot of pressure about this, this raised a lot of concerns about the management of the hospital. And the managers, I think, may, you know, there's some suggestion that they may have found it easier to blame the the unusual, that number of deaths on on a particular individual than to acknowledge that maybe they were there some procedural changes and how they were handling high risk cases, how they were handling genetic defects and so on that, that, you know, that that may have, you know, there may have been some management decisions that that contributed to the increase in death rate, that death rate that, you know, that were more easily that it may have been more easy to blame it on an individual than to acknowledge a systemic issue. But in any event, she was tried and convicted in Dutch courts twice. And, and a prominent part of her prosecution was this computation of p values, the very low probability of seeing this many deaths by chance. But, you know, it turned out when when the case eventually attracted the attention of some academics, and particularly Richard Gill, who, you know, almost by happenstance, they were drawn into the case by people that they knew who happened to be peripherally involved and thought maybe even CIO was getting railroaded and said, you know, you're a statistician, look at what the statisticians were saying when they testified. And, you know, another issue in Lucia de Berk is that the people who testified to the statistics were, I would say they were not well trained statisticians. They're not professional statisticians, not academic statisticians. They were medical doctors or medical people who had maybe had some statistics courses, but perhaps we're not as sophisticated about uses of statistics as they could have been, but they were basically, you know, computing p values in ways that were reflected biases and were using them or drawing conclusions from them that were somewhat fallacious. And when that became, you know, when academics like Richard Gill wrote articles pointing this out, and this was re-examined by the court, and they asked, Well, what else do we have other than these botched statistics? The Dutch courts ultimately decided that there was not enough. I mean, there were, you know, when you go looking for things, you know, you can often find some things that look a little incriminating, you know, and so Lucia was, she did tarot card readings. And so you know, and in tarot cards, there's cards of deaths, and so on. So apparently, she had been doing some tarot card readings with some of her patients, and so there's some suspicious things that may have arisen from her, her fascination with occult things. And there was a diary that was seized during the investigation in which she talked rather ambiguously about having compulsions or dark urges, or something like that. But I don't think it was clear that her dark ages were about murdering people as opposed to some other dark things she might have done. But I mean, I think, you know, you can imagine that any of us if our lives were examined closely enough might have some things that might suggest that we could be secret killers. And so they found there were a few of those things, but they didn't amount to much. So ultimately, Lucia was exonerated. And, of course, the courts declared that she had been falsely convicted. And that became kind of a prototypic case. There have been many others. And these cases, I have to say, they appear to be quite rare, but they're not so rare that we don't see them regularly. And we see them in many, many different countries, as you know, there's Italian cases, there's a Dutch case, we've had several in the United States, several in Britain. And, you know, and once in a while there are cases where that involves serious serial killer medical personnel who who were the evidence makes it look like they're pretty darn guilty, like this case of Harold Shipman, and that we talked in the report about the Harold Shipman case in Britain, where, where this doctor who was treated elderly patients, and many of them ended up dying unexpectedly and which probably should have raised suspicions earlier than it did. Then, particularly when it later turned out that he apparently had been involved in altering their wills to designate him as a beneficiary. And, there was other evidence that he was, so apparently, and there was another German case where there was a doctor who was killing many people. I mean, these are inexplicable cases. And so as I say, it's not that this is not a witch hunt, where we're looking for a non-existent category of crime. But it's a very unusual, rare kind of crime that needs to be difficult to distinguish from innocence.

Rosemary Pennington
You're listening to Stats and Stories, and today we're talking to Bill Thompson about a royal statistical society report on stats and medical murder cases.

John Bailer
It's interesting, as you've been talking about some of these cases, that when it's only a statistical argument of rarity, or unusual illness of observation, that there's more to the story. I mean, you talked about the hospital having slightly different changes in processes that might be associated with greater mortality. You mentioned that in the Shipman case, that there was this other evidence that surfaced, it seems that this body of evidence is a critical part of telling and investigating a story like this, not just just relying on the unusual illness of the result. So it's, it seems like that there's like, by definition, young people that have opportunity, and possibly means, but how are the house motive established in some of these cases you gave us one was Shipman talking about the wills being modified, but how does that play out and understanding whether something like this is happening, where the motive is often very difficult to understand?

Bill Thompson
And I mean, often, the alleged motive is something like trying to gain attention. Or trying to put patients into a dangerous medical situation so that this person could be the hero who saves them, but maybe, you know, maybe doesn't quite work out all the time. So creating a medical crisis and then being the hero that solves that, and that kind of so those are the alleged motives. And, you know, this, in fact, you know, could well motivate some people to do terrible things to get attention. You know, and often the people who have been accused in some of these cases are, seem to be sort of strivers, people who are trying to get ahead, you know, people who are trying to look outstanding relative to their colleagues. And so, so sometimes, you know, the allegation is they're, you know, they're trying to cheat to look heroic. But in fact, it may be that they actually are sincere strivers, and they just, you know, they take on the toughest cases, and they they work hardest, and they stay, they stay there right before and leave after their normal shifts and, and see, you know, they stick around when that when a patient is on the edge of death, they stick around them to try to everything, you know, that kind of stuff. So, these are tough cases. And then I should also mention the part of these cases that I found particularly fascinating as a psychologist is that there's a lot of opportunities for contextual and other kinds of cognitive biases to creep into the cases in ways that affect the statistics. Right. So So and I think, in some ways, that's the the strongest part of this report is that we look at investigative procedures and how the investigative procedures can lead to biases, and how those biases and can then distort the statistics to create to make something that's that's innocent look guilty or potentially can. So it's things like, when we're trying to compute statistics on the probability of getting so many suspicious deaths, how do we determine what's a suspicious death? And how do we determine that a suspicious death occurred while a particular person was on duty? Like, when did we say their duty starts or not? And suppose you know, somebody who died right after they went off duty was that while they were on duty, I mean, so there's a lot of ambiguity about this, about the interpretations. And and what we were seeing, when we looked closely at the evidence in these cases, what we were seeing is what appeared to be evidence of profound bias in the way that the cases were classified to start with so while deciding whether the death of this particular infant, was this a suspicious death, or is this a typical death that we don't need to be concerned about? Well, if the person making that determination knows that the person died, while the alleged serial killer was in attendance, that might influence their judgments. In fact, I suspect, you know, we reviewed in the report a lot of evidence from psychological studies and studies in forensic science, suggesting that this kind of contextual factor, like facts that might lead an investigator to suspect a particular individual can change the way that they classify and interpret the evidence. So you're more likely to say that this was a suspicious death, if it was, while the alleged serial killer was around, or you're more likely to say it was well, while he or she was on duty, even though it occurred a little afterwards, but maybe it just took a while to detect the death. Or maybe it occurred in an adjacent Ward, not where they were working. But maybe if you investigate, you could find somebody saw them walking by, so maybe, you know, you could maybe, so there are ways of where the boundaries between you know, how you classify the cases have served as sort of fuzzy boundaries. And if you're defining the boundaries in a way that's gold directed toward the person who's already accused, that you can build, you can build a mountain of evidence out of randomness. And so part of what's really nice about the RSS report, I think, and this is Peter Green, who's one of our writers, helped come up with these, toy examples to show how you could take data that was not statistically significant and make it appear to be statistically significant by introducing minor biases in the investigation process. And so, showing how that can happen, realizing that can happen, that then tells you Well, if we really want to do these investigations, well, we have to take steps to try to avoid having that happen. And so we have some discussion in the report about how to do that, that is how to improve investigative procedures to reduce biases, and thereby improve the quality and the trustworthiness of the statistics that emerge from the analyses.

John Bailer
So you had about eight recommendations that came out of this report. And you're hinting at least one of them when you start talking about masking the kind of the experts to some of the conditions when data is collected? Yeah, could you talk about some of the other recommendations that were part of the report?

Bill Thompson
Well, I think there are quite a few of the masks in it. I think, from my perspective, that's probably one of the most important ones in its investigation. The other investigations or the other recommendations are that the experts need to be mindful about what p values mean, what they don't mean. So you need to avoid misinterpretation of the P values. Professional statisticians should be involved in these cases. So because these statistical issues can be kind of subtle, you know, having somebody with an undergraduate course in statistics, computing your p values is probably not a good idea. So having a statistician, it's really important that those doing the investigations involve medical and other medical professionals and other experts, who are likely to be aware of the full range of other variables that might affect their rates of death. So pretty tricky. Yeah. So often, one of the factors that's been important in some of these cases is that individuals get blamed for deaths that turn out later to be caused by other factors. There was a Canadian case, for example, where a number of infants were dying. And there was an investigation of a particular nurse who was thought to be involved. But later, it turned out it was not, it was the silicone tubes that were used for intubation, or something were causing allergies, right. So there's an unexpected change, there was a British case where the surgeon deaths corresponded to the hiring of a new nurse who came under suspicion, but it also corresponded to a change in the supplier of infant formula. And it turned out to be some contaminated infant formula that was causing the deaths. So knowing what those other factors could be, and having people independent of a hospital administration investigating those factors. So often, these investigations are initiated by hospital administrators who might have their own dirty laundry to hide right, and so, so or so it's good to have, it's good to bring in independent investigators.

Rosemary Pennington
I am curious Bill, what the response has been to the report since it's been out?

Bill Thompson
You know, I've heard good responses from academics who have read it. So the academic response has been doing a nice job summarizing a whole bunch of stuff, you know, so I think we've gotten kudos from our academic colleagues for being able to pull together a lot of disparate things into a nice package. We have not gotten a lot of good feedback from lawyers. I mean, I should mention, there's a case going on in Britain right now, involving a nurse named Lucy, who's being prosecuted for serial homicide of infants, she was an infant care nurse in a national health service Hospital in Britain. And then, when that case came up, members of our committee sent copies of the report to both the prosecutors and prosecution and defense sides, saying, you know, we understand you're dealing with these cases, you should look at our report. And the response that came back is, oh, well, you know, we don't need to be concerned with that, because we're not presenting any statistics, you know, in our case, our case does not depend on statistics. The problem is that they might not have any explicit statistics, they might not have anybody presenting p values, but they are dealing with the same issues of the unusual number of deaths, the unexpected numbers and so on, and how whether deaths are classified as suspicious or not so, I think lawyers have this odd idea that you're not taught, you're not thinking about they're dealing with statistics, unless they're actual numbers. And, and what they don't seem to understand is a lot of what statisticians bring to the table isn't about the numbers per se, it's about the logic of inference that goes into drawing conclusions from data. Right? And that's what statisticians are really good at. And that's what they bring to the table. But the lawyer is not understanding that they say, Oh, well, no numbers, no statistics. I don't need to worry. I'm not going to read your report because you know, we don't have numbers though. Are you no dummies? We need to, we need to educate them about this. I'm not quite sure how to do that. But, I suspect that we need to do more outreach between the statistical community and the legal community to bring this issue to their attention, because really, they're missing a bet if they were there, and they could well be doing injustice. I mean, I'm not going down the details of the Lucy lead FBI case. I don't know about the details of that case. But I would bet that the people who were looking at particular infant deaths and assessing whether they were suspicious or not, were not blinded to whether she was involved. I bet you know, they did other things that could well have distorted the data that they are interpreting. So anyway, that's my take on it. I think. Academics love it, lawyers haven't quite realized that they need it.

Rosemary Pennington Well Bill that’s all the time we have left. Thank you for joining us. Stats and Stories is a partnership between Miami University’s Departments of Statistics, and Media, Journalism and Film, and the American Statistical Association. You can follow us on Twitter, Apple podcasts, or other places you can find podcasts. If you’d like to share your thoughts on the program send your email to statsandstories@miamioh.edu or check us out at statsandstories.net, and be sure to listen for future editions of Stats and Stories, where we discuss the statistics behind the stories and the stories behind the statistics.