Brad Efron is Max H. Stein Professor of Humanities and Sciences and Professor of Statistics at Stanford University, and Professor of Biostatistics with the Department of Biomedical Data Science in the Stanford School of Medicine; he serves as Co-director of the undergraduate Mathematical and Computational Sciences Program administered by the Department of Statistics. He has held visiting faculty appointments at Harvard, UC Berkeley, and Imperial College, London. He has been recognized with the 2018 International Prize in Statistics.
+ Full Transcript
John Bailer: Every once in a while, in any discipline a breakthrough is made that revolutionizes the fabric of the way we understand things. We have Newton and gravity, Watson, Crick and Franklin and the structure of D.N.A. and Bill Walsh with the West Coast offense. In the world of statistics one of the largest breakthroughs in the last century is something called bootstrapping. That's the focus of Stats and Stories where we explore the statistics behind the stories and the stories behind the statistics. I'm John Bailer, Chair of Miami University’s statistics department. Stats and Stories is a production of Miami University's departments of statistics and Media, Journalism and Film as well as the American Statistical Association. Joining me in the studio is Richard Campbell of Media, Journalism and Film, Rosemary Pennington is away today. Our guest is Brad Efron, the creator of the bootstrap technique. A veteran statistician, an academic and now one of two people to be awarded the international prize in statistics. First, congratulations on being awarded the international prize, Brad!
Brad Efron: Thank you very much!
Bailer: Well you know the natural question to start, Brad is, can you provide a description of what the bootstrap is?
Efron: Well yeah, I can. I’m asked that often and then asked again when my description doesn’t work. The bootstrap is a general device for assessing the accuracy of a statistical estimator, familiar from the kinds of plus or minus numbers you see after a political poll. The trouble is that political polls are very simple so you can have a formula that goes back a couple 100 years for a plus or minus. For more complicated statistics, the formula doesn't exist, and the bootstrap is a device for using the computer to get the plus or minus as if there was a formula. And I could tell you more about how the bootstrap is done if you want to hear more.
Richard Campbell: I would like it, and this is the journalist, so no, at the beginning, the only thing I know about statistics is whatever John tells me.
Efron: Okay! So Richard.
Campbell: So my question is, how do we know the bootstrap is accurate?
Efron: Well, that's a question that's been the subject of more than a thousand theoretical papers. The accuracy is founded on a basic theory. The basic theory is really very simple behind it, but of course people have done, and I've done lots and lots of cases, where you know the answer, to see if the bootstraps give you the right answer, and I could tell you more about how that all goes. But look, let me just tell you, I’ll give you an example.
Campbell: Okay!
Efron: So, right now I'm working on a problem from the medical school. I consult with the medical school when I have time in the Medical School and Dr. Lu, a young doctor over there, she works with diagnostics for cancer and she is interested in tongue cancer and she has 245 slides from mice and she has a new method, we'll call it, based on a pictorial thing, where they take a picture of the tongue and measure frequencies, energies and frequencies, and then she thinks it works really well, but the older method involves looking at things like colors and textures and now she has two big matrices for each of these 245 mouse tongues. She has her new method and the old standby method and so we have two huge matrices that relate the two things and measure the two things and she wonders what the relationship is. So, a colleague, a nurse and I looked at the data a lot and we worked out a complicated method of getting a correlation between the two things. And then the question comes up, what's the accuracy of that correlation and now you run some sort of plus or minus but there's no formula. So here's how the bootstrap works. You have these 245 slides you can imagine putting them in a hat, drawing out one at a time, each time putting one back. So you'll get some repetitions and some misses and that'll give you, by the time you get 245 of them, you'll get a different correlation using our same method. You can do that as many times as you want, and then the variability of these made up samples of slides 245 are the bootstrap estimate of the variability of the original number. That's what gives you the plus or minus. So I know that's a pretty quick description, but the basic idea is, the computer allows you to resample the original data lots of times. You could never do that by hand. And then each time you resample it, you re-compute your statistic. You get a bunch of those and before you know it, you have a measure of how variable your original statistic is.
Campbell: Very good. I understood, I think, most of that actually! Thank you!
Efron: So, if indeed I ran a statistical poll and it said 60 percent of the people were Democrats and 40 percent were Republicans, you could use that ancient formula to get the plus or minus. You could also use the bootstrap putting the 100 numbers in a hat this time and you'd get the same answer.
Bailer: Very good. So what motivated or led you to the discovery of the bootstrap?
Efron: Well this is a lesson in having smart colleagues, amongst other things. My thesis advisor and then my colleague when I was first here at Stanford was Rupert Miller. There was an older method called the jackknife coming out of John Tukey’s work, and Rupert had written a paper about whether the jackknife was dependable. And that started me thinking about…the jackknife was a method of getting plus or minus on the computer. But it had its limitations and I started working on that. I was at Imperial College. David Cox, the previous winner of the same prize, the 1st winner of this same prize suggested to me that it would be a good idea to work more on my idea of the relationship, and that's what got me going on.
Bailer: So everyone would love the name of this, I mean it was a brilliant name to come up with. So could you talk a little bit about the background of why did you call it the bootstrap?
Efron: So, first of all, where does the name come from...in the German story Baron Munchausen, he has many eccentric adventures. In one of them, he's stuck at the bottom of a lake and he's drowning and he reaches down and grabs his own bootstraps and pulls himself out of the lake.
Efron: And when you boot your computer, it's the same root.
Bailer: Ah!
Efron: And if you look at my original paper, which was 1979, right at the end there's a joke about other names I could have chosen.
Bailer: Oh OK! You got to tell us! What are some of the other ones that were candidates?
Efron: Well there was…I remember I was working with the jackknife in mind and that was Tukey’s jackknife which he said was a general tool for analyzing data in a rough and ready way and my joke was that I should have called this a shotgun for blowing the head off of any problem.
(Collective laughter)
Efron: And I don't think John would have appreciated the humor in that!
Bailer: No!
Efron: Anyway it's my only paper that ends with a joke.
Campbell: You know…the use of stories, I read one place where you talked about, that the numbers only become knowledge when we understand what they mean, you know, that you have to be able to explain the data, the numbers. You also have used a metaphor that I just love, statistics is about learning from experience, and then you said it is a detective game. And I really like that metaphor a lot. I have used that to talk about what journalists do, that they're detectives, searching for clues often, in order to tell the right story.
Efron: The thing about clues and good detective stories is no one tips you off completely. It's an accumulation of a lot of little clues that get you to the answer and that's the way statistics works.
Campbell: You called it a slow game.
Efron: It is a slow game indeed, and it's not a very natural way for people to think. Most people like that story in which the “Aha”, The Smoking Gun and stuff like that. Well, if it was all smoking guns, there won't be any statistics.
Bailer: So what led you to become a statistician?
Efron: So this is the usual story where you read the…child of the famous football player becomes a famous football player, that sort of thing. My Dad who was a salesman, loved numbers. He was a good athlete and then, after he was an athlete, he was the analyst for baseball. He kept score for baseball leagues and bowling leagues and so there were always a lot of numbers around my kitchen table and I didn't realize it but I was being trained to be a statistician.
(Collective laughter)
Bailer: So what do you like best about working as a statistician, about your career as a statistician?
Efron: Well, the most fun is doing applied statistics. So I’m having a lot of fun with Dr. Lu’s data and I'm sure Dr. Lu is enjoying this fun as much as I am. But I just love taking apart a data set and using not fancy stuff but old fashioned stuff a lot. You know, I was trained by Rupert Miller, I mentioned before, Lincoln Moses, Bill Brown and over there in the biostatistics workshop most of the things you did were T-tests and student T’s and things like that, very simple things, but they were so good at boiling a problem down to the essence and what they simply told you was, you had to look at the data and you really had to see the data. You know, you get a bunch of data like this tongue thing, it just looks like a huge pile of numbers and then you start taking it apart by taking combinations or averages or correlations and things like that and pretty soon you start seeing something, but it takes a lot of looking. When I do good work in the applied world it's because I have the energy to do a lot of looking.
Campbell: So we talked before about statistics being a slow game. So as the journalist representative here, I would argue that a lot of journalism is a fast game, where information and news often has to be developed in a story in one day. Could you talk about your experience with journalists and what you see as a problem sometimes? Because what journalists do is, you know, a lot of times what we understand from scientists and statisticians is what the journalists tell us, because they're often the ones that tell the story about scientific work or statistical work. So can you talk a little bit about you know, what journalists do well and maybe what they don't do well?
Efron: Well I'm a big fan of scientific journalism. So each Tuesday the New York Times has that science section and I eagerly read that, I clip out articles that I think have illustrated one point or another. There are very often articles that I think are way too credulous of what they're reporting, especially things in health. You know, this new treatment may very well cure Alzheimer's or something like that. Well, things that cure major disease are few and far between, and most of the time, the scientists, like anybody else are trying to put the best face on their results. So journalists are at the mercy of the scientists who report this stuff usually and I wish the scientists would be more modest about what they claim. I've had good results, good experiences with my own stuff been quoted.
(Background music plays)
Bailer: All right! You're listening to Stats and Stories where we discuss the statistics behind the stories and the stories behind the statistics. I'm John Bailer with Miami University statistics department and I'm joined by Media, Journalism and Films Richard Campbell. Today we're talking about bootstrapping with Stanford University's Brad Efron. Brad, the bootstrap has been cited by over 200000 articles and 200 journals since 1980. What is the most interesting or even unusual application that you've seen addressed with bootstrapping?
Efron: Well my favorite is always the last one I was working on. That's the tongue data right now! There's nothing particularly unusual about it. What I liked about it was how just straightforward going about the bootstrap without barely thinking how to use it. Once we had the statistic we wanted in mind I gave the answer. And the bootstrap is usually not the star of the show, because it's designed…you're working statistician, you've got a problem, you think of a statistic to estimate something you're interested in and then as a second step, you use the bootstrap or something else to see how accurate your first step was. So the bootstrap is sort of the best supporting actor kind of role. But recently, there has been more attempts to use the bootstrap in a primary way and I'm thinking on these kind of machine learning techniques the method called random force for example, directly uses the bootstrap to get the original statistic. And so that's different. So I've been quite interested in that, because I never thought of that when the thing was started. I was always interested in the secondary question, the plus or minus question.
Campbell: This reminds me of another quote I saw of yours, where you said, one of the charms of statistics is that you can peek into other fields and it's also reminded me of one of my attractions to journalism, is when you know, you get to learn a lot about things you don't know very much about. Could you talk a little bit about…while statisticians are trained in specific ways, I mean one thing that I've always admired about John is he's always involved with different scholars, doing different kinds of work and he's the statistician on board. Can you talk a little bit about the joy of peeking in the other fields?
Efron: Yeah well I get a lot of that. So I’ve been half the time in the medical school my entire career and I know no more about medicine than anybody else on the street. I know a lot about doctors and how they work and I'll tell you, it's fun for a statistician to work with doctors, because first of all, they have a pretty good natural feeling for randomness because medical things are so…the way that humans react to things is so random and noisy and they generate interesting questions usually. So yeah…statisticians… I've worked also some of the time with astronomers and that's been my main other one, occasionally with social scientists and stuff like that. Some of my colleagues are much more determined, over in the medical school for example, to become really experts on some particular disease, maybe mouse tongues, we don't have any of those right now! And they become part of the basic group and that's become a model over at the medical school, I'd say more often than not. The big studies have…the doctors have to incorporate a statistician as the working group and that's quite different. But for myself, I've always liked the amateur method - they bring the statistics to me and I ask questions and then hopefully we get somewhere. My mentor Lincoln Moses used to talk about having a good data-side manner.
(Collective laughter)
Efron: He really did! He was so good at listening and so was Rupert Miller. The thing is that when the doctors come to you, they often hide parts of the story, because they think you're not interested in the little details, whereas they've taken lots of measurements on each thing, but they only bring you the average. Well, we want the original measurements and stuff like that. So asking a question about what data do you have is a good one.
Campbell: You know that's another characteristic you have in common with a good journalist. Good journalists, the best journalists are good listeners. They're looking for the good story, they're paying attention, they're asking follow up questions. So that's another interesting thing.
Efron: Yeah. Journalism really interests me. They've gotten better on statistics stories in the last 10 years or so probably because there's more of them.
Bailer: Yes!
Efron: More statistics stories. The graphics have got much better.
Bailer: Yes.
Efron: That they use.
Bailer: So you mentioned this I think wonderfully, the Moses quote about the good data-side manner. Do you have a favorite Rupert Miller quote or piece of advice?
Efron: Well Rupert was a man of the clearest thinking I've ever seen. If I was going to teach a class I just go to Rupert, get his notes and they were just perfect. If I missed the lecture, a seminar, I’d go look and there was Rupert and you'd have the notes written down in clear form where you could actually understand the lecture probably better than if you'd been there. So Rupert's main thing was clear thinking. I’ll tell you one Rupert Miller story. When I was first here, Rupert was selected as a fellow of the American Statistical Association and he turned it down on the grounds that these honors aren’t fairly distributed, he said. They gave it to him anyway but it made a big impression on me. So I shouldn't - maybe I should turn down the I P S!
Bailer: No no no no!
Efron: But no I don't have that kind of grace he did.
Campbell: So I'm a little bit interested in - first of all, you've complimented journalism here but you also have dabbled in this, and I was wondering, your sort of attraction to stories. I mean back in Stanford, where you were the editor of what I call the equivalent of the Harvard Lampoon at Stanford, you got in some trouble, but somehow you must have had some kind of editing and writing experience to be able to do that job, as I think you were a grad student at the time right?
Efron: Well, at Cal Tech where I was an undergrad, I wrote a weekly column in the paper called The 5th Column or the 5th one over and it was humorous or tried to be humorous in a sophomoric way.
Campbell: So where did that interest come from? You talked about your father before and the numbers being around the house, but I would say, present company not included, most statisticians are not known for having good writing abilities. So where does that come from?
Efron: Well first of all let me defend most statisticians!
(Collective laughter)
Efron: I’ll tell you, statisticians have a better sense of humor than most scientists, because they have to, because they get to deal with most scientists. I have no idea but I've always loved humor writing, I love reading humor and there just isn't enough of that in the world. The main trouble with the statistics literature is it's so technical and difficult to read and putting a little humor is just something that breaks the mood for a second. It’s a really good idea! Just a few turns of language, it doesn't have to be belly laughs…just that unusual use of language, maybe it’s just a word that you don't ordinarily see, can really help things along. I was the editor and I was the founding editor of The Annals of Applied Statistics and I had hoped to make the journal more fun to read than most high level journals and I didn't succeed at all because as soon as the editor, I was the chief editor but I had editors and associate editors and referees and they just enforced that same kind of writing.
Bailer: Yeah. That's a real…huge challenge, I mean I was thinking back on things like was it a box that talked about proceeding a test of means, the test of variances, like sending out a rowboat to see if the waters were calm enough for an ocean liner to sail?!
(Collective laughter)
Efron: That's very clever, I’ve never heard that! George was very good, he doesn't fall into the writing category.
Bailer: Yeah, that's one that always tickled me, because it’s just such a wonderful image and it's a great idea to think about sensitivity, in a way that doesn't feel sort of boring and plain, and I like that. Well, when you think about how data analysis and research is reported, is there particular aspects when you see it done in ways that drive you crazy, that are there certain things that you know, you just recognize and go gosh! I wish they wouldn't do it that way but they often do?
Efron: Well yeah. A lot of stories about statistics and statisticians used to call the people mathematicians and now they call them computer scientists. We should just call them statisticians, because we exist too.
Campbell: Do you think we have a brand problem?
Efron: Yes, we do have a brand problem. Trevor Hastings and I just wrote a book published a couple years ago on a computer age statistical inference and they really were determined to put in a subtitle that said data science in it so we put it in that said data science. Well I don't mind the term data science but the term statistics is 200 years old at least in the current way as it is thought, at least 100 years old so it's at least as worth saving, as say the word astrophysics or something, but that's a small gripe.
Campbell: You know I've also heard you talk about you know, we're living in these kind of uncertain times and you know I think both for journalist and science and statistician and sort of questioning of science and data, you know, the sort of fake news conundrum…I consider your work at Stanford Chaparral, that was real fake news back in those days!
Efron: That was supposed to be fake! People took it seriously! That was how I got into such trouble.
Campbell: But you had a nice quote I saw in one of your presentations where you talk about politics undercutting scientific facts. Could you talk a little bit more about that and if there's anything we might be able to do, especially in a world that as you talk about it's a slow game and we live in a world that's moving so fast that we seem not to have the patience for the slow game…
Efron: Well you can't have a better example than climate change,
Bailer: Yes!
Efron: Which is a very slow game and I must admit, for the first 10 years I didn't believe it. I thought it was a sort of hype…I didn't think it was fake, I thought it was stated excessively. But I got it, it's turned out to be worse than even the hypers thought and I have a feeling that, you know, politics can go along for a long time until something real intrudes like a Hurricane Katrina, or something like that and then there's a bracing moment when people face up to what's really at stake and that's when we should be ready, we being the scientists in the world in particular, and statisticians to make our case honestly and not with hype.
Bailer: So let me ask you, what advice would you give to students and others preparing to work in a data rich world?
Efron: Well, I have a lot of them. I run an Applied Math major here at Stanford and I'm asked that question by people who want to know what to do. The advice I usually give, which is intended for people who have some kind of data, math computing interest, is to learn some statistical inference, not just go for the computer stuff. The computer stuff is like eating sugar and stuff for people, you know it gives you a high, and it is wonderful stuff but there's a reason underneath it, there's a reason why something, A is done and B is not and learning anything about inference is much harder than mastering the latest computer program or computer package. So, that's my advice…learn something about the basics of Bayesian, Fisherian sharing and inference, in particular, learn something about parametric models which are fading out of the data science kind of curriculum. Everybody does what we would call non-parametric systems.
Bailer: All right! Well Brad, thank you so much for being here today. That's all we have time for today.
Efron: I enjoyed it.
Bailer: Us too!
Campbell: We did!
Bailer: Thanks again! So, Stats and Stories is a partnership between Miami University’s departments of statistics and Media, Journalism and Film and the American Statistical association. You can follow us on Twitter or iTunes. If you’d like to share your thoughts on the program, send your e-mail to StatsandStories@miamioh.edu. And be sure to listen for future editions of Stats and Stories where we discuss the statistics behind the stories and the stories behind the statistics.
(Background music)