David B. Allison is the current dean, distinguished professor, and provost professor at the Indiana University School of Public Health-Bloomington. Prior to Indiana University, Allison was a distinguished professor, Quetelet Endowed Professor, and director of the NIH-funded Nutrition Obesity Research Center (NORC) at the University of Alabama at Birmingham (UAB). Allison has published more than 500 scientific papers with research interests including obesity and nutrition, quantitative genetics, clinical trials, statistical and research methodology, and research rigor and integrity.
+ Full Transcript
John Bailer: Red meat increases your risk of heart disease. A glass of wine is good for you. Carbs are evil and lead to a life of sedentary couch surfing. We’ve all heard these claims about what is good for our health, but what is true? Clinically evaluating these studies that serve as the foundation of these claims is the focus of this episode of Stats and Stories where we explore the statistics behind the stories and the stories behind the statistics. I’m John Bailer. Stats and Stories is a production of Miami University’s Department of Statistics and Media, Journalism and Film, as well as the American Statistical Association. Joining me in the studio is regular panelist Richard Campbell of Media, Journalism and Film. Rosemary Pennington is away today. Our guest is Dr. David Allison. Dr. Allison is Dean, distinguished professor and provost professor at Indiana University in Bloomington. David thank you so much for being here.
David Allison: Truly my pleasure.
Bailer: So, David how did you first get involved in obesity research?
Allison: I’ve actually been studying obesity at one level or another since I was an undergraduate. I took a course in my sophomore year titled Human Emotion and Motivation. And in that course we study the theories of experiments of Stanley Shachter who at the time was a very prominent psychological researcher and theorist form Columbia University and I was struck by the enormous creativity of his studies, and how his repeated experimentation demonstrated the so-called hypo deductive method of science He’d propose hypothesis, he’d collect some data – it would never work out quite right. He’d refine his hypothesis and then he’d collect some more data and the most creative and interesting experiments involving making clocks go fast and see if that made people feel hungrier sooner and things like that. And I love the idea that not only could one study obesity from many angles, meaning psychological, physiological, economical, genetics, etc., that in my view one needed to study from all those different angles to understand it. And that interdisciplinarity and that creativity of the experimental studies is really what attracted me.
Richard Campbell: David, this past summer I taught journalism in Kosovo in the capital city Prishtina, and one of the things I noticed was how few overweight people I saw in the capital of Kosovo. And when I got back I just looked up obesity data and saw that the US was- and I can’t remember what site it was or what study, but it said something like 36% of US adults were obese. And in Kosovo it was 20 or less than 20%. So just sort of my anecdotal observations confirmed there was a difference but my question for you is – we have this data for obesity for I think pretty much every country in the world- where does it come from? How do we know that this data is accurate? And how do you measure what percentage of the adult population is overweight?
Allison: How do we know is perhaps the best question that you or anybody else could ask in situations like this, it is the vital question. In this particular case, how do we know or how we know is a result of recording heights and weights and calculating something called body mass index or BMI for short. BMI is calculated as the weight in kilograms divided by the square of height in meters. Or if you’re hopelessly stuck in the metric system it’s pounds over inches squared and the whole thing multiplied by 703. That was devised by a man called Quetelet who was a Belgian astronomer, statistician and epidemiologist in the late 1800s, and a wonderful man. We still use it today. Some of those heights and weights are self-reported and that’s an additional problem because for places where they’re based on self-report they tend to underestimate obesity levels, and some of them are measured.
Bailer: And no one is going to report their height as less than it is or their weight as more than it is.
Allison: Interestingly thin men tend to overreport their weight.
Bailer: Oh really?
Allison: Women tend to universally underreport their weight, men tend to overreport their height, and they tend to underreport their weight when they’re heavy and overreport their weight when they’re light.
Bailer: Oh interesting.
Campbell: So in your study some of the things that I’ve read you talk about the difference between data collection and intervention, and that was interesting to me and I as long as we’re on self-reporting- you talked about when you’re trying to measure childhood obesity the self-reporting is often not very reliable there either. Can you talk about data collection, intervention and self-reporting a little more? Especially as it relates to childhood obesity.
Allison: Sure. Self-reporting has a long history of use in obesity research and many other fields. Sometimes it’s vital, sometimes it is valid for collecting data. Often it is a useful clinical tool even if it’s not valid as a scientific tool. For example, asking someone to write down what they eat every day is an effective way of helping people eat less and lose weight. Even though it’s not a good method of measuring scientifically how much people eat. It turns out not to be valid for that. In terms of the use of – the distinction between data collection and intervention, often people look at something like BMI in particular and will get into arguments on whether it’s good or not, and before taking a step back and saying good for what? And it turns out that BMI is pretty good if what you want to do is estimate the number the people who are overweight or obese in a population. It works pretty well for that; however, people sometimes get a little defensive and you have to remember the numerator of that is just weight or mass, it’s not body fat. And so a big strong NBA player or body builder is going to have a high BMI but not be obese, to which we reply of course we understand that anybody would look at such a person and know that and so from a clinical point of view that might not be the best way of defining obesity. But for just counting the proportion of the population that is obese it works just fine.
Bailer: I think it’s interesting to consider the measures that might be used to define obesity for adult populations versus measures that would be used to define obesity in a childhood population. So, does the BMI metric work when you start thinking about childhood obesity studies?
Allison: A little bit, but not quite as simply because there are normal developmental differences in BMI and so what is a normal BMI at one age in childhood is not the same as another, whereas it’s a little perhaps more constant over adulthood. And so, in children often the norm is used BMI Z-scores which are a nonlinear transformation because they’re based on historical- it’s not a standard Z-score in the statistical sense of just subtract a mean and divide by a standard deviation. It’s sort of norming it to some prior data and those are age corrected. So that’s the more common thing used for kids.
Bailer: So, then you just start worrying if a kid has too many standard deviations above the average weight for their age.
Allison: Right or people will just use percentiles and too many or too high percentiles.
Campbell: So, some of the problems and challenges studying this, some of the early controversies from- I think the beginning of 2010-2011- where there were studies of the impact of soda and soda manufacturers and I know that you were involved in some of this. The question I have is how do you tease out when someone is obese there are lots of factors involved, there are lots of variables involved in this, and how do you sort that? And I know sometimes what comes into this is the differences between correlation and causation, we’ve talked a lot about that on Stats and Stories. Could you tease this out a little bit for us?
Allison: Sure. There’s a distinction that’s often made in the legal context that I think may be useful in response to your question, which is the distinction between specific causation and general causation. So, for example, in a legal case where a person has been exposed to something x and has an outcome y, and we ask did the exposure to x cause the outcome for that person? Usually there’s first a general causation question that’s answered. Which is can x cause y in general? And then and only if the answer is convincingly yes, do we go to a specific causation to say does this person’s instance of having y result from there having been exposed to x? And those are two very different questions. The general causation question we have much better methods for dealing with. And the ideal of course there is randomized control trials. So in an ideal world we randomly assign individuals to be exposed to x or to not be exposed to x or to eat x or not eat x if it’s a food, and then we measure y and we do all the right controls and there’s good experiment and good randomized control trial, and then we have the ability to draw strong causal inferences. Many people will correctly say that there are times when practical or ethical or other issues we cannot do those randomized control trials, or we can but simply haven’t yet got those in hand, and we may sometimes draw tentative judgments about causation in the absence of those, but we have to then recognize that those judgments are tentative and not extremely well supported. Now the specific causation question is different. You say how do I know that this child’s obesity or this adult’s obesity is due to their having consumed x or not done enough activity or whatever it is. In those cases, it becomes much more conjectural and we don’t have such well-established methods, but you sort of look and say were they constantly exposed to this? And sometimes you can do things say if they withdraw it do they lose weight? Clinically that’s really how it’s done; clinically you just say if for example you had a child who drank a great deal of soda and was overweight, you might say I don’t know that the soda is what is making you overweight. You don’t need to drink the soda, so stop drinking the soda and let’s see what happens. And if you lose weight that’s great, and if you don’t probably good to stop drinking the soda anyway and let’s find some other things that we can try to change to help you try to lose weight.
Bailer: So I was really interested in the paper that you worked on with a team of colleagues last year that was related to childhood obesity and best intervention studies and you talk about this is a narrative review and a guide for investigators, authors, editors, reviewers and readers to guard against exaggerated effectiveness claims. I am glad they don’t charge by the word in your titles here… [laughter] That was a mouthful. So, what is it that inspired you and your colleagues to work on this paper? And try to clarify these issues for these different audiences.
Allison: I think it just comes from years of reading the literature, responding to journalist’s questions, of talking to teachers and parents and funders and public health people who are often confused and mislead by statements in the mass media. And when you trace those back they are often being misled by statements coming out of academics or other government researchers and so on and we often – in academia we like to blame the journalists a lot, and that’s okay because they think the journalists do a lot of poor reporting at times and deserve some blame, but guess what? People who have looked at this carefully have also shown that journalists often get the health stories wrong when the scientist has fed them exaggerated information. So, we’ve met the enemy and it is us. And it was that recognition of how much misinformation was out there and how it was leading to wasted resources and repeating the same failed efforts over and over that led us to write this paper.
Campbell: In working with journalists over the years what’s been your biggest frustration in terms of how your own work is covered? And what are some of the things that we can tell our student journalists when they’re covering science and statistical studies? This is complicated material and a lot of journalists aren’t trained in this. What should they be doing to ensure that they’re going to tell the best story they can?
Allison: Perhaps what they need to do is try not to tell the best story, and to tell the most accurate story even if it’s not always the most compelling story. And I think that’s where uncertainty comes in. There was a single word that I would say is at the root or essence of the response to your question; it is uncertainty. I think part of what makes a scientist a good scientist is, perversely perhaps, the acceptance of ignorance. And that is not that we accept that we will remain ignorant, we’re always working to become less ignorant but that we accept that we are ignorant, and we don’t have to cover that up. And yet the journalists often want to clean their story. A story in which we know things. A story in which things are more certain, as opposed to a story where we say well, we’ve got some initial results here and I’m not promising but we’re not sure it’s causation.
Bailer: You’re listening to Stats and Stories and today we’re talking with David Allison, Dean and provost professor at Indiana University in Bloomington. You know, I thought your point about acceptance of ignorance is a really important one and I think there’s this general willingness to be wrong that’s part of the scientific process. That you’re willing to reject what you’ve held to be true. I found it interesting in your article that you mention this idea of a white hat bias, which I had not seen before.
Campbell: Yeah that was one of my questions.
Bailer: That was one of your questions? I was peeking at your notes.
Campbell: I love that. So, can you take a little bit about how that’s really – what that is and why that’s a really bad thing to have happen?
Allison: So, human beings are human beings and we all have biases and there’s no way around it. Science itself, as Adam Smith once said, is the antidote to the poison of superstition and enthusiasm, or you might more broadly say of biases. Science is a bias reduction process, but we need to use it to reduce bias. So, we rely on things like blinding and randomization and so on. Often there’s a thought that only certain kinds of biases prevail but in fact everybody has biases, and one of the most potent ones is what we call white hat bias, and that makes a reference to the good guys in the old cowboy movies wearing the white hats, and that if you wear the white hat you think you’re righteous, and if you’re pursuing righteous ends then perhaps it’s okay to bend the truth in the pursuit of righteous ends. As a scientist I don’t think it’s ever okay to bend the truth. But we see that this seems to be occurring in certain areas and childhood obesity is one where it seems especially strong, where people are passionate, they have good intentions, they want to protect children as we all do and should, they want to improve the public health as we all do and should, and they believe they know the right answers for that and they are willing to push those right answers even if it means bending the truth sometimes, and that’s not okay.
Bailer: So, what are the costs of this? I mean there- so talk a little bit about when you have these claims of exaggerated effectiveness even if it’s based on someone who in their own minds has a good heart, what’s the damage ultimately that could accrue as a result?
Allison: The costs can be pursuing treatments that are not only not effective, but that are harmful that’s rare, but possible. For example, Dr. Gary Foster and his team published a paper about a year ago showing that free breakfast provision in schools seemed to lead to more obesity in children. And you only know that if you do the study, and he did a rigorous randomized controlled trial and found that out. But that’s uncommon. What’s more common is that we will spend resources and efforts on things that are ineffective. Examples would be promotion of increased fruit and vegetable consumption for children, or adults for that matter, as a weight loss strategy. That alone is not an effective weight loss strategy, and yet it’s still promoted even though there’s evidence that it is not effective. What that does is it leads individuals to not try other things because they are trying that, it leads researchers to continue to try to invest money in those things when they should be investing and investigating others, it leads school districts and funders and public health organizations to be finding ineffective things, it leads us not to be focused on effective treatments we have, which for example, in the case of adults include things like- that are much less feel good like bariatric surgery and pharmaceuticals and cognitive behavioral treatment in a treatment center.
Campbell: One of the things that was interesting in reading your background is how you negotiate this sort of getting research finding from food industry, beverage industry… scientists in general have to negotiate this if you’re an academic and working at a university, and then you sort of open yourself up to charges that you’re doing science that’s going to promote a particular agenda that the food industry might have. Over your years as a researcher and scientist how have you negotiated this? And is this something that journalists could do a better job of covering this area where you’re doing both industry research and non-industry type research?
Allison: Sure. I think journalists could do an enormously better job of it. Journalists, like many others, cater to their audiences to a large extent. And the audiences, again, they’re us- humans, we have our biases. We love stories. We love stories with human elements, we love stories with good guys and bad guys, simple black and white distinctions as opposed to complexities, nuance, no one is wrong, no one is bad guy or good guy. But the science is what matters. The journalists unfortunately often look to make it personal, and they engage in that attacks and talk about characteristics of the researcher including funding source and that just adds a theatrical to the science. We need to say is the science that matters, as I have said repeatedly in my writings and speaking, there are three things that matter in science: the data, the methods in which the data are collected which give them their probative value, and the logic connecting the data and methods to conclusions. That is the stuff of science and everything else is a distraction.
Bailer: Let me change gears just a little bit here. David you recently contributed to a National academy’s report on reproducibility and replicability in science. Can you talk a little bit about what is reproducibility and what is replicability? And if you’re feeling really ambitious you could even talk about generalizability, and why are these important ideas for us to think about in science?
Allison: I would be delighted. The first thing we do is define those terms and that was what our committee did, because those have been used sometimes interchangeably, including by me and others. Sometimes without clear definition. The academy then, our committee, defined reproducibility as taking the exact same electronic data, running it through the exact same code as originally used and getting the exact same results. So, this is essentially computational reproducibility. It’s really just showing that you can get what the original investigators get. It doesn’t say it was right, doesn’t say the data were any good. It just says you can find the data, reproduce their calculations, reproduce their answer. Replicability is collecting new data. It’s running a new study that essentially answers the same question as the original study and then asking do you get roughly the same answer. This gets much more complex because what roughly the same means comes to the floor. And that is both what does roughly the same study or roughly the same question, and what does roughly the same answer mean. And that’s difficult to define and the source of much debate in dialogue. As roughly the same study begins to branch out to be broader and broader we move from replicability to generalizability. So, for example if I do a weight loss study of the general public in Bloomington Indiana assigned to diet A vs. diet B, and then you do the same study in Minnesota, one might argue that that is a replication. Alternatively, if I do that study in all men and you do that study in all women, is that a replication or are you testing generalizability? It’s a matter of judgment at some point, the important thing is if we define precisely what we’ve done then people can know and make those judgments.
Bailer: This is like – this is the way science works. I mean that suspicion or over emphasis of a single study is often what gets us in trouble. I think that a lot of time when you see about an overselling of a certain study and the outrageous engaging headline, it’s often that one study and not the preponderance of evidence from multiple studies that you’re seeing.
Allison: I think that’s exactly right and that’s the way that nutrition and obesity studies are often portrayed in the media. If you think about reading a new study about some advance in the space shuttle, you probably wouldn’t immediately say where do I get my ticket to be on the next shuttle? You might read and say this is interesting as a member of society at large; I’m interested in the science, but it doesn’t affect me today. And yet people read the next study of broccoli or the next diet and go how does this mean I should change my food intake today?
Bailer: So, if you were going to advise someone interested in working on these childhood obesity or research in this area what are some of the things that students should do to get ready to work in this space?
Allison: I would say that the first thing that I would hope every student would do is to take a long hard look in the mirror, and look themselves in the eye, and say am I in this to actually collect data to learn something? -as opposed to just advance an idea that I’ve already decided is true. And if they say that they’re in it to learn something, to really test ideas, not just to promote ideas, then I think make an unwavering commitment to doing that. To registering your ideas in advance when possible, to collecting data as rigorously as possible and then to the most unvarnished scrupulous commitment to truth in reporting possible.
Campbell: We often talk about one of the things that science does for us is we sort of build on a body of knowledge and we end up knowing more than we used to know. In your study of obesity, how have the studies- have they improved? What do we know now that maybe we didn’t know ten years ago? How do you feel about that and in terms of your own work as well?
Allison: That’s interesting, someone paid me a nice compliment after I gave a talk once and said the thing I like about David Allison’s talk is that I always know less when he’s done talking. [Laughter]
So I think that in the field of childhood obesity we do know a lot, but it may not be what we wish we would know. We often know a lot about what doesn’t work. So, we know, for example that ordinary physical education classes in the schools are not effective in the rates of childhood obesity. That’s not a happy story. People don’t want to believe it in many cases, but we now know that. We know within a reasonable degree of certainty that regular breastfeeding as opposed to not breastfeeding is not an effective way to long term prevent obesity. Breastfeeding is a good thing, people who can do it should do it, but it just doesn’t seem to prevent obesity as much as we would wish it did. On the other hand, we know certain things can be helpful and effective. So, exercise can be effective for health even if it doesn’t always produce weight loss. We know that exercise can produce weight loss but- or prevent weight gain- but only if done in sufficient quantities, and so that adherence to large amounts is important. These are a few of the many things we’ve learned over the years. They’re not always as exciting as we wish they were in terms of easy public health fixes, but we have learned some things.
Campbell: So, John and I want to know how many donuts we should eat each week. Should we cut down our donut intake? Is that a good thing?
Allison: Good for what? Like I said earlier.
Bailer: I’m going to work on that donut trail t-shirt this year, come on Richard.
Allison: Dying to be a sumo wrestler.
[Laughter]
Bailer: Well David that’s all the time we have for this episode of Stats and Stories, thank you so much for being here.
Allison: Thank you, truly my pleasure.
Bailer: Stats and Stories is a partnership between Miami University’s Departments of Statistics and Media, Journalism and Film and the American Statistical Association. You can follow us on Twitter, Apple podcasts, or other places you can find podcasts. If you’d like to share your thoughts on the program send your emails to statsandstories@miamioh.edu or check us out at statsandstories.net. Be sure to listen for future editions of Stats and Stories, where we explore the statistics behind the stories and the stories