Robert Santos is vice president & chief methodologist at the Urban Institute as well as President-Elect of American Statistical Association. He has over 40 years of experience designing research and evaluation studies as well as sample surveys. His expertise includes quantitative and qualitative research design, sampling, survey operations, and statistical analysis; specialty areas include Hispanics, blacks, undocumented immigrants, and other disadvantaged populations.
+ Full Transcript
Rosemary Pennington: For the last several years the US Census Bureau has been gearing up for its count of the American population. Researchers have been working out sampling strategies as well as what types of questions Americans will be asked. One of the concerns going into the 2020 Census is the possibility of miscounts, which could impact everything from representation in Congress, to federal funding for a variety of projects. Assessing the miscount risk is the focus of this episode of Stats and Stories, where we explore the statistics behind the stories, and the stories behind the statistics. I’m Rosemary Pennington. Stats and Stories is a production of Miami University’s Departments of Statistics and Media, Journalism and Film, and the American Statistical Association. Joining me in the studio is regular panelist John Bailer, Chair of Miami’s Statistics Department. Media, Journalism and Film’s Richard Campbell is away. Our guest today is Rob Santos, the Vice President and Chief Methodologist at the Urban Institute, a nonprofit research organization focused on issues of public policy. Recently released is a report assessing possible miscounts in the 2020 Census, and Santos is one of the report’s authors. Thank you so much for being here today Rob.
Rob Santos: Thank you for the opportunity.
Pennington: What do you mean when you say you’re assessing the miscounts for a count that hasn’t happened yet?
Santos: Okay, what’s meant by a miscount is an inaccuracy. There is a true count of all people in the US and all people by state, and there is what the Census Bureau gathers. There have always been miscounts in the census, and what they strive for is minimizing the size of the miscounts. Unfortunately, we believe that based on the 2010 performance and what we expect in 2020, is that there will be a noticeable miscount for 2020, which means that the actual number of people residing in the US will be so different than what actually gets counted, that there will be a difference that has implications to why it’s important. It’s important because congressional house seats get allocated to states based on census counts and if there is a miscount and inaccuracy, then some states will get less than they should, and some states will get more than they should. That also extends to federal funding. There are billions of dollars that get allocated, to the extent that there are inaccuracies some states will get more than they deserve, and some states will get less than they deserve.
Bailer: That’s very helpful thank you, now if miscounts is this is it always an undercount?
Santos: No actually it isn’t, there can be overcounts, and in fact the 2010 census is a great example of overall presumed accuracy or estimated accuracy, that masked an unfair census. And let me explain how that happens real quickly. In 2010 there was an overcount of individuals who were white, non-Hispanic and elderly. And there were undercounts of minority populations like Latinos and African Americans. There were also undercounts of young children 0-5 years old. And the net when you put the overcounts and undercounts together ended up with overall accuracy, but that had some real implications when it came time for federal funding.
Pennington: So, you mentioned earlier that to judge the miscounts for the 2020 census that your group ran some scenarios, can you talk about what some of those were?
Santos: Certainly. We started with what was supposed to be the most accurate census of all time, the 2010, and we took the results of the coverage performance that the census bureau did after the decennial census, and took those results that showed where there were over and under counts, and applied those to the 2020 census. To a 2020 population projection to get the results, so they had simulated a 2020 census. That was a scenario we called the low-risk scenario. We also had a medium-risk scenario where we took the operating plan to the census bureau, they projected well, we think we’ll get so many forms back, we’ll go knock on doors, and we’ll do this and that, and we used that on a projected 2020 population and that was our medium-risk scenario. Then we did a high-risk scenario where the responses by households and people were lower than what the Census Bureau projected. We also included the impact of the citizenship question, regardless of whether or not it was included. And that was the high-risk.
Bailer: So, what’s different about the 2020 census from the previous censuses that have been done?
Santos: The 2020 Census will be a bellwether for the future. It turns out that even if we simulate those, the magnificent performance of 2010 decennial census counts onto a projected 2020 population, we would get an undercount of the population. And the reason that happens is because our wonderful US population is diversifying; we are a more diverse nation. We have higher rates of individual mixed races, of Latinos, of African Americans, than there has ever been in past censuses. And because those also happen to be populations that are harder to count, that ends up triggering the undercount that we expect for 2020. And it gets worse when you start superimposing more realistic assumptions about what could happen.
Pennington: Why are the populations harder to count? Is it an issue of trust with the instrument itself or a census worker not being as well versed in those communities? Could you talk about that?
Santos: Certainly. It is a variety of reasons, for instance African American and Latinos tend to be more mobile, and it’s easier to count someone who owns a home and is stable and has a track record for where they’ve been and where they are going and are more stable than renters. So, renters historically are harder to count. Latinos are more of an immigrant population. There are higher concentrations of immigrants, and moreover among citizens, Latinos there are also non-citizens in those mixed immigration households. Historically those have been harder to follow as well, or to get to participate as well.
Bailer: So how has the report been received?
Santos: There has been an enormous amount of media coverage and we’re happy- not because we’re saying that the sky is falling and it’s going to include a miscount, but we wanted to use these different scenarios that all show the same stories, the same ending, so to speak, which is a miscount, we wanted to use those to motivate a grass roots effort to get out the count. We believe that simply having public service announcements and the usual complete cap committee activities, and having mayors and governors speak out on the importance of the census is essentially talking down to people, not necessarily a bad way but having an official to talk to individuals is not necessarily the way to break through the historical resistance for the hard to count to public relations. You really need a peer to peer discussions and encouragement, and that’s what we think is needed, and we use these scenarios to help motivate them.
Pennington: You’re listening to Stats and Stories and today we are talking with Rob Santos vice president of the Urban Institute. You mentioned a moment ago, Rob, that there’s been media coverage on your report, and I wondered what your thoughts were on how journalists can communicate this issue of miscounting without raising alarm? Because the issue with coverage like this is that you suggest to people that this is a problematic thing and maybe not trustworthy, and that might suggest to other people that they maybe shouldn’t participate, so what advice would you give to journalists who are going to be covering this, and the possibility of talking about a miscount, and how they should think about covering the census more broadly?
Santos: Certainly. I overemphasize the opportunity loss of not participating. The message that we communicate is that you are a human being who resides in the US and deserves to be counted. And it doesn’t matter whether you’re white, black, mixed race, Native, Asian, what gender, what religion or immigrant status. You deserve to be counted and you deserve it because it helps you, it helps your family, it helps your community, it helps your state. And it helps in multiple ways. Not only in terms of the usual political representation, but it also helps in terms of getting funds; federal funds, state fund allocations are also tied often to decennial census counts, and decisions on where to put infrastructure, fire stations, schools, commercial buildings like Walmart’s and grocery stores. Those all rely on a complete and accurate count, and that is why that’s the message that we have. And we use the threats of potential miscounts and undercounts as a way of illustrating why it’s so important to recognize that you’re deserving of being counted.
Bailer: So even if the best-case scenario that you describe, the low-risk scenario leads to a significant miscount, specifically for certain groups, what’s the remedy?
Santos: The remedy is to do the best that you can. All censuses, even the 2010 census have issues. I’ll contend that the 2010 census, while was accurate, overall was unfair in that it undercounted one group and over counted another and that has really policy implications. We have to live with, that so what we’re trying to do is to minimize the risks associated with an inaccurate census. So, I say let’s do the best we can to get the most accurate and fair count. Which means counting all peoples equally, rather than overemphasizing one or the other.
Bailer: I’d like to change gears just for a bit from the census to the Urban Institute and hear a little bit about what things you study at the Urban Institute and what you’re involved with, and in particular are they’re examples of projects that you really would like to describe as insightful and impacting policy?
Santos: Certainly. I have been honored to be at the Urban Institute for about 16 years in two stints, the most recent about 13 or 14 years. In my position as Chief Methodologist, I get to play in everyone’s field. Everyone includes about a dozen different substantive research centers that cover everything from justice policy, to families, to immigration, health policy, education policy, etc. etc. and because of that I’ve been able to be P.I. on a variety of studies. We just talked about the 2020 census miscount, there’s also been projects that I’ve worked on that involve justice policy. Doing challenging things for the Bureau of Justice Statistics, like trying to design samples to estimate pretrial releases, or not of individuals, looking at [inaudible] there are projects that have to do with – I’ve actually done a fire fighter safety study, I’ve done transportation research as part of this. I’ve worked on a number of housing discrimination studies where we take a pair of testers and randomly scrape the web for sales and rental homes and homeless, either sales or rental, then we send out a pair of testers, one which is basically white and the other is minority, or one that is single and the other has a family, or one that doesn’t have a housing voucher and one does, whereas either one is basically financially capable and then measure discrepancies in how they’re treated. So, there have been some really important studies we’ve done to help look at the policy landscape. I’ve also been involved in immigration and refugee studies on my third annual survey for refugees for health and human services that we’re conducting. And ten years ago, I worked on the impact of the I.C.E. raids on the communities to show how not only were immigrant communities devastated, but whole towns almost collapsed because of I.C.E. raids on large manufacturing firms.
Pennington: How do you do research like that? Because to survey refugees and work with immigrants, some, who I imagine are not in the country legally, how do you connect with populations that may be vulnerable and distrustful of someone coming in to do research, concerned that maybe you’re working for I.C.E. or work for the government? How do you navigate that space to do research that will help these communities?
Santos: The beauty of policy research is that it is a quest for knowledge and insight on specific policy research issues. And we attack those in different ways depending on the specific question. There are many, and most research questions we have at urban institute are ones we deal with quantitatively and involves statistics and statistical inferences. Other questions that are equally important involve issues of exploring why something is happening, what the type of issues are, without necessarily quantifying them in a statistical way. As an example, where we use qualitative research approaches, so the impact of the I.C.E. raids on communities was actually a qualitative research study, where we drew off sites around the country where raids had occurred, and talked to families and the employers and the community folks, people in grocery stores, etc., whereas things like the annual survey of refugees for H.H.S., we get a list of all refugees that were admitted over the last five years and draw a probability random sample and conduct a survey in over 17 languages. And we have done that for three years now. It is incredibly important. It shows how refugees are integrating into US society and pursuing the American dream.
Bailer: Can you talk a little bit about the distinction between quantitative and qualitative research and study design? I know that’s a gigantic question to talk about, but a lot of people that might be listening are very familiar with the quantitative ideas, but maybe not so much with the qualitative. So, a simple illustration would be very much appreciated.
Santos: So, qualitative research gets to issues of why and how, whereas quantitative is more of how much to quantify things. In terms of qualitative research, an example would be the I.C.E. raids study, where we wanted to know how the I.C.E. raid was impacting the communities. So we’d go into the community and find individuals who were undocumented; undocumented families, and ask them after the I.C.E. raid, what happened to them, what did they do? And we find out things like there were all the lights were turned out for a week, and they closed all the doors and wouldn’t answer the doors and simply stayed inside because of fear of someone coming and taking their children. So, it helps provide some textualization, so to speak, around the types of things that are happening to people and the types of impacts. So, we were able to pull from those types of communications that some people were in such fear that they simply left town with almost all their belongings behind. Some people hid in fields for three days. Some people turned out the lights and didn’t answer the door for a week. Grocery stores basically had to shut down because nobody was buying groceries in immigrant neighborhoods. Those are the types of things that don’t necessarily warrant, and you don’t need a statistical point estimate for. You need to know that it’s happening, so you can essentially put together a conceptual framework about how policy decisions like an I.C.E. raid impacts the different communities, and in what ways it impacts. That can then help stage a later study that’s quantitative that says how much of an impact occurs, and then you can go and randomly sample people and ask them specific questions that were motivated by what you found out in your qualitative study, and then put statistical estimates in. 30% of people moved out of the city entirely, that sort of thing.
Bailer: Thank you. I liked in some of the writings that I read of yours that you talked about actionable insights as something that was really important for the work that you do and the work that you have done. Is there a particular actionable insight that led to a change that you would be really excited to talk about?
Santos: Actually, I was an AAPOR president and I was able to give an address-
Bailer: You need to talk about what AAPOR is.
Santos: Yeah, AAPOR is American Association for Public Opinion Research. It’s basically the collection of pollsters, academic survey researchers and government survey researchers, commercial as well, in the country, and there’s a good international representation as well. And I was President in 2014 and my address focused on the need for the survey research and polling world to transcend the classical survey research and point estimate methodologies, and recognize that we need a quest for knowledge, and actionable insight. And actionable insight- the best example of that I had was that there was an instance where we found out that there was in the I.C.E. raids qualitative study, that there was a person, a mom, who was separated as a matter of policy, who was separated with her child who is breast feeding, and that went on or three weeks. And it was absolutely horrific for that person and for that family. And the fact that it was an N=1 observation, and the fact that it was policy, was an actionable insight that then led to a policy by I.C.E. saying we will not separate families from their children. Now it wasn’t just because of our study there were other instances being reported in the media by that time, but still you can have an actionable insight without necessarily it being a statistical insight. And vice versa, you can have statistical insights that could never be made without qualitative studies that are really important, like clinical trials in the pharma world.
Pennington Well Rob that’s all the time we have for this episode of stats and stories thanks so much for being here.
Santos: Oh, it was a delight thank you.
Bailer Thanks Rob.
Pennington: Stats and Stories is a partnership between Miami University’s Departments of Statistics, and Media, Journalism and Film, as well as the American Statistical Association. You can follow us on Twitter or Apple podcasts or other places you can hear podcasts. If you’d like to share your thoughts on the program email us at statsandstories@miamioh.edu or check us out at statsandstories.net, and be sure to listen or future editions of Stats and Stories, where we discuss the statistics behind the stories and the stories behind the statistics.