Constance (Connie) Citro is a senior scholar with the Committee on National Statistics and an independent consultant in which capacity she worked on the project that produced A Nation’s Data at Risk. She was previously CNSTAT director from 2004-2017 and senior study director from 1986-2003. Citro was an American Statistical Association/National Science Foundation/Census Bureau research fellow and is a fellow of the American Statistical Association and an elected member of the International Statistical Institute. She served as president of the Association of Public Data Users and its representative to the Council of Professional Associations on Federal Statistics, edited the Window on Washington column for Chance magazine, and served on the Advisory Committee of the Journal of Survey Statistics and Methodology. In 2018, the American Statistical Association established the Links Lecture Award in honor of Citro, Robert Groves, and Fritz Scheuren. She will give the 32nd Morris Hansen Lecture in September 2024.
Jonathan Auerbach is an assistant professor in the Department of Statistics at George Mason University. His research covers a wide range of topics at the intersection of statistics and public policy, including urban analytics, open data, and official statistics. His methodological interests include the analysis of longitudinal data, particularly for data science and causal inference. He is the current president of the Washington Statistical Society and the former science policy fellow at the American Statistical Association
Episode Description
The democratic engine of the United States relies on accurate and reliable data to function. A year-long study of the 13 federal agencies involved in U-S data collection – including the Census Bureau, Bureau of Labor Statistics, and the National Center for Education Statistics – suggests that the nation’s statistics are at risk. The study was produced by the American Statistical Association in partnership with George Mason University and supported by the Sloan Foundation and is the focus of this episode of Stats+Stories
READ THE FULL REPORT
+Full Transcript
Rosemary Pennington
The democratic engine of the United States relies on accurate and reliable data to function. A year-long study of the 13 federal agencies involved in US data collection, including the Census Bureau, the Bureau of Labor Statistics and the National Center for Education Statistics, suggests that the nation's statistics are at risk. The study was produced by the American Statistical Association in partnership with George Mason University and supported by the Sloan Foundation. It's also the focus of this episode of Stats and Stories, where we explore the statistics behind the stories and the stories behind the statistics. I'm Rosemary Pennington. Stats and Stories is a production of Miami University's departments of statistics and media, journalism and film, as well as the American Statistical Association. Joining me is regular panelist John Bailer, emeritus professor of statistics at Miami University. We have two guests joining us on the show today. The first is Connie Citro. She's a senior scholar with the committee on national statistics. She was previously Committee on National Statistics director from 2004 to 2017 and senior study director from 1986 to 2003. Citro was an American Statistical Association National Science Foundation and Census Bureau Research Fellow, and is a fellow of the American Statistical Association and an elected member of the International Statistical Institute. Jonathan Auerbach is an assistant professor in the Department of Statistics at George Mason University. His research covers a wide array of topics at the intersection of statistics and public policy, including urban analytics, open data and official statistics. His methodological interests include the analysis of longitudinal data, particularly for data science and causal inference. He's the current president of the Washington Statistical Society and the former Science Policy Fellow at the American Statistical Association. They both also worked on the study that produced this report about the risk of two federal statistics called a nation's data at risk. So thank you both for being here today.
Connie Citro
Thank you for having us.
John Bailer
Yeah, and congratulations on producing this work. I mean, this is a really interesting and fascinating piece. There was one quote in there that really struck home for me as I was reading this. And it was, federal statistics are a public good, just like national defense and the national park system. And, you know, I bet people can picture monies going to national defense in their heads or to the park system, but maybe not so much about national statistical agencies. So I guess I'd like to start with a question. Can you give an example of a national statistical agency and one of the products they produce that people might recognize and value?
Connie Citro
Well, I'll give an example that is the most basic. It's in the Constitution, which says there's to be a census of the population every 10 years, so that seats in the US House of Representatives can be peaceably reallocated as the population moves and different areas grow more than other areas. The US Census Bureau is the agency that takes that census, and the data also feed into annual population estimates for states and counties and local governments, and those are used for all kinds of purposes, including allocating federal funds to programs like Title One, education funds, highway funds, you name it, and for localities to figure out where they're going to put the next school.
Jonathan Auerbach
Other examples are the inflation rate, the poverty rate, the unemployment rate. I think listeners will be, once they think about it, they'll realize that they're familiar with a long list of federal statistics.
Rosemary Pennington
I was struck by something else, and I can't remember if it was the summary or the highlights, but in one of those, there's a phrase where you are comparing federal statistics to infrastructure, which really struck me. So, could you explain sort of why or how federal statistics sort of serve as infrastructure for the country?
Connie Citro
Yes, the analogy came up. Actually, Jonathan and Steve Pearson, the ASA Science Policy Director, came up with that a number of years ago. Every few years, the civil engineers raid our bridges and highways and railroads and so on, and, you know, give us c plus or b plus or d minus or whatever. And so they thought that federal statistics should have a similar effect. Or evaluation. We're hoping to make this actually an annual evaluation, and the infrastructure analogy is again, the Federal statistics support our democracy, our economy, our society in so many ways, including there's actually a sector of the business world that takes federal data and adds value to them. The whole polling industry depends on federal statistics. Polls these days get minimal responses. People don't answer surveys, so they need the federal statistics to benchmark the responses they do get to make sure they are representative. And those are just a couple of examples of how federal statistics permeate our society, economy and democracy.
Jonathan Auerbach
I just wanted to expand on something Connie said. We saw in the beginning of the Biden administration that it would build it back better. How effectively the work of the civil engineers was used to advocate towards funding physical infrastructure. And I think the hope is that as a statistics community, we can be better at advocating for our statistical infrastructure. And that's the role of the American Statistical Association as being one of the professional associations under which professional statisticians unite.
John Bailer
So some of the people listening to this episode may not know how many statistical agencies there are in the US, and kind of what this, this span in scope of their responsibilities might include, can you give us a quick rundown?
Connie Citro
Sure. There are, right now, 13 so named principal federal statistical agencies, the Census Bureau, the Bureau of Labor Statistics, they are in education, health, criminal justice, agriculture, transportation, Social Security, science and engineering, and I've undoubtedly left out some, they are agencies whose purpose is to produce statistics in a particular area. There are also about 100 or more statistical programs throughout the US government. The National Institutes of Health, for example, do a lot of surveys themselves and various other statistical activities. And all this is coordinated by a chief statistician's office, which is in the US Office of Management and Budget.
Rosemary Pennington
This is a massive infrastructure. We can continue using that analogy. How did you, and we know that you have found in your research that it's at risk. What did you do to figure that out? Like, what did the study entail?
Jonathan Auerbach
I think what's important to note is that the study builds on a long tradition at the American Statistical Association of evaluating or helping shape the Federal Statistical system. In fact, the American Statistical Association, in its inception, followed controversies over the 1840 census, every major change to the system as the federal government expanded during the Roosevelt administration, there were reports being done by the American Statistical Association. During reorganizations or challenges during the Nixon administration, the Carter administration. There were large reports. So this report doesn't come out of thin air. It builds on this legacy. There are several differences, which we can talk about, but we already had a very good starting point, and then a big part of the research was engaging the community of federal statistics stakeholders, so we held a very big workshop. We had a large number of meetings with different stakeholders, and that all culminates into a lot of the findings and recommendations you'll find in the report.
Connie Citro
So did a pretty big survey of the principal agencies that asked them a lot of questions, everything from staffing to innovation to do they have problems hiring people, you name it, and they really came through. It was a heavy lift for them, and we're very appreciative of that.
John Bailer
So did you get a good response rate?
Jonathan Auerbach
Yeah. So the agencies did comment on the response burden of the questions we sent them, I would argue, so that, you know, once you get your PhD, you get the survey for a doctorate, that's a massive response burden. So I'd say we're even.
Rosemary Pennington
So what are the big challenges that these agencies are facing?
Connie Citro
Well, we catalog those, and we have a shorter, pithier version in the executive summary, they are facing what I would call internal challenges, internal to the federal government and extra. Internal challenges, we actually looked at how each agency stacked up in terms of the support it was getting, in terms of budget and staff and in terms of support from its parent agency. These are all lodged in different Cabinet departments or in NSF and Social Security. What kind of support are the agencies getting from the higher ups? And then, what kind of protections do they have for what we call professional autonomy, which means their ability to decide on the best method to collect, analyze, disseminate statistics without having political people tell them, Oh, you can only knock on doors twice, or you've got to always use the internet or something like that. And we found every agency had weaknesses in at least one of these three supports, and some agencies are weak in two or three. So that was one indicator. On the external side, people do not answer surveys. Yeah, when I was young, it was a big deal to get a telephone call asking for, you know, your opinion on a brand of frozen chicken, ala King or something. It was just exciting. Now, you know, people just, can you please stop on my web page and take a survey? And no, people don't do that, so that is a real problem. And generally, the statistical agencies do very well in getting response, but there has been a falling off that accelerated in covid, and so they do not have the RE and their resources have been declining in real terms, in most agencies. And so they've got a real problem.
John Bailer
There's just so much, so many moving parts here to think about, about this and and, you know, I wonder if, if people you know, does the general public appreciate, why is it important that these agencies have this degree of professional autonomy? And I like the fact that you made a distinction between professional autonomy and autonomy, that there is some accountability to the agencies that to the organizations that fund them.
Connie Citro
Yes, we're in a democratic society, and the Congress and the administration and stakeholders generally, you know, have a right to say we need more data on x, or we need in fact, historically, that's how these agencies were created a topic like criminal justice or education rose to federal level, not just the states. And invariably, a statistical agency was set up whenever there was a new cabinet department that was set up, but professional autonomy is not being told, as the Argentine statistical people were, when Argentina had an out of control inflation situation, they were, in effect, told either you're going to change the numbers or you're fired, and that is not what you want. No matter what your political persuasion is, you want a level playing field where the statistics are as high quality and objective as they can be, and that's what professional autonomy is all about.
Jonathan Auerbach
To answer your question more, does the public have an appreciation? I think the public does not, as we know, there's a lot of subjective decisions that go behind different data collection methods, and you want those decisions to be made by professional staff who are staying at the agencies that are seasoned and don't have political interests. And it's easy, you know, plausible deniability, oh, you know, we think that you shouldn't use sampling in determining the Census count. We think you shouldn't when those have an underlying political incentive to make those statistical decisions.
Rosemary Pennington
You're listening to Stats and Stories, and we're talking about the risk to federal statistics with Connie Citro and Jonathan Auerbach.
John Bailer
So, you have, you know, I was, I was struck by looking at this, that you talked about, how do you measure autonomy of function? And you had maybe six different dimensions on which you would think about kind of evaluating how well how autonomous the functioning is of these organizations. Can you talk about some of those different dimensions for us?
Connie Citro
Yes. And in fact, there was a paper published in statistics and public policy that Jonathan and I and another group of folks put together. We looked at legislation, we looked at regulations, we looked at statistical policy directives from the Office Chief statistician's office and. And we found that three or four of the 13 agencies are not even authorized in legislation. They were set up by the executive branch, and so they are at real risk if someone wanted to do away with them or undercut them because they're not actually established in law, then the aspects of autonomy is essentially professional control over things like hiring somebody high up in commerce doesn't tell the Census Bureau who to hire to run their research and methodology. Say that's one aspect, control enough control over IT systems so that they can protect confidentiality and guard against anybody fiddling with the numbers in the computer. Those are two aspects. There are several others that we identify, but it's that kind of thing. And again, we looked at legislation regulations. Our survey asked agencies about several of these components. And again, it varies more than you would expect. We are relying on norms right now to protect a lot of autonomy, and we really believe in this day and age that we need legislative protections.
Jonathan Auerbach
To go back to your earlier question about, you know, some of these issues are so complicated, how do you know? How do we know where to look? There's been a series of events that have happened over the past 50 years in which certain parties attempted to manipulate or have a desired outcome, and that kind of guided us, for example, in a previous administration, the presidential administration tried to plant in career positions, political appointees. And so that's one of the reasons why we looked at how easy it is to accomplish things like that.
Rosemary Pennington
I'm glad you raised that question, because as I was reading the materials around your report, I was thinking about, we've had Andreas georgieu on the show, who is a Greek statistician, who is still, I think, caught up in in legal cases in Greece around sort of the data he was producing, and the fact that the government didn't like it, right? And I wonder, as you're talking about the challenges the US faces, how do the risks to federal agencies, federal statistical agencies, in the US sort of compare to other places? I think a lot of Americans, I think Jonathan might have said it earlier, don't think about these agencies, right, which means they probably are also not thinking about the kind of that they could be interfered with in the ways that I think this report hints to at times. And I wonder if you could, just for our listeners who are not as, not as knowledgeable about this sort of, how do those challenges that we're facing in the United States sort of compare to other countries, or like Argentina, as you mentioned earlier, too.
Connie Citro
Well, we have been, I think, fortunate so far in fending off of some pretty intense challenges. Statistical agencies will tell you that it doesn't matter the administration or the party. They will always get questions from political appointees. Well, couldn't you hold up release of this report till after the election? Or couldn't you release it right now? And you don't hear about those, because the agencies explain that this is not appropriate, and there are standards for release and so on. But there have been occasions in the US that have come pretty close, cut pretty close to the bone, and you mentioned having had a program about the moving the Economic Research Service in USDA to Kansas City, and that was done to undercut that agency whose agenda the that administration not care for the adding of a question on citizenship to the census. The movement for that was not upheld by the courts. But that's another example, and that is actually quite an interesting one, because many countries ask about citizenship. And many surveys in the US, federal surveys include a citizenship question, but in the US, with the fraught nature of how people feel about immigration, there was a real danger that if that was on the census, there was going to be a big undercount of the immigrant population, and in fact, there was a bigger undercount of that population than in the 2010 census. But so far, we've managed to hold off things like what happened in Greece or Argentina. But we feel with heightened polarization tensions, that legislation would be very helpful in this regard.
Jonathan Auerbach
May also just point out that you're not just protecting the accuracy of the number which is important, or these numbers that are being produced, but we all have to agree that this is the number and what Connie alluded to is that these attacks don't have to actually be successful to undermine trust. You go around saying, Hey, we're going to, you know, use this information on respondents in order to identify undocumented, you know, residents, even if you can't do that, and the Supreme Court says you can't do that, that may be enough to actually undermine the statistic or its use.
John Bailer
You know, as you were describing this, you know, Connie, you said, if I get the quote right, relying on norms to protect autonomy, and that just scared the heck out of me. Yeah, I mean, I'm not sure I'd rely on norms for anything right now, you know, and as you were describing that, I developed a stronger appreciation for some of the actions that you had embedded in your report. And you know that you had, you grouped them into four categories, but I was struck by the fact that there were five of them that were targeting Congress, and the first bunch of them were tied to legislation. You know, it's like act on this act. You need to make sure that this is put in the law, to make sure that this remains healthy into the future. So I'm curious. I appreciate what you've done, but what's been the reaction to this report by those in Congress, or staffers in Congress that may have been reading and giving you feedback.
Connie Citro
Well, since we released the report just about a month ago, we have done a lot of interviews with the business press in particular, which really grasped, you know, threats to things like the unemployment rate, and we have had follow up interviews with agency heads and their bosses. This has not been a good time to reach Congress with recess, with the political, you know, the Democratic tickets sort of changing overnight, you know, this sort of thing. But I know Steve Pearson's agenda is reaching out to all, both parties, both House and Senate, to get that and start that dialogue. So that's the answer right now, we're again, right now is not, is not the time when you're going to get attention. But the business press in particular, I think, has been right on this and and really, again, quite concerned.
Jonathan Auerbach
I'll just like to point out the goal is long term monitoring. Yes, this is totally the first of regular reporting. And this first report did come out, I think, between well publicized debate performance, an assassination attempt and a new presidential candidate. So it was a, it was a busy, you know, two weeks, yeah.
John Bailer
Right. And a report on official statistics wouldn't pop to the same level.
Jonathan Auerbach
Yeah, what's going on? Go figure.
Connie Citro
But I will give you an example, the Committee in the House of Representatives. I believe it's just in committee, but they were going to attach wording to the Census Bureau's appropriations to say, hey, in the next census and in the big American Community Survey that everybody relies on, you know, you cannot go and knock on doors, or the equivalent, more than twice. And there's some ambiguity in the language. And I just read today that the Census project, which supports the Census Bureau, got a large number of business organizations to go on record saying this would be a terrible idea. We need the data and limiting the number of callbacks or reach outs that the Census Bureau can do to get responses would really undercut the quality of the data. Was an impressive outpouring from the business associations.
John Bailer
You know, one of the things I liked, there were a lot of really clear statements about kind of, what are the reasons these agencies exist. And one part was to say, produce relevant, accurate, timely, detailed and credible data for public and policy makers. And I just really, I love that framing, and that's such a critical element. So you, in addition to the 15 recommendations, you had nine findings and some of the findings, i. You know, it was not a surprise to me, like the idea of resource efficiency. I mean, I suspect there's every organization that you would sample right now would might, might respond that there's deficits and resources for operation, but I think that it was also telling to me to see things like the fact that the office of the chief statistician, going from 40 to 12 people over the course of decades, which I think was quite strong. You had other findings about autonomy. We've talked about the idea of the importance of parent agency support was mentioned. You know, some of the others were issues about data quality. Can you talk a little bit more, maybe, about that set of findings or others that you'd like to bring to the floor for our conversation?
Connie Citro
Yes, but just a minute on resources. Yes, I've said exactly what you said in interviews. Every organization doesn't have enough money, right, but we document that while the rest of the domestic non defense, you know, discretionary budgets been going up in real terms, lot of these agencies have been going down, and some of them quite steeply in inflation terms, for instance, the Bureau of Labor Statistics and one consequence of tight resources is that long running surveys or data collection programs that support things like the unemployment rate, they get outmoded because the agencies don't have the resources to keep them up to date. When you have a survey that's the basis for a really important statistic. You just don't go from the old method to the new method cold turkey. You have to have resources to run parallel and to do a whole lot of testing, burst experimental and then at scale. And that takes serious money. Right now the Census Bureau, which does the data collection for the current population survey for the unemployment rate and other statistics, and the Bureau of Labor Statistics are working on an improvement agenda, like adding an online response option and some other things, but it is taking a long time, and there are other improvements that I think should be made, which, again, they just don't have the resources to do.
John Bailer
I wanted to ask, what's next?
Connie Citro
Great question, what we're trying to figure out, we're asking a lot of people, we're going to do a second report. Jonathan, maybe you elaborate on that?
Jonathan Auerbach
Yeah, I think, and I can't speak for the entire team, as Connie suggested, we're still thinking about it. I think we can identify two critical holes in the resources we had available in order to evaluate the state of the Federal Statistical system, and part of it was that we didn't have a lot of resources to understand the needs of data users. So we know the statistics being produced, but we don't necessarily know who's using it on the other end. So we can jump up and down about, hey, you know this data set. You know it's being cut, but if nobody's using it doesn't matter. Or maybe a small change could have a huge effect on the data user group. And then another thing, we didn't, I don't know, maybe, appreciate this. I didn't appreciate the extent to which it's a civics education problem just explaining to people why it matters. And I think we're looking at different ways of doing a better job of reaching the public. Just as, you know, being a citizen of this country or a resident in this country, you have rights and obligations. The same is true when it comes to the Federal Statistical system.
Rosemary Pennington
Well, that's all the time we have for this episode. I do want to point out to our listeners that you can find the report on ASA’s website, amstat.org. Thank you both so much for being here today. Stats and Stories is a partnership between Miami University’s Departments of Statistics, and Media, Journalism and Film, and the American Statistical Association. You can follow us on Twitter, Apple podcasts, or other places you can find podcasts. If you’d like to share your thoughts on the program send your email to statsandstories@miamioh.edu or check us out at statsandstories.net, and be sure to listen for future editions of Stats and Stories, where we discuss the statistics behind the stories and the stories behind the statistics.