Dennis Trewin is a pioneer of social statistics that are leading to meaningful measurement of social capital in Australia. He was the head of the Australian Bureau of Statistics between 2000 and 2007, and held other senior appointments in Australia such as Electoral Commissioner and an Adjunct Professor at Swinburne University. Dennis is also a member of the Committee charged with responsibility for producing an independent report on the State of the Environment.
+Full Transcript
Rosemary Pennington
In 2016, the Australian government launched a program and said would make tracking welfare benefits easier. Instead it falsely told hundreds of 1000s of Australians they owed the government money, with some of those individuals taking their own lives as a result. Australia's robodebt tragedy is the focus of this episode of stats and stories, where we explore the statistics behind the stories and the stories behind the statistics. I'm Rosemary Pennington. Stats and stories is a production of Miami University's departments of statistics and media, journalism and film, as well as the American Statistical Association. Joining me is regular panelist John Baylor emeritus professor of statistics at Miami University. Or we have two guests joining us on the shoulder show today. The first is Noah Cressy. And the second Dennis trewen. gresty is distinguished professor at the University of Wollongong, Australia, and director of its Center for Environmental informatics. He's also adjunct professor at the University of Missouri and affiliate at NASA's Jet Propulsion Jet Propulsion Laboratory. Cressy is the author and co author of four books, three of them on spatial and spatio temporal statistics, and have more than 300 peer reviewed publications. Chris has won a number of awards for his work, and is a fellow of the Australian Academy of Science of the Royal Society of New South Wales, and a number of other Learned Societies. Denis trewen is a pioneer of social statistics that are leading to meaningful measurement of social capital in Australia. I'm gonna start that over again, in three to Denis trewen is a pioneer of social statistics that are leading to meaningful measurement of social capital in Australia. He was the head of the Australian Bureau of Statistics between 2002 1007 and held other senior appointments in Australia, such as electoral Commissioner, adjunct professor at the University of Canberra as well. He was on the show in June of 2020, to talk about testing for COVID. They're two of the authors of a significance article about Australia's robodebt scandal. Thank you both for joining us today.
Dennis Trewin
Thank you. Good to be here. Thank
you. So
Rosemary Pennington
for those of us who were not following this when it was in the news, because I know it was big news when it was happening. What exactly was this scandal?
Dennis Trewin
Well, I'll take back question, Rosemary. And thanks for the opportunity to discuss with Joe, a very important case study on the misuse of data science. And before I start, I'd like to acknowledge EPU orphan who can't be here today. Getting back to your question, Robo debt was a government designed scheme to identify incorrect or fraud, fraudulent claims on welfare, or using artificial intelligence. So for over four years, the scheme used a machine learning algorithm to estimate debts owed to the government by those who supposedly submitted incorrect or fraudulent claims. It then sent the so called debtors lenders to pay up will face legal action. So guilt was assumed rather than innocence, which is normally the case, the media referred to the scheme was Robo Robo debt, that's where the name came from, not from the government. But the name caught on because it was a very appropriate shorthand description of what was actually happening. As it turned out, most of desperate false, in fact, about 80% of them were false. So it was a bit of a disaster. And it resulted in untold misery for hundreds of 1000s of Australians and the fish on par even said, some people says suicide as a result of the pressure they received from the government, because we were not talking about the normal citizens, we're talking about people who are really struggling and need to get into the welfare system.
John Bailer
So you know, sounds like a, you know, an 80% false positive rate would not be viewed as a very strong predictive model, you know, so I don't think any of us would find that passing any lab test, but But how did you? How did you get involved how to do to get involved in exploring and investigating this in more detail?
Noel Cressie
It says no, here, I'll start with that. And Dennis can add his own piece. I got interested, because of what of the injustice of the system that guilt was assumed. And then you had to prove your innocence. And I mean, it was just more of a social rung that I was interested in. But then it became clear that the there was an algorithm behind it. and it was something that statistical scientists and more broadly data scientists should know something about. And I took some time and a lot of reading of the popular press to realize that that algorithm actually wasn't being published, talked about something called income averaging, which basically meant that there was a check, there was a gold standard. And the gold standard was a yearly income that was reported to the taxation department, what you call the IRS in America, and but the welfare payments were made on a to weekly basis. And you can see that somehow, rather, you have to match to weekly income to yearly income. And it's just not a matter of multiplying by 26. In the gig economy, some people had no income during some of those two weeks, in which case, they got the maximum reimbursement from the welfare group called Centerlink. That's Australia's sort of welfare agency. On other occasions, they got nothing. And it just became clear to me that there was this income averaging was faulty. And I can say more about that later. But perhaps Dennis could share with us how he got interested.
Dennis Trewin
Yeah, well, no joined Nick and Fisher, a little while along our interest in this subject, Nick and I, since the COVID pandemic, have chatted over zoom on a regular basis, often over a glass of whiskey. And this topic rose up and it was clear, as Nigel said, its statistical methods were behind it, that there was complete lack of thinking the statistical methods were being used. And rather than why nots white particular, the there was, the use of machine learning was being undertaken by data scientists without any involvement of a person who really understood statistics. So that's what really got us in. So now, as I mentioned before, a incredibly high number of false positives, but they didn't know that until till after the event. In fact, you know, two measures of escape, they felt that it was more than successful, because they were finding out more what they thought were more fraudulent clients, what the is, we're actually in existence. So they think the screen was extremely successful. It didn't actually occur to him that having such a high number of so called fraudulent claims, like the a very strong hint that something was wrong.
Rosemary Pennington
Reading this. Sorry, go ahead. Well,
Noel Cressie
I was just going to add up to the Genesis. You know, after after what Dennis said, I want to add that I sort of pushed a bit further, I got into the technical area, and I felt that Jensen's inequality was somehow rather involved in these this high what looked like, you know, a high success rate. And, and so, but I couldn't find the algorithm. And it wasn't until there was a class action lawsuit that that algorithm was actually published in an appendix of Gordon legals class action lawsuit. And once I got the algorithm, it was clear that it was Jensen's inequality. And so I ended up writing an article about it in a popular science type outlet called the conversation. And that article, and various other things, sort of led Dennis and Nick and me into this. And we not only wrote something for significance, but we made a submission to a, a what's called a Royal Commission, which is like a congressional inquiry in the United States.
John Bailer
So so when you were talking about this, I, I had this question in mind about just how transparent the methods were, that they were using, you answered that in part by saying it required, and if some discovery document as part of court proceedings before you could finally see that documentation. You mentioned that that led you to thinking about this, this inequality result that that kind of gave you some insight about kind of what was happening. So so I'd like to challenge you to explain this to this general audience that we have about how Jensen's inequality helped help, you know, report on why they were this this income averaging that they were doing was was flawed.
Noel Cressie
So, Jensen's inequality basically says that, if you average something, some function, it's not the function of the average that there's an inequality. And if that function is convex, the index Well, I think goes one way, if it is concave the inequality goes the other way. If you look at the curve, and it's not really a curve, it's a series of straight lines. If you look at the curve, which reflects on the x axis, and the amount of money you earned, and on the y axis, the amount of compensation you received from Centrelink, the welfare agency, that curve is piecewise. Linear, as you can imagine, it's also decreasing. So the more money you earn, the less you know, compensation or welfare, you see. But there are, there are places, it's not a curve, it's a bunch of straight lines, but there are places where the curve turns downwards. And then there are places where the curve flattens. And in the places at the, at the top, where it turns downwards, that is a concave function. And at the places where it actually comes down towards almost zero and then goes to zero, that's a convex function. And so what happens is you've got this piecewise, convex concavity. So most of the people who are in very little should have actually received a robo credit, they sort of got a check in the mail. But if you're in the gig economy, and you were a certain amount of money, you are on the borderline of receiving nothing, they were the ones who got the robo debt. And, and you know, 500,000 people later, getting Robo debt, and zero getting credit, because I did a Freedom of Information. And I found that nobody ever was issued a credit. So you can see there was some dishonesty anyhow built into the algorithm.
Rosemary Pennington
This is hundreds of 1000s of people who are impacted by this. And I was struck in the article by the fact that it seems like there wasn't much pre testing of this algorithm before the government decided to roll this out. Can you talk a little bit about sort of that?
Dennis Trewin
Well, they did claim that they ran a pilot test, but obviously, it either didn't work or they ignored results, because as we've mentioned several times there is an incredibly high false positive rate. And they still proceed with the scheme. In fact, it wasn't stopped until there were successful legal challenges.
Rosemary Pennington
What would a robust pretest of something like this look like?
Noel Cressie
Well, I think somewhere in our submission to the Royal Commission, we talked about doing a digital twin, which would involve, in a sense, taking test cases. But you know, letting the computer generate typical incomes. And they could be, you know, in their concave part, they could be in the convex part. And they could be a mix of all of them. And running that for hundreds of 1000s of fictitious people, of course, they're in the digital twin, and looking at what the false positive rates were, and the false negative rates. And so I'm shaking my head, I just cannot understand. Now they could have possibly done anything close to that. And, and put up with false positive rates on the order of 70 to 80%, which would have happened.
Rosemary Pennington
You're listening to stats and stories. And today we're talking about the robo debt tragedy in Australia with Neil Cressy. And Dennis trewen.
John Bailer
You know, the the other thing that I was wondering why it didn't didn't flag as a concern was you said there were 500,000 or more that that end up getting these kinds of identifications. He's this hate mail about about being, you know, you're being fraudulent, you owe us money. Whereas in the past, when they were doing the the human processing of these records, it was more like 20,000 a year. And you mentioned in your
Dennis Trewin
there was a lot of public concern. There was a lot of media coverage, but the government basically ignored it in its size. It was good politics to be looking like they're tough on number of welfare recipients. So they put up with that flag or asked about error rates. And I quoted in an error rate of 1%, which was forgotten to detail what it was about it was something like that and inputted in a type error rate. So I think it's a large extent it was a politics driving it and public servants trying to do what it are our political leaders were wanting to do.
Noel Cressie
Well, on the grand scheme of things, I think we the three of us, agreed that there was not a lot of So what you might call really good data science that went into this, so that the metrics by which robodebt was achieved and finding cheats, and correctly and incorrectly sending out notices that metric was ill defined, there was clear that there was never an answer that was got from politicians or public servants, or civil servants. As to what that, you know, false positive rate and false negative rate was. So not only was the algorithm flawed, but the metrics by which the algorithm could be judged. were older, defined or undefined, as far as I can tell.
John Bailer
So when did this start to surface, you know, as a as a story in Australia, you know, so this was rolled out, it was starting to be used, what were what were sort of the early signals, that there there was something wrong, that there was a there's something that's happening, and it started getting this kind of attention in the popular press, and ultimately, the attention of people like you to look at what's happening technically, with these, this these models.
Dennis Trewin
Well, to be honest, it took us a few years to get really delve deeply into the statistical side of things. But as I mentioned before, there was a there are a lot of media comments about about a scheme and people being wrongly accused, and, and so forth. And I think the government really didn't take it seriously until legal claims or legal judgments found the scheme actually illegal. And the succeeding government decided to have this royal commission. And that's when we decided that we needed to do something that the statistical aspects of it couldn't be ignored in something as important as royal commission, we were recommendations that are made are generally taken up.
Noel Cressie
My mum, sorry. There was also a lot of media coverage about individual cases, tragedies, people taking their own lives, often young people with debts on the order of five or $10,000. And you can imagine receiving such a debt, and maybe not sharing it with anybody or telling your parents and then, you know, something tragic happening. It and there was also the shame that came with it. And it was pointed out to me by Anna Britton, who is the significance editor that there was something similar that happened in the UK was something called the post office scandal, similar in the sense that people were guilty, you know, and they it was up to them to prove their innocence. But worse in a way, and that that whole post office scandal in the UK took about 17 years to run out. And people were put in prison for things that, that the accounts didn't match that 1000s of pounds from gone missing. And these were subpostmasters people who had little side interests and ran post offices. So little, I mean, and it's still going on in the UK. But here in Australia, the robo debt idea has wound up with a Royal Commission, with a number of recommendations, some of which involve things that we put in to our submission.
Rosemary Pennington
When I was reading this, I was thinking about my mother, who is an elderly lady who is navigating right now sort of the worlds of Social Security and Medicare and her VA pension in the States. And every day seems like she gets another piece of mail saying, you know, your income has changed. And this way, you have to notify us when it changes. And she is just mentally exhausted from sort of navigating the system. And so as I was reading through this, I was thinking about these people who you just mentioned know who sort of were struggling under this and sort of facing these untrue debts. Right, and sort of the mental toll. How much money are we talking about sort of in the in the sort of grand scheme of things, did this robo debt suggest that these people were falsely owing the government?
Noel Cressie
The claim was that it was going to save the government on the order of several billion dollars over a period of several years. So let's say, you know, well, maybe about $2 billion a year. And that just sort of took off. You know, obviously governments want to say that sort of money and the assumption was that there were a lot of cheats out there. It was a conservative government. And it wasn't a good time, there was an attempt to actually balance the budget. And there were all these sayings like back in the black, were going back in the black. So the government was really looking for money and to be able to boast that they had balanced the budget. And they were looking at the welfare system to try and save that sort of money. So that was the, that was the feeling of the time. And it just, I mean, what happened, of course, is that people found that wasn't true, and they were managing to get certain amount of documentation. But the government would take a single person to court. And if that person wanted to go further, they would eventually get enough documentation so that the government would finally go what I'm sorry, no, no harm done. Whereas obviously, there was quite a lot of harm done. So it was like divide and conquer, it was taking individual cases, and treating it that way, rather than treating the whole Robo debt system as a whole and it being flawed. And it was only finally that Gordon legal took a class action lawsuit, which is where I found the algorithm, and that class action lawsuit was successful. And the government agreed that it was found to be illegal at the end of 2019. And it was wrapped up by 2020. So in four years, 700,000 people were sent notices, and of those about 500,000, was seen to be illegally requested. reimbursement.
John Bailer
You know, this certainly has has the flavor of a cautionary tale. You know, this is this is the case study of of, you know, kind of suspiciously motivated model development for application without really careful development and concern. Do you worry about this sort of being rolled out and other by other agencies by in other countries and other contexts? I mean, you know, you mentioned this idea of sort of a similar story being told at least analogous story being told in the UK. But but, you know, this seems like it's, it's, it's easier to do harm now, in some sense with with a model that can be built this, this quickly and applied. So broadly. So what if this is a cautionary tale? What are some of the lessons that we need to learn about better, better development and in rolling out of such such programs,
Dennis Trewin
if I could just quote somebody, one of the judges said, machine learning decision making systems are surely the way of the future, properly designed and monitored, they offer a trifecta of greater accountability, greater accuracy and responsiveness, and greater efficiency of administration. We're not arguing today that you you shouldn't be using data science or artificial intelligent methods in government work or government analysis. But what you should be doing is doing it properly. And in our submission to the Royal Commission, we made two two recommendations. One, that there be a guideline or good practice that civil servants would actually use, you can't expect every civil servant to be an expert in data science, but they they should know enough to know when to ask for help. And that's what, that's what one of our recommendations was, we also recommend that a position of Chief Data Science speed point we have a chief scientist now. And they were suggesting that there should be a person called the chief data scientists who all these types of schemes are passed by this, in that chief data scientist could then decide whether there should be a more detailed analysis, there should be more testing. We have good practices actually being automated. So you know, we're not arguing at all, but you shouldn't be doing this type of thing that Robert should be done properly.
Noel Cressie
In fact, this came out in the commissioners recommendations. By the way, a Royal Commission is a big deal. This all Royal Commission lasted a year and a half. And after these recommendations, there may be some prosecutions when things make happen against politicians and against senior civil servants that has not run its course yet. One of the I think there are probably 20 or so recommendations, but one of them said the evidence before the commission indicates the need for an office with broad remit to improve the use of automation and AI in public administration and our submission and was referenced in conjunction with his recommendation. So we feel that, you know, we work away trying to be good statistical and data scientists, but there is an opportunity to do public good. And every now and then, you know, the idea of doing it right, statistically, surfaces in these sorts of public domains.
John Bailer
You're mentioning that there was at least, there was a nod and some comment about one of your recommendations, that the idea of good practice. So we talk a little bit more about sort of this, these these two recommendations and other feedback that maybe you've received as an as part of including them and your your submission to this this royal commission.
Dennis Trewin
I'm sorry, what was your specific question, John?
John Bailer
Sorry. That's always a mystery. That's the the guests are supposed to read my mind. And so sorry. So when you make these kind of recommendations to this royal commission, did you have you received, received really some feedback about about them? Do you think it's likely that Australia will have a position of data scientists? Do you think there's some possibility of that? Do you and who might be involved in the good practice guideline?
Dennis Trewin
I think there is one thing that has happened as it hasn't gone as far as what we're, we've recommended, but the head of the Australian Bureau says strange vo statistic notices strange so decision has been given responsibility for providing a level of education of data science through the whole public sector. So I guess it's not teaching people to be data scientists, but to actually give them some understanding of what it's all about, and when they need to call for help from experts. So that's one thing that has happened in recent years. We're still the minister, responsible minister is considering the replication for a chief data scientists and No, no, I just had a chat the other day that I'm going to take a different go take a different approach to trying to get government consideration of that. I think it will happen. It may not happen tomorrow, or won't happen tomorrow, but it may take a year or two. But I think it will happen. Because data science is just becoming such an important way. It's so important to the way we operate. Look.
John Bailer
Go ahead, sorry.
Rosemary Pennington
No, did you ever?
Noel Cressie
Yeah, I do. The three of us have also been involved in use of superior statistical methods and data science in the in the COVID, during the COVID pandemic, and in a different Committee of Inquiry, we have also made submission in that respect. So it isn't just Robo debt, if you like leading the charge, there are other places where I think we can do public good. And we've done the best we can on that front as well.
Rosemary Pennington
That's all the time we have for this episode of stats and stories. Nolan, Dennis, thank you so much for joining us today.
John Bailer
Thank you for this important work.
Noel Cressie
Thank you. Great pleasure to be here.
Rosemary Pennington
Stats and Stories is a partnership between Miami University’s Departments of Statistics, and Media, Journalism and Film, and the American Statistical Association. You can follow us on Twitter, Apple podcasts, or other places you can find podcasts. If you’d like to share your thoughts on the program send your email to statsandstories@miamioh.edu or check us out at statsandstories.net, and be sure to listen for future editions of Stats and Stories, where we discuss the statistics behind the stories and the stories behind the statistics.