Nick Fisher left his position as Chief Research Scientist at CSIRO in 2001 to found ValueMetrics Australia, an R&D consultancy that carries out R&D in Performance Measurement, in which area he has consulted to a wide variety of business, industry and Government clients in Australia and overseas. He is a Past President of the Statistical Society of Australia and of ISBIS, and was founding Editor-in-Chief of the ISI online journal Stat. He is also originator and leader of the International Data Science in Schools Project (www.idssp.org).
Dennis Trewin was trained as a Statistician but has had 40 years of executive management experience in official statistics in Australia and New Zealand. He was the Australian Statistician from 2000 to 2007. He has also been an Electoral Commissioner and an Associate Commissioner at the Productivity Commission. He has chaired and been a member of Boards/Councils in the superannuation and university sectors. He is the current Chair of the Australian Mathematics Trust. He is a Past President of the International Statistical Institute, the International Association of Survey Statisticians, and the Statistical Society of Australia. He has been an Editor of the International Statistical Review. He is professionally accredited by the Statistical Society of Australia.
+ Full Transcript
Rosemary Pennington: As the COVID-19 death toll continues to rise researchers and public health professionals around the world are working to understand just how prevalent the disease is. News stories of the last several months have talked about contact tracing and featured images of drive-through COVID testing. One of the issues that’s come up with testing is whether we should test people who don’t show signs of infection. That’s the focus of this episode of Stats and Stories, where we explore the statistics behind the stories and the stories behind the statistics. I’m Rosemary Pennington. Stats and Stories is a production of Miami University’s departments of Statistics and Media, Journalism and Film, as well as the American Statistical Association. Joining me are regular panelists John Bailer, Chair of Miami Statistics Department and Richard Campbell, former Chair of Media, Journalism and Film. We have two guests joining us to talk about COVID today. The first is Nick Fisher who worked at Australia’s Commonwealth Scientific and Industrial Research Organization for 32 years before founding the research consulting firm ValueMetrics Australia. Fisher is also a visiting professor at the University of Sydney. Our second guest is Dennis Trewin of Dennis Trewin Statistical Consulting. Before founding his company Trewin worked as a government statistician for 35 years, eventually heading the Australian Bureau of Statistics from 2000 to 2007. Fisher and Trewin penned an op-ed together arguing that to really get a handle on the burden of infection in the COVID-19 pandemic, we have to start randomly testing people for the disease. Nick and Dennis thank you so much for being here today.
Dennis Trewin: Pleasure.
Pennington: What compelled you to write- and I’m not sure, is it one or two op-eds about this sort of issue of testing you know, all kinds of people for the disease?
Trewin: There’s only one op-ed but we had a few letters published
Pennington: Okay.
Trewin: The Australian Chief Medical Officer made some early calculations that unless something was done the pandemic would overwhelm our health system, and there would be something like 150,000 deaths. There’s some important assumptions in those calculations. One was that the base reproduction coefficient or infection coefficient if you like, was rendered at 2.6, which was the situation in Wuhan. The second key assumption was about the case fatality rate, which was somewhat less than what the WHO was saying early on. There were also assumptions about the incubation period. As a result of the high projected death rate, the National Government developed a strategy to flatten the curve, and what this in effect meant was that the reproduction coefficient needed to come down from 2.6 to something less than one. So, they needed to calculate that reproduction coefficient to ensure the strategy was working. To be able to estimate it they used the number of positive tests for COVID-19. This still required that they make a number of assumptions one of which was that they were finding 80% of symptomatic cases. They also made another assumption, which I think was very brave, that there were no asymptomatic cases that were actually pre-symptomatic. And these were the two of the unknowns that Nick and I were talking about in the article we referred to as “known unknowns”. And if you want to get reliable estimates of the reproduction coefficient you need to know what these two unknowns are. Now, in Australia, as estimated the reproduction coefficient has gone down from 2.6 to 0.4 as a result of social distancing, restrictions, quarantine, and so forth. I believe that estimate of 0.4 is a bit too low, mainly because the particular assumption about no asymptomatic cases that were pre-symptomatic cases is wrong. If it is not correct, there would be a negative bias in the estimate. 0.4 is the lowest number I’ve seen around the world, even lower than what New Zealand achieved, Norway achieved, using actually stronger restrictions than what we used in Australia. So, both Nick and I have got some suspicions about that number. Do you want to add anything Nick?
Nick Fisher: Thanks, Dennis. I’d just like to take a step back because the letters that we initially wrote to two national newspapers and then the Opinion Editorial that we were subsequently invited to write were born out of frustration with the fact that the national debate and the National Committee that was set up had no involvement with statisticians; they had input from epidemiologists, but not from statisticians. And there were some really obvious statistical questions that didn’t appear to be occupying anybody’s mind, like: How many people out there haven’t contracted the virus? How many have it but aren’t showing symptoms? How many have got symptoms that haven’t been tested? All of these obvious statistical questions. Also, how are things changing? And there was no mechanism and there is still no mechanism to estimate these things.
Richard Campbell: When I read your article, it made me really mad because nobody is doing it- it seems so obvious. I am not a statistician and I wondered very early on, why isn’t anybody doing random testing here? We don’t know- we have no handle on this. And I think it’s certainly true in the U.S. I mean, we’re just now starting to get- like Ohio is going to start doing some random testing right now. But there is certainly no U.S. plan for this. And I could say more about that but won’t.
Trewin: It is happening in some countries, most notably the United Kingdom, but there have been surveys we’re aware of being carried out in Iceland, Norway, Sweden, the Netherlands, I believe. The problem we face in Australia −and this comment was made in Sweden as well − is that the health experts seem to regard a test of a non-positive person as a wasted test. Of course there will be a large number of negative tests so they think that’s a waste of testing resources. I’m not sure what the situation is in the U.S., but in Australia, testing resources are no longer in short supply. So, diverting some of them to a national survey to find out what really is happening in the population would be getting value for money, not a waste of money.
John Bailer: Yes, those are probably the same groups that is opposed to having placebos in controls and experiments because what could that tell you? I’m really curious how was your op-ed received? What kind of feedback did you get to this work?
Fisher: Well, we published the op-ed which only provoked one letter in response. However, we also managed to communicate the op-ed to two key groups. Dennis forwarded it to the Australian Bureau of Statistics and I managed to get it considered by the National Commission that had been set up to lead the response, and they took notice and they actually wrote to us, saying thank you, we understand and we’re going to have discussions about it with the ABS..
Trewin: I should say the ABS has been very supportive. They have now done the design work and are ready to implement the survey. The resistance is coming from the health people. The National Coordination Committee have said that they would like to do it, but they can’t get the health people on board at this time.
Bailer: What’s the opposition? Why would they be opposed to this?
Trewin: I think- it’s part of what I mentioned before, there’s this feeling that the many of the tests would be wasted. We had another complication, Australia bought a whole lot of serological tests, millions of dollars’ worth, and it turned out that they were faulty. So they want to discard them but John, as you know, if there is measurement error, and you know the properties of those measurement errors, you can still use the tests.
Pennington: So, if you were going to explain to my mother why doing random testing of individuals is important to understanding the scope of COVID-19 how would you explain that to them, to a layperson? We talked a lot about sort of the statistical reasons in your first response, but sort of, you know why should the everyday Australian or everyday American be supportive of random testing? What does it tell us that what we’re doing now won't?
Fisher: It will tell us how things are now in terms of how many people are infected or likely to be infected and how these things are changing, especially in response to initiatives taken by governments.
Trewin: If I could just add something, the public debate is such in Australia that I think most people have a broad understanding of what the reproduction coefficient is, especially those who can influence public opinion. The Chief Medical Officer in his presentations has shown graphs tracking the effective reproduction rate over time so I think there’s a general understanding that if we’re going to control a world pandemic we have to keep that less than one. We’re at the stage now where restrictions are being lifted stepwise, and it’s still important that that reproduction coefficient is less than one. So, I think that is the useful way of trying to explain to lay people exactly what’s happening and why restrictions can’t be lifted faster than what a lot of people would like them to be lifted.
Campbell: Can you also explain- you mention in your editorial both regional testing and national testing. What are the advantages of doing both of those? I know now in Ohio they’re talking about doing random testing, but why shouldn’t we do national testing and just forget about regional testing? What are some of the advantages of doing both?
Trewin: The pandemic has been managed by a National Cabinet that was set up especially for this purpose. It’s comprised of the Prime Minister, equivalent to your President, and the heads of each state government. There has been a mixture of managing this pandemic nationally, a number of national initiatives including a broad framework for establishing and lifting restrictions and the stimuli on the economic side, and state-specific initiatives within this framework but some flexibility on timing depending on their circumstances. The other thing is that a number of State borders have been closed. So, it does make sense to measure both nationally and regionally. The regional numbers are actually very important to the state governments because they have an extra decision to make- whether to open state borders or not.
Bailer: So, as you’ve looked at this, has this the first time either of you has written an op-ed like this or letters to editors about an issue in your community?
Trewin: For me, it’s the first time as my person; I was invited to write a few as the Australian Statistician. I don’t know about Nick.
Fisher: I’ve written a few letters when I’ve been annoyed about something. And my batting average was very low initially but it’s improving now. I think I’m learning to speak in English.
Campbell: How has the news coverage in Australia been? Do you have criticisms? Has it been pretty good? Are there stories they’re not getting? Like this one which clearly is not getting like this one, why we haven’t been doing random testing earlier, but generally, how is the Australian press been?
Trewin: Oh, I think it’s been fairly balanced. There are people who think we should be continue to be in a more restricted situation and others who think we’re going too slowly. I think the point of views of both sides has been covered pretty well.
Bailer: So how quickly do you think this could be implemented? If there was a decision to say okay we’re going to launch this, do you have a sense?
Trewin: It could start this week.
Bailer: Wow. So, can you describe the design that’s being considered for doing this collection?
Trewin: I’m not across all the detail but they do have an address file that they could use as a framework for their sample so it’s relatively straight forward to implement that. They were also considering- and I’m not sure where they got to in the end, actually using a subsample of the Monthly population survey that gives them an advantage that they got all the contact details now, not just a street address, and they’ve got a respondent who understands the ABS., and also they’ve got a whole range of data from past surveys that they would be able to use, so I know they’re seriously thinking about doing that but I’m not quite sure where they ended up.
Fisher: There’s another component to this, Richard, and that is: you don’t just want to know who’s got it, who hasn’t got it and might get it, or who’s got it and is asymptomatic. You want to be able to study the evolution of it, how long does it last in people who get it, can they re-contact it, and so on. So, part of the survey process −it’s not just a survey it’s an ongoing process − would be related to tracking some individuals, in other words, a longitudinal study as well as a basic random sampling study.
Pennington: You’re listening to Stats and Stories and today we’re talking about COVID-19 with\ government statisticians Nick Fisher and Dennis Trewin. So I think Richard asked about the news coverage, and then obviously this issue of not randomly testing individuals, obviously is a story that maybe needs more coverage, but are there stories in the numbers related to this pandemic that you think journalists are not covering or they could be covering better?
Fisher: I think I’d like to refer to the title of our editorial, which got changed before it was printed, which was Estimating the Known Unknowns. And I think there are still a number of known unknowns and the journalists have not got data about that so they can’t write stories. They don’t know what they’re missing out on.
Pennington: Right, yeah I mean journalists- we tend to like to cover things that you know- we chase the facts. And I think, even prior to COVID-19, covering anything around medicine or science can be difficult because uncertainty is in the name of the game when it comes to this research, and journalists are often deeply uncomfortable with the idea of uncertainty when it comes to their own reporting, so I was just wondering.
Trewin: Well perhaps I should correct my answer, the media have done a good job of reporting the numbers that have been presented to them. Behind those numbers, as you just said, there’s a high degree of uncertainty, there are some really important assumptions which I think could be challenged. In terms of the language that we’ve been using, more knowledge of the unknowns is required. However, in terms of what journalists have been given they’re presented it in a reasonably good way I think.
Fisher: I wonder if I could segue to something that might be of more general interest to John, for example, and that is that maybe there is a huge opportunity here for statisticians to work with journalists to get the message across that you don’t need to collect an enormous amount of data to get actionable data, that’s the first thing. And the second thing is the importance of having a general statistical voice feeding in at the very top level of government decision-making. Now, you might think, oh well the head of the Australian Bureau of Statistics – there’s a senior statistician feeding into that National Commission. Well, first of all, the head of the Australian Bureau of Statistics doesn’t need to be a statistician, it could be an economist; and secondly, that’s a person whose interests are in one particular area. However, there are often more generic issues, about which a more general statistician would be able to have informed opinions, and such opinions aren’t being fed in. The thing that struck me in all of this is how few epidemiologists have got any sort of broader statistical knowledge: they don’t seem to be interested in broader statistical questions. And one of the worrisome things I think for both Dennis and me has been the fact that (and this is not just in Australia) the different epidemiological groups they tend to compete. At least a few governments, particularly the Australian government and the British government, have been relying on input principally from one group of epidemiologists, rather than from the whole industry. So, there are some broader issues that could form the basis of an investigative report.
Bailer: I’ve been pretty impressed at some of the rapid modeling results that I’ve seen come out. I mean there’s been work that’s tracking something like 32,000 cases in Wuhan and there has been some other work that’s been focused on the tracking of cases in India. So there has been some very quick turnaround in work that’s been presented. You know, your comment, Dennis, about the estimates of the infectivity rate, you know, and some of the uncertainty about this, and I look at some of the model projections and I often think there’s this almost false sense that’s communicated when I look at this, that people are not realizing that those are extrapolations where there’s a future where there’s uncertainty in the inputs. And I wonder just what are some of the strategies that you might suggest for better communicating some of that uncertainty, you know Dennis and Nick as you think about that and think about these models and how they’re being used and promoted?
Campbell: Can I add to that too John because that reminds me of an early number I saw, and early prediction, that in the United States, eventually when this thing was finished two-thirds of the population will have had COVID 19, and I always wonder how do they know that? Where is that prediction coming from? How do you make these kinds of predictions without doing this kind of random testing? That’s- I was flabbergasted.
Trewin: Well I think as statisticians, if we looked into some of the models that were making those sort of estimates, you would find significant flaws. A lack of any sensitivity analysis to the assumptions is one flaw. Using a well-publicized case, the Swedish epidemiologists made similar model based estimates and seemed to have followed a herd immunity strategy, so even though they have a very high number of positive COVID-19 cases, surveys have estimated the proportion of the population of antibodies was only 7% whereas I think they were expecting it would have been close to the 35% at this time. Is the models that are not right or is it the assumptions. I suspect it’s the latter actually, but sensitivity analysis would have resulted in range of estimates possibly leading to a different strategy. At WSC meetings there are joint sessions with other professional associations and at the next meeting, John, I think we should do something with the international epidemiological association. There is going to be other pandemics, and hopefully, we’re through the worst of COVID-19 by July next year but I think there’s some really important learnings, particularly how statisticians and epidemiologists can work together better in helping resolve future pandemics.
Bailer: I think that that’s- that comment that there will be other epidemics, really comes back to your article. I mean having this type of mechanism for studying the population now, routinely in place and routinely implemented, seems critical to me. At times I worry that we think of public health as this luxury that we don’t have to consider until we do. You know until things are really at high-risk. So, I think that your call for this current pandemic is something that’s probably- it’s a framework and a call for something that needs to be in place as we move forward for monitoring and considering the health of our populations.
Campbell: Yeah, I have a question for the statisticians. So, I’m reading this article about what Ohio is going to do for random testing and it says they’re going to randomly test 1,200 volunteers. Now, can you do a random test from volunteers? I don’t- I know enough to think that’s not right; that was a headline. So, what’s the deal there? John, do you know what’s going on in Ohio? And explain why you can’t do a random test from a sample of 1200 volunteers.
Trewin: So, who are going to get tested if you are relying on volunteers? The odds are those that will volunteer will be persons who have got a bit of time on their hands, who feel a bit insecure about catching the virus, and perhaps those who have got some symptoms. It is going to be a very biased sample, I think you’ll find that the sample will hugely under-represent young adults. The lower socio-economic groups with less mobility options will be vastly underrepresented.
Campbell: Thank you that’s what I thought.
Bailer: Yes, what he said. You know- you know I saw that when they said voluntary I’m like well that’s just crazy.
Pennington: How would you imagine a national random test to work? If- because you know in the United States people don’t want to wear masks. So, there are protests about wearing masks in public, and so I just wonder how- how do you pull off a national-sized random test of people to try to figure out where the disease is?
Trewin: Well, a number of countries have done it. The UK is doing a very large scale survey right now; they’re even publishing results from it, so it can be done. The test itself apparently doesn’t require a lot of training. It can be administered easily by a nurse, but even someone who’s not a nurse can get a small amount of training and successfully undertake the test.
Fisher: Rosemary, you know in an ideal world you have the Bureau of the Census, in your case, design the sampling framework and then a drone-vampire moves around and sucks five to ten mils out of the selected individuals and takes it back for analysis.
Bailer: I want to see you write that up for review subjects approval process Nick. The Drone Vampire Solution; I love it.
[Laughter]
Trewin: In Germany, they used vehicles referred to as a COVID taxi I believe. It would go around to multiple households and get the test but the questions themselves, such as demographic and other details, could be done by telephone using professional interviewers.
Campbell: In a country like the United States with 330 million people, how many people would you have to test in a national sample? Random sample?
Trewin: I think you can, from memory, survey about 10,000. This is not a situation where you need a really precise number. If you get an order of magnitude number with a reasonable confidence range, it’s still quite useful so, you don’t need massive surveys. I would guess a survey of 10,000 would be adequate.
Fisher: Richard, it depends on the degree to which you want to be able to drill down and look at things at a state or even local level, to look at specific subgroups. But if you’re just forming a more general picture you can get away with what appears to be a ridiculously small sample size.
Pennington: Well, that’s all the time we have for this episode of Stats and Stories. Nick and Dennis thank you so much for being here.
Fisher: Thanks for the airtime.
Pennington: Stats and Stories is a partnership between Miami University’s departments of Statistics and Media, Journalism and Film, and the American Statistical Association. You can follow us on Twitter or Apple podcasts or other places where you can find podcasts. If you’d like to share your thoughts on the program send your email to statsandstories@miamioh.edu or check us out at statsandstories.net and be sure to listen for future editions of Stats and Stories, where we discuss the statistics behind the stories and the stories behind the statistics.