Nicholas Fisher left his position as Chief Research Scientist at CSIRO in 2001 to found ValueMetrics Australia, an R&D consultancy that carries out R&D in Performance Measurement, in which area he has consulted to a wide variety of business, industry and Government clients in Australia and overseas. He is a Past President of the Statistical Society of Australia and of ISBIS, and was founding Editor-in-Chief of the ISI online journal Stat. He is also originator and leader of the International Data Science in Schools Project (www.idssp.org).
Episode Description
The use of statistics to improve processes and business industry government and academia is the focus of this episode of Stats and Stories with Deming Lecturer Award winner Nicholas Fisher
+Timestamps
What got you into performance measurement? 1:00
How important is context in productivity measurements? 4:30
Measurements can affect behavior 9:12
How should journalist report of performance statistics 13:27
What advise would you give people studying performance 17:45
Who are some of the legends in this field 20:30
+Full Transcript
John Bailer: When I was starting my career local business and industry folks paid for my registration fee to attend a four-day workshop taught by W. Edwards Deming. They believed that University faculty members needed to appreciate his message and the impact of his quality improvement message on industry and business. The use of statistics to improve processes and business industry government and academia is the focus of this episode of Stats and stories where we explore the statistics behind the stories and the stories behind the statistics. I’m John Bailer. Stats and Stories is a production of Miami University’s Departments of Statistics and Media, Journalism and Film and the American Statistical Association. Joining me is regular panelist Rosemary Pennington from the Department of Media, Journalism and Film. Our guest is Nick Fisher. Fisher runs an R&D consultancy in performance measurement and is visiting professor of statistics at the University of Sydney. He was also the 2019 Deming lecturer. Nick, thank you so much for being here.
Nick Fisher: Pleasure. Thanks, John.
Bailer: Well, Nick you were awarded the Deming lecturer for outstanding leadership in promoting the discipline of statistics including remarkable contributions to broad areas of theoretical and applied statistics, especially in performance measurement systems, an area of great interest of W. Edwards Deming. What got you interested in the field of performance measurement?
Fisher: Well I fell into it. I was running a research group scattered around Australia. We had adopted a particular approach to how we’re going to manage things, and we had ourselves assessed against best practice criteria in management and we came out worst in the area of how we use data and information to run ourselves-
[Laughter]
Bailer: That had to have been painful, Nick.
Fisher: It was a huge embarrassment because some of my people were evaluated for the Australian Quality Awards, and so I started looking into this and I discovered to my great astonishment that nobody had ever come up with a solution to the question: What sorts of data do we need to run this organization? What sorts of data will tell us what we’re trying to achieve and how we’re going in terms of achieving it? And that goes for companies, Universities and any organization you like. To this day the regular monthly meetings have board reports which are almost certainly incomplete.
Bailer: Based on what you just said is what data would be needed to run an organization and the process of that includes determining what you’re trying to achieve and then evaluating your success at achieving that, is that a fair summary?
Fisher: Yes.
Bailer: So, could you give a simple example that would help someone get their claws into this idea of measuring performance of an organization or a process?
Fisher: Well the usual sorts of metrics that you use to evaluate how well an organization is going are financial, such as profit, profit before tax, …. A whole list of financial metrics. Sometimes there’s something about customer satisfaction and maybe staff turnover. Unfortunately, and this has been known for a long time even if it’s been ignored for a long time, there are a lot of other factors that affect the long-term viability of a company. How well they treat their customers, how well they treat their people. How they behave in relation to the wider community. All those sorts of things matter in the long term even if you try to exploit them in the short term. So how do you measure things like that so that you know that you’re being successful in your relationships with each of these other groups?
Rosemary Pennington: How important is it when you’re thinking of how to measure performance to consider the context and history of the company in the industry? Because I could imagine some wide-eyed person who says: well this measurement works at this one company, let’s take it some place else and see if it works for us and how there’s not maybe quite an easy transfer between industries or companies. Is the context and the history important as you think of how to measure things for your clients or for organizations?
Fisher: Well there are lots of different things bundled up in what you just said. So, let’s just pick one or two. There are generic principles that can be applied all the way across from company to company, industry to industry, but the history of the company may be important for some companies, it may be part of the reason that some people are attracted to work for a company or purchase from that company or whatever it happens to be. On the other hand, there will be other reasons why those things are totally irrelevant. And generally each company will have its own specific circumstances like, say, with the university you know you’re all competing within the student market, but you’re all delivering something that’s slightly different and you have your students and they’re different from other people’s students.
Bailer: You know in reading some of the articles that you’ve written on this I appreciated the idea of saying that the questions that you answer with the performance measurement system are: Where are we now? Where are we heading? and Where do we need to focus attention? I mean those all seem like really powerful touchstones, and you know that can apply it seems at any level of an organization you know. So, Rosemary and I could think about those questions in terms of our own individual careers. We could think about it in terms of our departments. We could think about it in terms of our divisions, in terms of our larger institution. So, I think that’s an example of some of those general principles that are any unit would be considering that?
Fisher: Yes. And beyond that it gets to issues like how do you measure? How do you assess research quality? How do you assess the different graduate programs in statistics in the U.S.? How do you make an appointment? How do you compare all those different researchers who have applied to be a professor at a university? There are a host of things and it’s when you start pulling on that little piece of wool that there are more and more questions to be asked. What does it mean to do high quality research? How do you measure high quality research? Who should be making that decision? Is it something where you count numbers of citations? Or numbers about papers published? Or is it something a little bit more subtle?
Bailer: You know the other thing that I wonder is as new forms of production emerge, say a podcast is one example of an intellectual product that really didn’t exist historically. Other examples might be software packages that are contributed or blogs. There are intellectual products that might have impact and to me it’s an interesting question on how that might be considered and evaluated in terms of the performance or these questions that you have about where do you want to focus your attention or how do you want to evaluate performance.
Fisher: Well as my late great friend Myron Tribus would say in relation to podcasts, who’s the customer here? For whose benefit are you doing this? Who is going to make a judgment about whether it’s a good podcast or a bad podcast? What are their quality criteria?
Bailer: I think they would say having Nick Fisher as a guest. A clear measure of quality criteria.
Fisher: Yes, but what sort of quality criteria?
[Laughter]
Bailer: You have to be able to anchor scales Nick.
Fisher: Well yes but I’m not sure which end that’s anchoring.
Bailer: We’ll leave that as an open question. So, I love the example that you talked about in some of your other writing about how measurements can incentivize behavior and have unintended consequences. That to me is a pretty sobering thought. You talk about the idea of teaching the tests or other things that you might do that really are not necessarily what you want to do. You gave some examples about physicians and their compensation based on numbers of patients, or arrival times of trains. Can you describe some of those examples for the folks that might be listening?
Fisher: Well a really good example is provided by the methodology of Six Sigma. The history of Six Sigma is not all that well known but I got it from a primary source so to speak, so I believe that my version is correct. Motorola was having big quality problems and casting around for a way in which they could improve what they were doing for their customers and they wanted a single metric they could drive right through the organization that everybody would understand really easily. And so, they decided on the cycle time of each process. Whatever it might be – a small production process all the way up to the complete sales process from the moment of initial order to the moment of the client or customer’s check being banked. They said right we have to minimize cycle time on everything, and they went flat out on doing that. That was fine, but it had unintended consequences because what people were focusing on was doing something as fast as possible but not necessarily doing it to specification. So, they soon realized they needed a second metric. So, then they said not only are you to do it fast but almost every time it has to be done correctly. This meant that they had to introduce something that says how well the process was controlled in relation to what it was trying to achieve. Another example was the notorious case of Sunbeam where the CEO, Al “Chainsaw” Dunlap who was the darling of Wall Street at one stage because of his supposed ability to turn companies around. He would come in the company, strip out underperforming assets, sell them off and suddenly drive the share price up and then move on and his compensation package reflected how well he did this. At Sunbeam there was a great incentive for him to meet quarterly and six-monthly sales targets and on one occasion Sunbeam recorded a record number of barbeque sales in the middle of winter, which was a trifle unusual for an American winter.
Bailer: It’s called global warming Nick.
Fisher: He kept on trying to raise the expectations of Wall Street and the share price was going up and up and his compensation package was going up and up because of the bonus system. The trouble is that what he had down was to bring forward the summer’s sales forward, so that the market collapsed in the following summer. And there was also creative accounting to explain things away. Eventually, it got him fired from Sunbeam, but he was never properly prosecuted for all the damage he’d done.
Pennington: Oh wow. So for journalists who are covering- because a lot of companies will when they have performance metrics that look impressive will make this big dog and pony show saying yes, we’ve had this huge surge in sales or all these customers are satisfied with what we’re doing, what advice would you give to journalists who are covering these stories? What should they be keeping an eye out when companies are sort of pushing or I guess even Universities certain performance metrics as examples that they are doing well- how should they approach these stories?
Fisher: Well, I’m afraid the journalists will have to learn to ask a few questions. Firstly, of themselves and then to the companies. So, a classic example of a metric that’s gone all over the world, and many companies including all the large banks in Australia were using at least until very recently, is Net Promoter Score. I won’t describe the details of it but it supposedly- it was introduced as a quick fix that avoided all sorts of expensive marketing campaigns to ascertain how the public viewed your offerings and in came Net Promoter Score and everybody picked it up and that’s what they use to compare themselves with other companies. Unfortunately, it is not fit for purpose. It’s actually not good for business and it’s not good for the consumer. It doesn’t lead to improved products and services, but it has made a lot of money for companies that consult in this area. You can read about that in some of my writings.
Bailer: You’re listening to Stats and Stories and today we’re talking to performance measurement consultant and University of Sydney visiting professor of statistics, Nick Fisher. Nick, I’m glad that Rosemary asked about that idea of covering the idea of measurement. Because a lot of times it seems like you know an organization could cherry pick a particular measure that shows them in the best light, and even though you may have other indicators suggesting that there’s a problem in an organization if you pick the right thing, so the example that you had of the winter sales of barbeques, there should have been flags there. So, you now, you’ve suggested that there’s some data self defense that journalists can do. Do you have other kinds of hints or clues or things that people should look for in this kind of assessment?
Fisher: Well I think the thing to ask is what sort of behavior is this likely to drive? If they’re using that sort of metric how are the people who are being judged by this metric likely to behave? I mean just look at the academic community. If you’re judged on how many articles you publish, what are you going to do? Are you going to write one long article that covers the subject completely or are you going to try and get it published in several short articles? If you’re a journal editor and what you’re being judged on is how often the articles are cited and the articles from your journals are cited what are you going to do? You’re going to encourage citation of articles in your journal by people submitting articles to your journal and that drives up that particular metric for the journal. I’ve had personal experience of this. Having had an article accepted for publication, I was asked by the editor when he sent the gulley proofs to check back issues to see what other articles i could cite. Measurement drives behavior.
Bailer: Although there’s a call there certainly for professional ethics that also is embedded in what you’ve just described.
Fisher: Well I’ve never seen that written into the list of duties or requirements for a journal editor.
Bailer: It should be present.
Pennington: What advice would you give to someone who wants to go into this field?
Bailer: You took my question Rosemary. That was on my queue.
Pennington: You’ve got to be quicker John; you’ve got to be quicker.
Fisher: Someone who wants to go into this field, like a research student or somebody like that?
Bailer: To work in the field.
Pennington: Who wants to work in this area of performance metrics, you know, what do they need to be keeping in mind? Are there things they should be studying in school? What advice would you give to them about making their way into this industry
Fisher: Start reading. There’s not all that much to be read. Start looking around: look around in their own daily lives to see how people are using measurements. For example, at a University, why are you giving me this course satisfaction survey? It doesn’t appear to be relating to the things that are important to the people in this class. Start asking questions. And you’ll suddenly find that you’re not getting good answers. As I said at the beginning, I didn’t set out to work into the area, I fell into it because I had a specific issue. I think this is the case for a lot of researchers, at least in statistics, because problems come to them and they’ve responded to that problem. The problem pulls them instead of them trying to push the issue. Unless they’re theoreticians:, then they can do what they like.
Bailer: So, Nick you received the Deming Lectureship last year. We haven’t
Fisher: Excuse me John? The Deming Prize is a very different kettle of fish.
Bailer: Thank you. Appropriate fish kettles are important. We’ve not talked about Deming in kind of very formal ways. So, could you give a little bit of context and history of the impact of not just Deming, and I know this is asking a lot. I’m not asking for your lecture revisited, but just some highlights you’ve mentioned- the giants of this field that helped influence and shape the direction that you’ve talked. Can you just give some comments about Deming and you mentioned Myron and others that have had great influence on you?
Fisher: Well I think that the- not all that many people in the American Statistical Association would actually appreciate the broad range of Deming’s activities. He was employed by the Bureau of the Census; he did wonderful work in the Bureau of Census, he did some important research after he left the Bureau of the Csensus. He was a consultant in transportation statistics; the leading consultant in America on analyzing transportation statistics and so on. However I think he was largely unknown outside survey sampling until 1980 which is when there was the broadcast on NBC “If Japan can why can’t we?”, and Deming was suddenly thrust into prominence for his role in the revolution in manufacturing in Japan. What had actually happened was in Japan that after the wat or towards the end of the war when MacArthur had been sent out to see what he could do to- or what to take charge of post war reconstruction in Japan, the first thing he realized he needed was a communications industry capability so that he could get hold of some communication equipment that would allow him to go from town to town to find out what they needed. Did they need medical supplies or water or food or what did they need? Because everything had been destroyed by the bombing. So that led to an extraordinary person Homer Sarasohn, brought over to supervise the creation effectively of a whole communications industry and in the process of doing that Homer realized that for companies to operate well he had to teach them how to manage. Which he did do. He taught a leadership. He wrote out a course with a Charles Protzman on principles of industrial management and he taught a number of industry leaders of companies he’dset up or brought back to life how to manage. You may have heard of some of these companies by the way. National, Sony, Panasonic, Matsushihta, … .
Bailer: Yeah they did okay.
Fisher: And then after a while - and there’s a lot of stuff that I’m leaving out - it was clear that he needed to teach some statistical process control to help them manage the processes and their companies, and he actually got as far as writing a short book about this. But then MacArthur left to go to Korea and took Homer with him so Homer contacted Shewhart and said: can you come over and teach these people statistical process control? Shewhart said, I’m too busy but there’s a young guy at the Bureau of the Census who knows this stuff. Why don’t you get him out? So that’s how Deming was invited to Japan to teach SPC. Now Deming had been out there in 1947 to design the first postwar survey but he came out in 1950 to teach SPC, but he had also formulated his own complete philosophy for management and so he taught that as well. And Deming, for those that have never seen him compared with the average Japanese person was a huge contrast: he was 6’7“ tall and he towered over the small people in Japan, he spoke with a slow, very deliberate voice, almost in biblical tones, and it was a powerful experience to attend one of his courses as you know John, and as I know. However Deming himself was never in line management. He never had a role of supervising other people and having run a system and so there were other people who actually had to find ways to capture some of his thinking and to make it work in companies and there are many other contributors. Myron Tribus was one and I was very fortunate to have got to know him. But at the same time as Japan started picking up, America was relatively unaffected by the Second World War and so they became in a sense lazy and their production systems became sloppier and their former good management practices lapsed and Deming’s session in 1980 was a huge wake-up call to say: you know you’re going to have to start to do better. You know there’s a whole country over there that is doing much much better than you are in terms of the quality of the products and services it’s providing. And they’re using your techniques to do it.
Bailer: And they do give a Deming Prize in Japan for this work for contributions in quality.
Fisher: Whether it’s a good Prize or not it is another question. The reason I say that is because it is incredibly demanding of the capabilities of a company, hugely demanding and in fact it is a goal that a lot if companies strive for and they do succeed: there are companies in India now, several companies, that have won the Deming Prize, but at least in one case in America the cost was too great and the company somewhat imploded in trying to keep those standards up.
Bailer: Well Nick, I’m afraid that’s all the time we have for this episode of Stats and Stories. Thank you so much for being here with us.
Fisher: It’s a pleasure John, thanks for the chance to talk.
Bailer: Stats and Stories is a partnership between Miami University’s departments of Statistics and Media, Journalism and Film, and the American Statistical Association. You can follow us on Twitter, Apple podcasts, or other places where you find podcasts. If you’d like to share your thoughts on the program send your email to statsandstories@miamioh.edu or check us out at statsandstories.net, and be sure to listen for future editions of Stats and Stories where we explore the statistics behind the stories behind the statistics.