Harrison Schramm is a Principal Research Scientist at Group W as well as President-Elect at the Analytics Society of INFORMS. Prior to joining INFORMS he was a Senior Fellow at Center For Strategic And Budgetary Assessments and has been a leader in the Operations Research community for the past decade. Before that, he had a successful career in the US Navy, where he served as a Helicopter Pilot, Military Assistant Professor at the Naval Postgraduate School, and as a lead Operations Research Analyst in the Pentagon. His areas of emphasis were large-scale simulation models, statistics, optimization, and applied probability. His research interests lie at the intersection of data and mathematical models.
Episode Description
Parents, educators, and activists have all raised concerns about the impact of COVID on the educational experience of students. For high school students, these issues are amplified as they consider graduation and what may come after. The impact of COVID on high school grades is a focus of this episode of Stats and Stories with guest Harrison Schramm.
+Full Transcript
Rosemary Pennington
Parents educators and activists have all raised concerns about the impact of COVID on the educational experience of students. for high school students, these issues are amplified as they consider graduation. And what may come after the impact of COVID on high school grades is a focus of this episode of Stats and Stories where we explore the statistics behind the stories. And the stories behind the statistics. I'm Rosemary Pennington. Stats and Stories is a production of Miami University's Department of Statistics and media journalism and film, as well as the American Statistical Association. Joining me our panelists John Bailer, chair of Miami statistics department, and Richard Campbell, Professor Emeritus of media, journalism and film. Our guest today is Harrison Schramm. Schramm is the principal research scientist at group W. President Elect at the analytic Society of informes. Prior to joining in forums, he was a senior fellow at the Center for Strategic and budgetary assessments and has been a leader in the operations research community for the past decade. Before that he had a successful career in the US Navy, where he served as a helicopter pilot, military assistant professor at the Naval Postgraduate School, and as a lead operation research analyst at the Pentagon. His research interests are at the intersection of data, mathematical models, and policy. Schramm is also a co author of an article and significance magazine detailing how COVID protocols have affected grades in one California High School. Harrison, thank you so much for being here.
Harrison Schramm
Thank you.
Rosemary Pennington
And as as the child of of a Navy man, I'm so happy to have you on the show today, by the way, I am, too.
John Bailer
I am to Navy. Yeah. And World War Two. Yeah.
Rosemary Pennington
Given your background and your research background, I'm wondering if you can talk us through how you became interested in this in this particular issue of grades during COVID.
Harrison Schramm
So it started because it was a question that just seemed like it was begging to be answered, both locally and nationally, seeing a lot of conversation about how this is impacting the education of kids. You know, we can while I'm not I think the jury's still out on what the whether this was worse or better, it is certainly different by any objective measure. So it just sort of became this question of gosh, you know, I'm a statistician, I ought to get some data, and I ought to put some data behind this. And so that's kind of how it got started.
John Bailer
So just, I'm gonna do it, my journalist colleagues love and that is, we're gonna get you not to bury the lead. So we're gonna so you know, even though there's part of me that loves to ask you questions about, like, how did you study it? What data Did you collect? You know, I'm sure I've learned to not do that. I'm gonna ask, what was the most surprising thing that you learn from this, this investigation?
Harrison Schramm
So so the most surprising thing that I learned is how cooperative the school district was, because, you know, when you're and and I'm sure you've had this experience, as a statistician, you want to study something, but it's not something that you have access to. So you have to go sort of around hat in hand, and you have to ask for data. And usually, when you ask for data, you ask for everything. And I told my two colleagues, one of whom is a teacher at the Pacific Grove High School, and the other one is my daughter, I said, I have very low confidence that they're going to say yes to this. And I pitched the question. We had one informal meeting where I assured them that I would be responsible with the data. And I assured them that I would let them have first peek at what the findings were. And today's too much to my surprise, came back and said, Yes. So getting started being able to do this in the first place was really the most surprising thing.
Richard Campbell
In your in the limited study that you did, did you expect to see grades improved?
Harrison Schramm
I wasn't sure. So there are three epochs, if you will. There's three regimes if you look in the paper, there's everything that's normal. And that was before COVID began approximately a little over a year ago, hard to believe that then the school district when COVID hit it was during the middle of second semester of academic year 20. And so they had a policy that was called hold harmless, and under the hold harmless policy, your grades could go up, but they could not go down. And because it's statisticians listening to this, there's sort of an optional stopping mechanism at work under the hood here, so it did not surprise me the grades went up during hold harmless. What did surprise me is that there was a lift of about two tenths of a grade point During the pure distance learning that we've been seeing for the first semester of this year, that's that's the data that we had available,
John Bailer
Although there was less than consistent across all of the subjects,
Harrison Schramm
Um, no, but there was a lift in all the subjects, I mean, qualitatively, all of the subjects had a big lift during hold harmless, and comparing the first semester 2021 against the historical, they all showed some small lift, which, which we thought was noteworthy and interesting.
Rosemary Pennington
I actually was, I'm interested to hear what it was like for you to work with your daughter on this. But I sort of raised this point, because I have a high school senior who was graduating and I told her I'm like, we're gonna interview a researcher who did research on grades during COVID. And she's like, yeah, COVID really helped my grades made them higher. And she like they did a hold harmless here, too. And then she said, like, she's also been a lots of music classes, because she's a musician. So she's like, that also probably helped. But her grades during the distance learning were also higher. And she said, I was able to use my notes a lot for a lot of things, which I wondered if you could talk a little bit about sort of what you sort of found that bubbled up as to sort of what people were saying might have been helping sort of keep the grades higher. Oh, it
Harrison Schramm
Was our It was our, our, our opinion, arch are finding that the lift came from the bottom. And what I mean by that is that kids who are doing well didn't do better. But there were fewer D's and F's assigned. By the way, there is a tremendous gender difference. In the assignment of D's and F's and I, it was known to the educators and I hadn't really realized it and eyesight so starkly on the page. But freshmen and sophomore boys get a lot of apps compared to the young women that they're in school with, but the lift came from the bottom. And I think that that is because teachers in this regime are much more hesitant to fail a kid. Because Because COVID is hard. I mean, and you know, not to get too much off track. But just getting up and getting through our days during this has been hard. So I think there was a lot of forgiveness given.
Richard Campbell
I'd like to follow up on the daughter question, too. So when my daughter was 16, she and I wrote an article together for television quarterly on our shared interest in Harry Potter and Buffy the Vampire Slayer. So I wanted to know what your daughter's role was in this. And I know she's interested in data.
Harrison Schramm
Oh, yeah. So So what was interesting working with my daughter, so first of all, to do the piece responsibly, we needed to have the voice of a student. And because of COVID restrictions, and also just, it turns out, the obvious choice of someone to work with was my daughter. And it was interesting, I think it was interesting for her, because she got to see how people who are doing this work actually do it, which is not how it's taught, right? She got to see me fumble and write code that didn't work, and then go back and try again and do things that work. So she got to learn that it's a journey of discovery. You know, it's not that we're all brilliant and sit under the tree and an apple hits us on the head. And we know there's gravity, there's a lot of fumbling and false starts. So she got to learn that what I thought was particularly refreshing, is my daughter doesn't have a college education. Obviously, she hasn't been through the journal process or the review process. So she wrote with a voice that was more honest, and fresher than sort of the stylized, the stylized musings that academics can be prone to.
John Bailer
So you hinted at some of the things you saw that she contributed. What did you learn from both from the teacher as well as from the students input into this project?
Harrison Schramm
Isaac was wonderful to work with. I first met him several years ago at a Technology Council discussion for the local high school. And then he sort of had the same question that I did. And some of the things we talked about is that it and this is in the article, the movement has moved away from assessments like tests, and more towards the sort of smack, Smash Mouth football, if you'll forgive the analogy of just the grind of getting the work done. And that is happening simultaneously with California, looking unfavorably upon standardized testing for college admissions. So I am concerned as a professional, that we are moving towards more towards the ability to crank through and do do your work and get it done well and on time, as opposed to what I will call brilliance. Now, let's be clear, brilliance without brilliance without work is just undeveloped talent, and that's not useful. But there is a place for brilliance. It's what American ingenuity runs on. So I I think that one of the things we'll be thinking about a decade from now is did we change towards favoring, grinding it out over flashes of brilliance? And what are the long range consequences for the United States.
John Bailer
So we should talk a little bit about the data that you that you say that you analyzed for this, because you know, the number like observations, the years of data, you talked about these three epochs of data that you encountered, you know, kind of the pre, the hold harmless, and then the post virtual world. And then just a little bit of some of the variables that you had access to? I mean, you talked about some of the comparisons of grades that were received and distribution shifting. But can you tell us a little bit more just about kind of what data Did you get one it looked like.
Harrison Schramm
So we were very careful when we originally went into this, about professional ethics. And I am very sensitive because of other areas of my practice. I'm very sensitive to personally identifiable information. So so we got a bag of grades. And by what I mean by a bag of grades is if if a student was taking six courses during semester, they were represented by six lines of data. And those lines were not joined. So I didn't have any sort of identifier, either their actual student identifier or a scrambled identifier, I simply didn't have that. And I explained to the school that I didn't want that. Because even if the even if the identifier was scrambled, it wouldn't be that hard to reverse engineer and figure out who some of the outlier kids were. And I just didn't want to do that. So so that was the level of the information we had, we had their grade, age, and gender, and the grade received. And so out of all of that we focused on on the four core areas that we talked about in the paper, which were English, science, math, and social studies history, partially because those were the most, those were the things the parents were the most interested in. And also, they were the bins that were the easiest. It was our sense, although we did not test the hypothesis. But it's our sense that kids do pretty well in electives during high school, and that they probably would do still be doing pretty well during COVID as well.
John Bailer
Just as a quick follow up in some of the analyses you did you looked at like the percentage that got certain age categories. But you also looked at the average grades in the classes. And one one sort of complimentary question I was curious about is did you see less variation in the scores as well as shifts in the means?
Harrison Schramm
Yeah, well, since since they got the lift from the bottom, that did tend to tighten up the variance. It tend to it tended to squish it together.
John Bailer
Yeah. I mean, you got a ceiling. So you know, you're gonna get hit hit that. Yeah, yeah. It's on a fixed it's, it's not on an open interval.
Yeah. Yeah. So I was just curious if that's it, you know, and if, if some, you know, I was wondering it could have I could have imagined that there. There could even be some some kids that don't, that don't do well in this new environment, that maybe you end up with some by modality if there was any sort of changes and shifts in the shape of these distributions as well.
Harrison Schramm
Yeah, what we really are interested in doing as a follow on work would be, if we had the, if we had the identifiers, and we had the appropriate compute environment, what I would like to do is start and look backwards at the characteristics of the students, and then look forwards at their performance, and then see if I could find what markers would be useful to identify students who might need more attention. It is my sense that the kids who got failing grades are highly correlated. It's it's not very common that somebody has three days and then an F and one in math just because they didn't get it. But if we had the level of data that was appropriate for that, that would be something that would be very useful to do. I think
Rosemary Pennington
you're listening to stats and stories. And today we're talking with Harrison SRAM of informs about the impact of COVID on grades. Harrison, you said you were gonna take this back to the school district once you were done with this? Did, were they surprised by any of the findings of this work? Um, no, we
Harrison Schramm
presented it, we actually presented it at the school board meeting. And to be honest, having, you know, worked in policy in Washington, DC, and having spent a lot of time at the Pentagon, local school boards, a tough audience. It's a very tough audience. However, this was more or less a good news story. And you know, I don't have to be very qualitative here. But a lift of two tenths of a grade point is is not a particularly good or bad news story. If it was an entire letter grade lift, then that would be concerning. If it was much of a much of a decline that would be concerning, particularly for the seniors who are in the midst of their college application and college acceptance process. So overall, I think it's a good news. story
Richard Campbell
you mentioned, qualitative piece of this. And your, your significance piece reminded me of a number of anecdotal articles I've read from journalist interviewing kids that liked doing school from home on a computer because they felt bullied at school, they felt made fun of they were smart kids, and they felt sort of free. And it would be interesting in a study like this to sort of combine I think, interviews with different kids at some point and their experience with this, I just found that really interesting to see kids that felt freed up from not having the social pressure.
Harrison Schramm
So anecdotally, that is very true. You know, as far as we, we intentionally did not go down the survey methods, whole for this first effort, because I just wanted to stick to the hard data in in something like this, you know, with with when applying a survey, and by the way, I am not a survey specialist. But in something like this The touches on public policy, the people that you get the best responses for are the people at the tails of the distribution, and kind of the median doesn't really care that much. So we intentionally stayed away from that, I would just if you could bear with me for one more minute, the kids who are math inclined and might feel bullied in school, you know, you might get called four letter words like nerd and geek. And I've got news for you, that's going to continue for the rest of your life. But when you grow up, you're going to get called for letter words like boss and paid. So so there is hope it does get better.
Rosemary Pennington
You raise this issue of accessibility when you were talking about what Nora brought to the writing of this piece. And I wonder if you could talk about because it is a very, I mean, significance publishes accessible pieces. And I but I wonder if you could talk about why why having an accessible voice is important for you?
Harrison Schramm
Well, well, it's really important. You know, one of the soap boxes that I've been on informs and in my practice at group W, is that, that there is this temptation for people in analysis, to want to do the work, and finish the work and write the report and then hand the work over to someone else to be the champion of it, the voice that carries it forward, both in organizations implementing change, and also in the popular discussion, read media. And when, when that happens, when we abdicate our, our voice, then we lose control. And I have had one instance, which I'm not going to go through here. But I've had one instance, where I did that and I abdicated my voice and I was horrified by what eventually happened to the work that I had done. So I am increasingly of the opinion that in order for us to be truly impactful as professionals in this age that we live in, that we have to be willing to carry the water ourselves. If we rely on others to do it, we're not going to like, we're not going to like where that takes us speak a
Richard Campbell
little bit to the role of journalism in this because a lot of times for scientists to get their work out there to the public, they're going to have to do it through media, they're going to have to do it through journalists through programs like ours. I mean, we're very much interested in the whole accessibility of data and science. I think that's why john and i got into this and rosemary was joined us eventually.
John Bailer
John Bailer: She wasn't kicking and screaming, either. That was willing.
Richard Campbell
She was that That's right. So what's your experience with having journalists actually translate your work? Because that's what journalists? Do? They have to take science, and they have to tell a story about it. And what's your experience with that?
Harrison Schramm
I have some I have some experience with it, you know, and knowing that you're a journalist, journalists are motivated by making an impact. But the sort of impact that a journalist wants to make is not necessarily the sort of impact that professional statistician wants to make. No one wants to read an article where you say, you know, I studied something that's important to you, and I found out everything's just fine. That's not provocative, and it's not attention grabbing. So I try very intentionally, to, in my own mindset of potentially ometer of how provocative I want to be. And that has to be backed up by the data. It has to be. It has to be fact based, but it's difficult. It's the age that we're in. You can find somebody who would be willing to do something to support just about any point of view you wanted to support. And if Austin sort of the main sequence of the statistical practice, don't chime in, then we're abdicating that voice to others.
John Bailer
So this this leads to a very natural question for me. Which is what kind of press coverage? Did your did your reporting to the school board? Get that it make it into the local papers? Did you get any kind of interest? No,
Harrison Schramm
no. But school board meetings, school board meetings is as close to a town hall is, as we have in the internet age. I mean, I don't I don't it didn't make any local press. But what would the point be? Because everybody that was interested saw it firsthand.
John Bailer
You know, I could have seen a press release coming out of this just talking about I mean, things like you talked about sort of the the DF rights method, that was a pretty interesting story that came out of this. And also, you know, your comment about lift associated with not just the idea of the the hold harmless component, but actually the lift that occurred across, you know, even when you went to distance learning, and the fact that it was the lowest in math, I was, you know, I was looking at this and I was thinking, you know, if I've tried to figure out why was it ordered in the way that science was the highest? And then social studies in English were similar. And then math was the lowest of this. Do you have any kind of thoughts about why that, you know, hypotheses or conjectures about that pattern?
Harrison Schramm
I don't, and Isaac Reubens, a math teacher. So I gave him a lot of grief about this. Results. But, but there's, there's a number of things going on here. Right. And one of the things I was careful to point out to the school board when we were talking is that I'm measuring what with this data, I'm not measuring why I like and there's a lot going on behind the why. And we won't know for a decade pedagogically which which teachers were effective and which ones were not. I certainly, as a mature professional, have a very different opinion on which of my high school teachers were important and impactful to me than I did when I was 18. So and then I also have to point out this is one small school in one small town. My sense is that if you were to repeat this in other parts of the United States, you might get radically different, radically different answers. For example, there's two laptops per person in my home. And not everybody has that. And so kids in areas where the technology isn't assessed as as accessible, where the parents aren't able to give as much, let's call it loving encouragement to get inside to get assignments finished on time, you might see a very different result. I expect I expect the result we had here to be highly localized.
John Bailer
You stole my question there. Harrison, I was gonna ask you about the generalizability. But yeah, so But since you did that, you answered it before I asked. I get I get an extra question. Rosemary. I was curious about the the issue about what would be next. I mean, I, you know, the thing that came to mind when you were just answering that was the idea of there are certainly other predictors that would be important here beyond what you'd look at. There's this issue, like you mentioned about the generalizability, and seeing if it would apply in other circumstances and other districts. So what would be the next study you'd like to conduct in this in this area?
Harrison Schramm
So I think it would be very interesting to see, to see what the results were from different school districts in different parts of California or even the country. And, you know, we have to be very careful about our terms here, seeing a point to lift. I'm not sure. I don't know what success is, right. I don't know that that this was successful, I just know that it was a measure. And again, with California, not smiling on standardized tests, these grade point averages are going to be the be the determinant for kids in the college application process. So you know, we're talking about people's futures in their lives, and the workforce that's going to support all of us when we're retired. So so this is a really important issue, I'd like to do that. I'd also like to take a look at record level data where I did have the records and was able to correlate, because I think it would be. So a statistician is never going to know the kids as well as the teacher does, that's just never going to happen. So the best thing that we can do, particularly if we were going to predict which kids to look at, or which kids to pay attention to in the unfortunately likely event that we may have to go into quarantine again, in the next several years. I would be horrified if we created a machine that told the teachers which kids they should be paying more attention to. But if we created a machine that told the teachers Hey, in addition to your own judgment, you know, please look in on this kid and this kid Oh, yeah, that makes sense. So as algorithm to support a teacher's judgment is probably the best use of this.
John Bailer
No, No, you didn't. That's that. That's interesting. I I can tell you, I've done some analysis of data from my department. And, you know, we've seen that if students that don't do well, that get you know, get a F or W in an intro stat class, a large fraction of them get a D F or W in other classes they're taking in the same term. It's a signal of something broader than just struggling in a single course that's often being manifest.
Richard Campbell
I also like the point that you made in the article under the student perspective, the removal of test taking anxiety in a classroom. And emphasis on other kinds of like note taking, and just discussion and participation. I thought that was powerful in that in that part of the article,
Harrison Schramm
well, well, that was my daughter's contribution. And I'm at cross purposes there because test anxiety is real for kids. And I think it's worse for kids now than when I was growing up. Having said that the world is performance based. I mean, you're talking to somebody who used to fly helicopters off shifts. So, so my perspective is the world is performance based. We don't always like that, but it is.
Rosemary Pennington
That's all the time we have for this episode of stats and stories. Harrison, thank you so much for being here today. Stats and Stories is a partnership between Miami University’s Departments of Statistics, and Media, Journalism and Film, and the American Statistical Association. You can follow us on Twitter, Apple podcasts, or other places you can find podcasts. If you’d like to share your thoughts on the program send your email to statsandstories@miamioh.edu or check us out at statsandstories.net, and be sure to listen for future editions of Stats and Stories, where we discuss the statistics behind the stories and the stories behind the statistics.