Math and Music | Stats + Stories Episode 317 / by Stats Stories

Dr. Kobi Abayomi is the Head of Science for Gumbel Demand Acceleration - a Software as a Service (SaaS) company for digital media. Dr. Abayomi was the first and founding Senior Vice President of Data Science at Warner Music Group (WMG). He has led data science groups at Barnes & Noble Education and Warner Media; as a consultant, he has worked with the United Nations Development Programme, The World Bank, The Innocence Project, and the NYC Department of Education. He also serves on the Data Science Advisory Council at Seton Hall University, where he holds an appointment in the Mathematics & Computer Science Department. He serves on the Advisory Council at the Ivan Allen College at the Georgia Institute of Technology, the Faculty Council at Barnes & Noble Education, and the Advisory Council for Modal Education.

Episode Description

Long after Harry Nilsson said, “one is the loneliest number,” and after Bob Seger sang about feeling like a number, music streaming services are using data to help of discover new music that connects to our frequent plays and preferences. Dr. Kobi Abayomi helps break that all down in this episode of Stats+Stories.   

+Full Transcript

John Bailer
Long after Harry Nielsen wrote that one is the loneliest number, and Bob sang about feeling like a number, music streaming services are using data to help us discover new music that connects to our frequent plays and preferences. Today's Stats and Stories episode will be a conversation about music and data science. I'm John Bailer. Stats and Stories is a production of Miami University's departments of statistics and media, journalism and film, as well as the American Statistical Association. I'm joined in the studio by Rosemary Pennington from the Department of media journalism and film. Our guest today is Dr. Kobi Abayomi, head of science for Gumball Demand Acceleration, a software service company for digital media. Dr. Abayomi was the first and founding Senior Vice President of Data Science at Warner Music Group. He has led data science groups at Barnes and Noble education and Warner media. As a consultant, he has worked with the United Nations Development Programme, the World Bank, the Innocence Project in the New York City Department of Education. He also serves on the Data Science Advisory Council at Seton Hall University where he holds an appointment in the mathematics and computer science department. Kobi, thank you so much for being here today.

Kobi Abayomi
I'm glad to be here. Thank you for having me.

John Bailer
Well, Kobi, I'm tempted to start with me singing a medley of my favorite songs involving numbers. Come on, Rosemary, but I think all would prefer hearing about how you got started working in the mixing zone, where music and data science meet.

Kobi Abayomi
Sure. I use, so let me just back up to, I'm thinking of that movie by Steve Martin, The Jerk. He starts off in the story: I was a poor black child. Which is funny because most Americans know Steve Martin wasn't. You can start the story anywhere. But just I started recently with sort of my escapade or experience in the corporate world I had been outside of academia for about eight years, working in generally what's known as advertising technology, a relatively I'll say, new field that grew because people have smartphones that generate lots of data, that data is useful and predictive for how they are and what things they might like, right. I was working for Warner Media Proper, the large, which is now Warner Brothers Discovery, their merger with the Discovery Networks, monetizing their ad inventory. And so what an inventory is, is space, either time, you know, linear TV, or some space advertising space. But on a digital platform, CNN, for example, right. And so what that job redounded to was figuring out from these reams of data of people's viewing preferences, who was most likely to see what, at what time, and then from that, it gave you just basically a map of affinities over time at this sort of person, that sort of person, that sort of person. And then you'd go to say, hey, you know, people who are more likely to buy Downy say, at this time, and so that science is going on, you know, all the time, at most ad agencies and ad servers, Google, for example, matching available ad space, which is either time or time and physical space on well, physical digital physical space on some platform, and the likelihood of a particular sort of person audience segment viewing it. From there, I was contacted by Warner Music, for a couple of reasons, not to the job. One, I love music. Two, it looked like I was really staying at the same company, because it was still Warner property with the hip separated back in the 90s, I believe. And then three, I really, really strongly believed that a deeper understanding of audio segmentation, from the music production side, will allow people to make better music, right? Like if you really could tune into people's affinities, the music that would come out, could be more nuanced, less bad, whatever. So I took that job and I worked that job for several years and learned a lot and that's how I came to be in this field.

Rosemary Pennington
So what exactly were you doing?

Kobi Abayomi
Much of the work in the beginning, was really just organizing, codifying the data that were/ are available to, I'll say owners of digital content. One of the beautiful things about digitalization is the way in which people can consume content. I don't just mean music, I mean a movie, as well as an on demand video on demand movie, that when you do it digitally, you are leaving a transaction level sort of fingerprint, right? Kobi, who has a subscription to Netflix, you know, watching Star Wars, five o'clock in the afternoon, Kobi was scripted to Netflix, and the next day at five o'clock is watching, I don't know, Starship Troopers or whatever. It's another sort of science fiction movie. And from this really granular, rich data, which has information about my identity. And then information about my demand preferences, you can build up a scheme to see, you know, what sort of person likes what sort of thing, and that's, that's the engine of it, right? Once you have that map, that data available and codified in a useful way, you can move on also to other problems, prediction problems, optimization problems. In the beginning, I say, he was not the only person working in data science to say this, he spent a lot of time just organizing things like whoa, like, what's, what's here, open up the box. And the tools are everywhere. I spend time organizing that, as that continues, you know, you're trying at the same time to do science on top of what you have, as you're pushing it forward and making it better organized and more useful. So we were spitting out models, predicting demand, or predicting demand as sort of a top line level like without sort of nuance and audience preference or segmentation. And then as we got more audience information, predicting demand with that nuance, right, this sort of person is listening to house music at nine o'clock at night, which is different from another person who's listening to house music at nine o'clock in the morning. And that's useful for how you market push produce, right curate buy, decide to buy right through different music.

John Bailer
So I'm curious about when you talk about some of this data that some of the data that you're using, are there other ancillary sources that you're bringing in? You know, kind of? Is there a lot of connection once you have, once you've identified that Kobi is watching Star Trek and Starship Troopers? I think you and I might have a similar playlist, movies here. But once you have that, you know, so if Kobi is watching it, then and John's watching similar kinds of movies that maybe slightly different times, how do you how do you start? What other information do you bring into this kind of conversation?

Kobi Abayomi
Well, that's a great question. I'm not sure. I say two things. One, the sort of data described as your gold standard, right, the transaction level data. And I'll say this, most rights holders are people who have created some sort of media, which is being consumed digitally, are two or three steps like thermoclines, away from the generation of that data, right? Like me, I make up a TV show, but you know, Cox Communications is the one that shows it, and has a box, top of the TV that can tell who actually saw it. And depending upon the relationship between the curation, creation of the media, and, you know, the downstream consumption of it, what goes back and forth as data is almost never, you know, that sort of clean or deterministic. So, one way in which one can use this is a standard, I will say standard, but this is, you know, something that people do is to basically do probabilistic imputation. Here's some data that I've seen on these consumption records, how else can I augment this to things that I know that are deterministic, and then do a join across something that's deterministic, which is, hey, this stuff was actually consumed. And the other side of it, who consumed it? Simple example could be, I know, the gender distribution across many cities. And I use information from the gender distribution, because I have no other information about the eventual consumption to build a dataset, which gives me a notion of west of Denver, gender distribution of consumption. And I do it by joining data across the cities, and then, you know, augmenting that data with the gender distribution in each city, right? In the absence of gender information at each transaction level. So that's one of the ways in which we would use other datasets. And then another way we would use other data is so many things, I think, you know, can influence particular sorts of consumption, right? I remember being younger, this is sort of really just sort of a gestalt sort of thing, watching some documentary about music during the 30s. How there were a lot of light songs, yes, go the others, the sunny side of the street, things like that right to, to get people to be able to modulate their moods and things like that. So ambient effects. Ambient information has an effect on media consumption as well.

Rosemary Pennington
I'm always surprised I listened. I love music. And I listen to a lot of it across multiple streaming platforms. And I have pretty eclectic tastes like, I love stuff like Leadbelly, but then also like Nirvana and Taylor Swift and MC. Oh, God, mega round. And if you know that rapper, he's a nerd-core rapper who raps about like, video game music, and like, you know, anime and stuff. So like, I'm always surprised when I open up for you and some of these streaming services, and find, because I'm really picky about music, and I don't like people buying it for me. But when I open those things, I'm like, oh, most of the time, like 95% of the time, like, I'm jamming to that playlist that they've created. And it feels like I created it myself. And so like, how does the data that you are creating, gathering sort of help and form things like that, right, because I'm assuming it is some like views or inputs, that sort of coming together to sort of form how that algorithm shapes those playlists, right? Just curious, I use Apple Music most often.

Kobi Abayomi
My favorite, I love the best, richest, personalized algorithm recommendations. So let me say a couple of things. And after that, the first is, so that DSP is and by DSP, that digital streaming providers, Apple Music, Spotify, Deezer, Amazon, they're gonna have the richest take on who you are, right? Because you subscribe to their service, you buy stuff to it, they turned off IDFA. And you can only enter using it through their phone, right? So they have the richest take on who you are. Apple Music. One example. And so I can't verify that this is actually happening. But um, I would get up on Saturday morning, start off Apple Music and start on a you know, he would start on its progression of recommended songs, take the phone, go get in the car, start driving on the road. Now the phone is aware that I'm moving, and it would change slightly, the sort of music that I'm listening to, I am hard pressed to not believe that they weren't aware of the motion of the phone and how that changes people's music preferences as well. So that information, there's a recommendation engine. And one of the things is if you're on the service for a long time, it gets a really good pick up on who you are. But it's also using his information from other people who have similar listening habits. And likely, although I can't verify for sure, where they've got some encoding mechanism for the music itself, right? Beyond just easily consumed or not. But this thing sounds like this, they're able to take the sounds, break it up into some sort of embedding that's useful for prediction. So what are they trying to optimize? They're trying to keep you on the service, right? And when the billing comes around, and, and you look at your account, you have to be like, I don't want to cancel this is so good. And that's a different thing to optimize than, say a music label would be trying to optimize or a particular sort of artists, right? Because once you listen to their music, or their content, there's a tension between the optimization that goes on at the user interface level, you know, Apple Music and new and UMG and what they would prefer, you listen to.

John Bailer
You're listening to Stats and Stories. Our guest today is Kobi Abayomi, head of science for Gumball Data Acceleration. Kobi, you describe your work in another place I was reading that said that you're trying to understand the quote, intersection of demand and supply curves for musicians, artists and music listeners. And this seems like it follows up a little bit about the comment you were just making. But can you help us think about what demand and supply curves mean for those different groups?

Kobi Abayomi
Sure. Let me answer it this way. When there's a talk I give, and I go through an example of a sample of a song, the meter sissy stretches 70s I wouldn't say funk to it's really sort of a unique band but they're sort of syncopated style and behind a lot of different music and sampling. Fast forward to 2006 or seven, six baby and that was a single one like to him wander by an artist called Emery where they're basically lifting the entire T of the breaking says he stuck to that song and song but it was an amazing single did really well. When you look for the antecedents of that sound, what you find is that it's in sort of mainline New Orleans music, you look further back, it's in Samba, it's in Brazilian music. And so the supposition, you know that I haven't, and then there's people who, who understand ethnomusicology in a way, much richer than what I do is that you're making things that are appealing to vibrations, nor have logical memory, and nostalgia, cultural currents that, you know, are old and instantiated. And so, in that way, our desire to hear certain things, our response, or joy or affinity for them, when we hear them, the demand side of music, is very old, very rich, and can be well understood. Now, on the supply side of a music company, more or less these days, especially with digitalization, where, you know, four fifths of the money coming in through the door is via the streaming services, you're getting paid for the volumetric fraction of listening at a particular period of time, right? In, more or less, it's not visible, but all subscription fees, say for a company. In January, I started using label X. How much of the listing by all the listeners who paid all the subscription fees was on content that I own the rights to music label x, divide that by the overall listening, that's how much money I get in January. So it's very different from I got to excite somebody to go to the music store, buy something once and then I forget about it. Right. So if you've got to make money in this new regime, you have to be able to sort of ride these changes in musical preference, and be able to predict them, identify them, and predict them very differently from second artists based models. Well, you know, I found Taylor Swift, and that she's very successful. So I mean, you'll say anything like that. But I found these artists and I'm going to mine them, right and chase after their super fans. And what you can see in the data, as this streaming becomes more ubiquitous, and the underlying audience becomes more heterogeneous, less concentrated, people start to go towards pocket, the familiarity, things they listen to. And my notion is these are these demand currents that are older and nostalgic and sort of more sort of well worn. And if you study the data closely, my supposition is that you can predict what's coming up next, by seeing changes in listening patterns, by codifying the sound as information and seeing which sounds are becoming popular, and where that leaves.

Rosemary Pennington
Yeah, I wonder, given the work you're doing, you know, like one of the things I think a lot of artists are kind of searching for in this environment. Although I will say there seems to be a swing back to people buying like standalone albums, right vinyl is about this huge increase. But it seems like you know, you have to find that one hit that's going to go viral, right? It's going to pick up on social media, people are going to spin it out in the work that you're doing, or have done around music and sort of what people are doing with it. How did you find anything in the data that sort of suggests like if a song has this kind of a rhythm, it's going to be something that gets picked up and gets popular or like these, how did you do any of that kind of work? Or is that something you're interested in doing down the road?

Kobi Abayomi
Sure. For sure. So I'll say a couple of things. And just in general, about the sort of virality that I think there's that appreciation for how many preconditions need to be set or how much dosage goes on before the right before the appearance of veracity. So to take for example, 2020 Summer 2020 That song, by Fleetwood Mac, there's a fellow riding his, his skateboard, sort of dog days of summer, drinking cranberry juice playing with Fleetwood Mac song very tranquil Fleetwood Mac song, a lot of people started listening to a new so called viral moment. What do you think about the conditions that say that it's hot dog in the summer, in the middle of the pandemic? There's a lot of sort of anxiety, there's a need to search for this sort of tranquility. And I think many other songs would have worked and elicited the same sort of demand response. That's the U is just like probability overall, right? Like I can't tell you exactly what's going to happen. But I can give you sort of the curvature of what does until you sort of be there in that space, right? Hey, stand near the mean of that Gaussian distribution, likely something's going to fall on your head, right? And don't stand out near the extreme values. I think the way that people in the music industry and this is something that, you know, I think should change is where people have an interest in sort of believing in extreme values, right? And believe that it's driven by the fact that hey, I found the black swan Haha, I'm the guy who finds the Black Swan, and stuff like that. Another example is, I'd love to talk about this example. So James Ambrose, known to the world as Rick James, Buffalo, New York, with a lot of bands, a lot of different acts, before he happened upon, I'm going to do a funk act that's going to have a new wave sort of sound in a documentary about him. He's arguably his producer, about, what's that? What's that song? The MC Hammer sample can't touch the Super Freak. And it's produced like the song is good, we haven't very well, it has got that sort of thick bumpy baseline, doo, doo doo doo, doo doo doo doo. And Rick James is arguing for sort of this 145 chord change with a synthesizer that sounds similar to DeBose with it, which had just been out, I think, a year prior. So they're going back and forth with him and as producers, like, Why do I? Why do we need that he's like that, because that'll get the white audience, right. Like, I want this new wave sound in this song. That's who I'm trying to appeal to, he wins, you know, and you turn that song on in a second. And you recognize the baseline, when the chorus is committed, does that change? It's immediately recognizable. So people who produce music, you know, and I just sort of talk about what are aware of how to sort of prime the pump, and what to put into the song to elicit a certain sort of response from certain audiences.

Rosemary Pennington
Your first example made me think of Tracy Chapman and sort of the moment she had at the Grammys recently, with Fast Car coming sort of back, and how that was such an important song when it came out. And then thinking about this sort of cultural moment where we're all very tired, everything is very difficult. And this story sort of, like, the hope within hope that is kind of shattered sort of comes back to, you know, to popularity through white artists. And then she again, on that stage on that beautiful moment just made me think of like, again, that idea of, you can't sort of predict what inputs are going to help something become popular.

Kobi Abayomi
You know, what, uh, first off, what a beautiful two. Second one, what a beautiful album that was a creation. By her I think of that song I think of there was a sort of one hit sort of thing that went viral so called that Richmond sort of tune last summer, if you can remember that. Remember that fellow's name, I O. And I think the music companies started him to awaken to it like Hey, what are we what are we done the country? And, you know, the country now and that's not my understanding of the country right? And what I thought it was out ends up being outlaw country, Merle Haggard, and stuff like that. Like that's not where country Guy Yeah, well, you probably had to erase some of this, my opinions about. But I say take Beyonce friends, and I haven't listened to her whole thing because it hasn't been released. But that first song was just a so-called country song wave. That's not a country song. And I was having a discussion with somebody online. And they were like, oh, you know how radical this and bla bla bla bla. So there's 1971 Pointer Sisters, I had a song, which was an actual real country song, which won a Grammy first black egg. In my opinion, there's a lot to be mined in so called country and I think you can see in the data, and there's a yearning people's ears, as you get again, you get more population going back to stuff that's older, because they're not firing newer representations of it in that genre, so called genre. And I think there's room for my opinion, authentic artists, to gain some lifting in that genre. Just because it's sort of lifeless right now, in my opinion, right. And sort of craving for, for that richness to come back. I mean, John Denver, Bill Withers, those were country artists, right? Could they get on a country radio station?

John Bailer
So, you know, I'm interested in what's been the most surprising thing that you've learned, while working with some of the data that you've you've been seeing and with music, and what kind of did you not expect at all given the analyses of such data?

Kobi Abayomi
The most surprising thing is something that we sort of stumbled upon last summer. I was working with a grad student at Georgia Tech in the music school as part of the computational media program. And we were trying to look for alternate models for predicting music demand over time, right. So again, under this new regime of music consumption, it's not just hey, I go to the store and buy it, it's I listen to it sometimes to it sometimes, and I sort of tails off, and you see these sort of convex curves of demand arise from when a song is released. So we found a way to go from just the sound. So take the sound and process it in a certain way to this, you know, convex curve of demand over time, that was really very precise. And I'm still working on the mathematics behind it, but it works so well. And what it makes me think is the way in which we join music and the encoding for that when it's proper, and I think they'll think these things are well understood. And one of the ways in which you can encode music is via the MEL spectrogram, which is to find basically the frequency, the height of the frequency distributions that resonate with the melodic scale that is really very predictive over large populations. And I thought that it was pretty amazing that music has a determinism about it, with respect to consumption. When you think about it, being composed of different parts is being able to be deconstructed as a data object. So that was pretty amazing to me. And so we're still working on that.

Rosemary Pennington
What do you think's the next big exciting thing that is coming down the pike when it comes to data science and music?

Kobi Abayomi
Well, you know, there's so called AI, and all of these large language models. I read a quote by somebody the other day, it was an economist. And he was saying, oh, you know, AI is not really AI. It's really just statistical correlation and pattern recognition. Okay, well, there's a lot behind that, right? The word statistics contains multitudes. So there's correlation. So just pattern recognition. And then there's the richness of the methodology that people have uncovered comprehensively over the last 10 years. It's really not fair to just call it pattern recognition. And the typical correlation, I think, and we use some of these tools in the work that I do to process music. So, for example, there are so called AI tools that when trained correctly, allow you to clean up a sound or extract the vocals from it. These were tasks, you know, 10-15 years ago, very difficult tasks to do signal separation and things like that. And so the models have become so much better. And it makes it easier to record an album, right, maybe not in a perfect environment, and then clean it up. So it sounds, you know, good and listening, I live in a bubble. On the other side of the so-called AI proposition. I think people are most worried about creation, an importation of new content with an asterisk by it from extant media, without attribution. Right. And without compensation. I've heard crazy things such as, here's my new generative model, that's the sort of zero hot like, I don't need any training data whatsoever, you'll think that does sound unbelievable. But um, I think and I've seen some papers already, where if you stick in training data, and analyze only the output of the model that you're able to resolve and score, you know, what the model was, was using, or the fraction or volume of training data from different sources. So I think there's got to be some work there so people can get compensated right? And just like in all things I you know, I'm sure people were concerned when the theremin was created, like, Oh, my God. People need to know how to use their fingers. Good. Just go like this. I made music.

John Bailer
Well, I'm afraid that's all the time we have for this episode of Stats and Stories. Kobi, thank you so much for joining us today. Stats and Stories is a partnership between Miami University’s Departments of Statistics, and Media, Journalism and Film, and the American Statistical Association. You can follow us on Twitter, Apple podcasts, or other places you can find podcasts. If you’d like to share your thoughts on the program send your email to statsandstories@miamioh.edu or check us out at statsandstories.net, and be sure to listen for future editions of Stats and Stories, where we discuss the statistics behind the stories and the stories behind the statistics.