Game Economist Cast
What does the new wave of open economies mean for monetization? Will negative externalities overcome cosmetics economies in the long run? What exactly does a game economist do?
Game Economist Cast is a roundtable discussion of the latest developments in mobile, HD, and crypto games through a bunch of people figuring it out using the economic tool kit.
Game Economist Cast
E16: Should Match-3 Players Choose Their Difficulty? (w/Dr.Julian Runge)
You won’t want to miss this episode of Game Economist Cast.
Guest! Finally! Dr.Julian Runge is here to add some much-needed seasoning to the regular crew, bringing the takes. We discuss the role of art, science, and academia, analytic organization structures, the role of flow in retention, the role of sales versus personalization, and the best theories of difficulty.
We cover a lot of PUBLICLY published research in this episode (most by Dr.Runge!) that most industry venters haven’t seen!
[1] Getting ML-Based App Personalization Right: The Engagement Engineering Framework
[2] Price Promotions for “Freemium” App Monetization
[3] Why Free-to-Play Apps Can Ignore the Old Rules About Cutting Prices
[4] Quantity discounts on a virtual good: The results of a massive pricing experiment at King
Grandma season is ahead.
Speaker 2:Grandma season. Grandma season.
Speaker 1:Yeah, they call it grandma season Because all the grandma's like the more temperate weather.
Speaker 2:That's how they all come out during the day, and stuff.
Speaker 1:Yeah, the early bird specials are wild.
Speaker 3:Let's start with utility. I don't understand what it even means.
Speaker 4:Everybody has some kind of utils in their head that they're calibrated.
Speaker 3:There's hardly anything that hasn't been used for money. In fact, there may be a fundamental problem in modeling when we want to model.
Speaker 1:Episode 16, game Economist cast and we have a guest for the first time. I think we promised this in one of the first episodes at some point. It's finally happening. Someone has decided to walk the plank on this shit show and it's none other than Julian Dr. Julian Dr Runge is here. This is your second podcast. I was listening to the one you did with Eric Seyfert on mobile dev memo. He cleared out all the kinks for us. It was a really fun lesson, super informative. We'll have a link to it in the show notes and I'll let you introduce yourself, because you have a wide ranging career between academia, tech and gaming.
Speaker 3:I'm excited to be here where I start. So I'm an economist by training, which is, I think, nice Since the game Economist podcast. Then, after graduating in economics, I actually joined Vuga, which is a Berlin based game developer. I was the first analyst at the company and grew with the company. I ended up building the analytics and data science team there, and then I've always wanted to do a PhD. So I started my PhD in qualitative marketing, also working with data sets that I had access to and from my work in gaming, and I went over to Stanford for research, visited and then got drawn into Silicon Valley and the Stanford environment and decided to stay there and spend the second half of my PhD over there. I've enjoyed Facebook as a researcher and rewriting papers and getting to work with these amazing data trolls that Facebook has, and now I'm spending more than a year as a visiting scholar at Duke University, and now I'm faculty at Northeastern University and principal scientist at Game Data Pros.
Speaker 1:You have gone through tech, you've gone through games and you've gone through academia. You've transversed, I would say, the holy trinity or the axis of evil, depending on where you're standing. Is there something that academics don't understand about games, and the inverse as well? Is there something about game industry people that they don't understand about academics, seeing as you work with both groups, wow interesting questions.
Speaker 3:It depends what part of academia we talk about, right? So I think I actually first, while I was working at VUGA, I spun up some academic collaborations more with computer scientists who were active in game research, and we did a bunch of predictive analytics work and the people there understood gaming pretty well. In my opinion, many of them were gamers themselves. When we then talk about economics and marketing academia, it can be a bit of a different story. I think there's a certain preconception that games are different from other empirical settings, right Like that. Something is different about them, and just the degree of immersion and the degree to which things are being made up, invented and virtual goods, for example. I feel like there's still so much research to be done that haven't really found its way into more mainstream marketing and economics outlets.
Speaker 1:Is there something that tech people don't understand about games? You worked at Facebook. There's always this lingering question why can't tech companies seem to get games right? Do you have any clues for us? I?
Speaker 3:think you know that probably better than me. So, to be fair, my time at Facebook was completely non-game related, right. So I worked on the advertising research team, which is part of the Marketing Science R&D org, so much more traditional kind of marketing research, if you so will. I do actually think there was a number of people at Facebook that also met that got games really well, and I enjoyed a lot to be able to take the early models of Oculus for tests because that was possible as an employee and enjoyed the games a lot as well.
Speaker 3:Honestly, I don't think I can comment super well on like why maybe game traditional tech companies might not be great at getting into gaming. I think overall and this is why gaming is such an interesting space, also from an economic point of view it's just really hard to produce hits in a predictable, rationalizable manner. Right, there's a lot of stuff that happens, that seems serendipitous or whatever, but it's not necessarily possible to know exactly that this is going to work in the way it then does. And games are just and it's also why I love gaming as an industry they're so creative and there's this component of unpredictability that I think it's just really hard to tackle from a large company vantage point generally, and I think that might also be where some of the larger tech companies have necessarily succeeded in the way that they had hoped.
Speaker 1:We often talk about games as being where art meets science and where science meets art. Do you think games need more science and less art? Oh wow, you're going right in.
Speaker 4:Wow, the intro fell Jesus.
Speaker 3:Christ.
Speaker 2:Yeah, this is good. I want the hard-hitting question.
Speaker 3:So I think, sure, I think games are craft, maybe. So I think both art and science are very conducive to getting good at it and better at it. But yeah, I wouldn't say that I believe that games need more science per se. I wouldn't say that I'm convinced of that. I think the beauty there is that science and art get to meet, or that creation and analysis get to meet, and that's always like a spot where, if you get that, you can have a lot of success with it.
Speaker 3:But very often these conversations go a little bit wrong, right, because this idea of all many very analytically-minded people they actually want to exert control over the environment, they want to have models, they want to have data, they want to measure everything and then there's more creatively-minded people who tend to rely more on introspection and maybe also qualitative techniques, and I feel like these conversations go a wrong way too often and I wish there would be more of a connecting issue there. But I think throughout my time working in game analytics, I've always tried to invest into making these conversations more fruitful, if you so will. I don't know, that's probably not a great answer to your question, but I think it's a very difficult question. I'd be also curious to hear actually what you think.
Speaker 2:So from what I've seen, like you mentioned, is it too much art or too much science? From what I've seen, in big companies or in even small mobile game companies, things tend to be overmeasured, over-datafied. They can't seem to think outside the box. They're like okay, I have to predict next year's sales based on last year's sales and I have to predict what genres would be popular based on what's already been popular. And I think those groups could probably benefit a little more from more art. But on the flip side there's all these indie breakout hits that never progressed past, for example Vampire Survivors. That was pure art. Some guy just thought this was fun and he made it. And then that just got ate up by the fast moving copycats who took it and optimized it and iterated on it a lot faster. And I think there's a component of that where the more novel mechanics, the novel game designs don't scientifically iterate and improve as quickly as they could. But in general I would say, especially where the industry is trending and the big blockbuster titles seem to be overmeasured and over-tuned.
Speaker 4:Phil to address this hittiness of the games industry compared to tech industry more generally.
Speaker 4:I think it goes back to the not immersive gameplay, but emergent gameplay we talked about, I think, last episode or maybe two episodes ago, where it's just hard to know what the combination of the combinatorics is going to look like the metagame. There's a whole bunch of different ways that people are going to interact with one another and it's the social aspect. So you think about Facebook, Twitter. There's pretty much only one action, maybe two or three actions you can take in that game of Twitter repost, requote or reply or create your own. So there's this very limited strategy space. In video games, the strategy space is enormous compared to the strategy space of one of those more typical tech platforms, which I think is what leads to this hittiness, where it's a lot harder to say whether something's going to work. I've heard pitches where I'm like that's a terrible pitch, a terrible idea, and then somebody else who's way smarter than me would be like I get it, this is amazing, it's okay, You're way smarter than I am, so you probably see some magic here that.
Speaker 1:I'm not going to see. I go back to the drawing line between HD and mobile, and mobile has always been, I would say, more science-based, and they've had to be right. When you're going out and you're spending millions of dollars on Facebook and you're getting LTV information and you have to make very real purchase decisions with immediately facing consequences that you can measure, I think it forces you to be more empirical, and so they've had to adopt a more empirical approach and that's why I think they've always been big into things like AB testing and I know, Julian, you're working at Game Data Pros now, which, to my knowledge, is a little bit more work with HD people. But HD is still primarily a part of the industry that's driven by creative directors rather than product managers, and to me it always comes down to the battle between creative directors and product managers. So I would argue that on the mobile side, they have at this point at the margin again, I would take this all at the margin they have a little too much of the empirical side and when I've seen companies pull back on that a little bit and add a little bit more creativity, they see it measured and I think we see it when it comes to performance average, and it took scopely and scopely is my primary example here. It took scopely a long time to understand that if someone who is a creative director understands the player in a really personal and captivating way, that they can build marketing creative that has measurable impact, or they can build a game that has measurable impact in terms of reaching users, and there's no better example of that than Monopoly Go, which was driven by a GM that came out of Zynga who is an incredibly creative human being and he had a great post on Neil Long's mobile website and I'll link out to this where they really went back to the core of what the monopoly IP was and they completely Changed what the game was. So, rather than being a clash royale style game, they ended up moving more and towards a coin master IP. But again it was. It was related to what monopoly was and and that was an insight I don't think you would have gotten from a product manager that was focused on optimizing KPIs or changing the color of buttons. It was someone who had to understand the creative piece of it and, again, like there was a measurable impact on that. So that, to me, is really powerful.
Speaker 1:On the HD side. They don't run an AB, they can't AB test their way out of a box. Just it's really disappointing. There's a complete lack of understanding of what science is, and actually I'd even pin that on mobile. They don't understand what science is either. They seem to think that data is science rather than the scientific method, which, again, you can do without empiricism, or at least empiricism in the way we traditionally think of it. So to me, at the margin, I think the HD side needs a little bit more Empiricism and I think the mobile side needs a little bit more creativity, and again, I think those will make both products better, if that makes any sense.
Speaker 3:Yeah, that's super interesting and I think that's a pretty good observation. It also reminds me back to just how fundamentally Publishing models in the games industry have shifted right, and I think the fact that mobile may be more data-driven and, by virtue of that, a little more Ventific and at least embrace experiments embrace experimentation and measurements more strongly. There's the autopilot freemium free to play games where you, I feel like the development cycles are much shorter, the budget's time to be smaller. You bring this to market more quickly means you get market data much more quickly that you can then iterate on, whereas in traditional triple-a games it wasn't about freemium. You built the whole game before you shipped it, and I feel that might even play into it.
Speaker 1:So it's a pretty, pretty good observation, I think so in your career, you keep boomerating, boomeranging, back to games. It seems like you start in tack, you start in e-commerce. You did some academia, you went into games, then you went into academia again, then you went into tech and now you're back to games. Why do you keep boomeranging back to games? Quick question when did I do e-commerce? Didn't you work at that German Zalando or something? That e-commerce website wasn't a really part of your career.
Speaker 3:Oh, well, that was. That was just like what. I was a student. I did some Consulting work there.
Speaker 1:I'm just gonna help me to make the boomerang point though.
Speaker 3:Oh well, yeah, Anyways, yeah, we can talk about that. So I wouldn't say it's the lot of boomeranging is. I knew I want to do a PhD and what I always and I think this is even part of my research approach I believe in the pretty close dialogue with a marketplace and embedding yourself with it. So it really struck me as odd to do a PhD in a business-related discipline without having worked and experienced this firsthand, and In fact I did have a few offers for companies to join back then. But I very deliberately chose gaming because of the creative component of it and also the economics component of it. Right, this idea of managing Massive virtual worlds that have their own economies and are played and enjoyed by millions of players just was I was like as a recent graduate, that struck me as like most fun environments could possibly hope to do data science and analytics I'm, and so I would less think about it as boomerang is more to me personally. It's like I did this condition to try industry full-time right after graduating and then, slowly Since then, I've been working my way into academia.
Speaker 3:To be honest, this is how I would think about it, and I've kept pretty strong links with with industry, for sure. And then, once again, I think it's part of my research agenda and method to be in a close dialogue with the marketplace and also embedded in it, to be able to run field experiments. And Frankly, this is also where I hope to push the needle, and maybe you see that in some of the papers I've been able to publish using game data, like they are about field experiments. They're about large-scale intervention in the field and bringing this to econ and marketing researching, also highlighting what an amazing study ground this actually is for human behavior and economic decision-making.
Speaker 3:Yeah, so this is. I personally think it's less about boomerang. I think it has a pretty. Maybe it's a little deep down and not super visible, but there's a pretty big trajectory into academia. The question is, of course and this goes back to what you're all asked like academia is shaped by very strong convictions and preconceptions, and my profile is a bit unorthodox, right like this, deep dabbling with industry. It's not super easy to land a full-time tenure-track position, to be honest, but that would be ultimately my goal, yeah where I want to head.
Speaker 2:Speaking of those papers with game data, how would you invite, what advice would you give to academics who want to get access to game data or tech data to do papers? And, inversely, if you're a game company, what kinds of collaborations should you be looking for to share your data?
Speaker 3:Wow, tough one. Though I will say that my personal experiences since I've I'm less centrally embedded in the game company, for example, running good field experiments is actually really hard if you don't have that in. And this is something where I was also Overoptimistic, I think, in how much I might be able to keep doing that Once my ties to industry become maybe a little weaker, at least in terms of like really running a data science organization full-time. So I will say that a lot of the work I've been able to publish is because I did like first spend the time in industry and then transition Because I was able to just run design Field experiments that you otherwise can barely stand up because they involve so much either technical effort you need to deal with also organizational politics in that need to overcome.
Speaker 3:So I think one paper we also might want to discuss a little bit today is the one on price promotions and premium app Monetization. It's published in quantitative marketing and economics, which is one of the premier quantitative marketing journals. By the way, that paper just won the best paper award from the journal. That's its man, nice dig, witting price. Yeah, I'm super excited about that. Honestly, like this is the first, I think this is a bit personal, but this may be the first professional accomplishment that actually brought tears to my eyes. That's how much, that's how happy I was about that award. Yes, with that paper, though, for example, we kept the long-term holdout group for nine months, while it was a pretty evident that conversion and also revenue are significantly up in the treatment Groups, and such an experiment you can only run if you really have some sway inside the organization or if at least as strong executive conviction that this is a key finding that the company wants to know about.
Speaker 1:Do you feel like of the result of that paper in your relationships are self-perpetuating. Now that you've had some wins, does it make it easier to set up the next one?
Speaker 3:Yes, that's what I mean, like I maybe thought so, but actually to a lesser extent that I maybe hope for, and it really is about you need to understand the institutional setting super well, be connected deeply to it, I feel, to run powerful field experiments, and so at Facebook, for example, I worked on this a lot, like on academic collaboration, how we can spin that up, and it's just you need to spend the time together and the thing is like, at the end of the day, the value, functions or the goal sets between academia and industry are just different, right, and so finding these overlaps is actually, yeah, it's harder than I thought it would be.
Speaker 1:So. You've worked in senior leadership positions as an analytics leader. Could you tell me a little bit about how you think we should do game analytics Organizationally?
Speaker 3:this is clearly a topic that was on my mind a lot as we scale things at Vuga, and Vuga was first following the fully so Vuga is that Berlin based game developer I started my career at, and we first had a fully decentralized approach where we had a rather Small centralized data infrastructure team, if you so will, taking care of well, bring the data into the databases and the gate format etc. And building tools. And then we had game analysts Embedded in the game teams, in the large game teams, reporting into the product leads, though that's pretty powerful because, like that can make sure that analytics happens close to the actual product questions, right? Almost going back a little bit to this, like Academia industry connection, like how do you actually keep data science analytics connected to the product issue was, though it's pretty hard to do data science initiatives that have value across different games and products and to have a career For game analysts that do not want to become product managers but really wanted to develop into data scientists and go deeper on the technical side. And then I went to strata, which strata is like a big data conference used to happen in San Jose I think it's now having happening in other places as well and I went there in 20 when was this?
Speaker 3:2014, I believe and Met some people from into it in that that company doing.
Speaker 3:What do they do? Again? Text advice or something I forgot, but anyways, we had some interesting conversations and they had moved to a hybrid model where you basically have a small centralized analytics and data science team and then you have embedded analysts in the product teams, but their circulation between, like, the embedded analysts come also back to the central team, get to spend time there and then move into another product team and so, like that, you can both realize Data science initiatives that work across different products of games and you can have a career path for data scientists while also staying close to the product. And so I push that we got to adopt that model and we did, and I think we did pretty successfully and to me today, this was the best way I've seen game analytics and data science work in industry. Honest, but I'd be quite curious to hear what you think, especially also in regards to game economics, because, to be honest, like that's yet another function that is quite analytical, but maybe it's still not exactly the same thing.
Speaker 2:On, that rotating analyst. No, at riot people tended to be embedded on certain teams and if you wrote switch teams, it wasn't there, wasn't like a rotating schedule, you just. It was like a position change. And I found myself pretty fortunate to be embedded on League of Legends and Very close to the product for a long time to develop like deep I product expertise and knowledge, to have a strong intuition and you need that strong intuition to come up with Hypothesis and test them effectively. But I think there are other folks who got unlucky got put on a team I don't know a marketing team. They got spun down after six months or something and they never really found their footing and a lot of those people Bounce out of the company pretty quickly. For me personally, I really liked being deeply embedded on one product, but I think it can be pretty hit or miss.
Speaker 4:Julian, I'm curious and these guys can correct me if I'm wrong but, I feel like the game economics field is split into two different camps that like more Analytics side of things and then the more design side of things. So you think about a class, like I think about Bill Grosso, the CEO of he's very much, who I picture as one of the founding fathers of the analytics side of game economics. How can we use a be testing? How can we use machine learning? How can we use these different statistical tools that economists are equipped with in order to improve the game? But then you've also got the other style, which is going to be your, your Eve online it always good mits, and and then the Castro Novas of the world, who are more on the design side of the economy.
Speaker 4:How do we create a fun, engaging economy? How do we make sure that there's enough of the good stuff being given to the players so that they keep, they stick around? Do you, do you see that distinction and if you do, how do you manage that? So I see I picture somebody coming into the, into this space, and being a little confused, not knowing what it is that they're Going after. How do you determine whether you want to go to the this analytics side or this design side and are they the same thing? Is it really just a difference between are you going to go be a PM, are you going to be go be a data scientist? And if that makes no sense at all, please I can clarify and I think it doesn't make sense.
Speaker 3:It's yeah, you're definitely asking the tough questions, which is good, don't get me wrong, and also really glad to call Bill Grosso, founding father, father of the analytical side of game, economics. That.
Speaker 4:He's humbling out that we can do it.
Speaker 3:Yeah, no, it's actually. What I'm just thinking is like maybe game economists could actually be that bridging function Right between the very analytical, data sciencey stuff and the more design aspects of the economies which, to be fair, yeah, I guess looking at games as economies is already a somewhat analytical account of what games are right, very much actually. Maybe again, and game economists could be that bridging function. I think, my mind at least. I'm not sure that there's like this golden pathway to how to do this. I think there's really strong and capable people on both sides and some lean more into the Own of build. I want to build also using your mistakes and introspection and smart ideas and the more analytical mindsets, working from models and large-scale data and so on. And yeah, I don't know, I don't think I have, at least not yet, the perspective like how to do that perfectly. I think there's very many valid ways to get to pretty impactful Ways of doing this, but I like this idea of maybe seeing game economics as connecting tissue, actually between the very analytical and the very dreams.
Speaker 1:I think you nailed it. I think you nailed, julian and I. That, to me, is what I see as a large-scale vision for game economists, is that we should be, that we should understand the analytical piece, because the game economy designers do not understand the empirical piece and I don't mean that in a bad way, that's just not their training. I would love for them to have that empirical piece, but they aren't going to understand what a regression does, discontinue design is. They're not going to understand what a AB test is. They generally don't have sequel skills and again, I would love for that to be the case and they are brilliant individuals with so many design talents. They usually don't have the empirical piece and I would say economists need to pick up the design piece, they need to pick up the game part of the game economist piece. And I, when I've seen People get into this industry that have the econ piece first Myself, everyone in this podcast, right, I think we started with the econ piece and then we picked up the game piece and we came more and more and raptured in oh wow, I now understand the empirical piece, but now I understand the design piece. I understand how these two things function with one another and when you can think in those two different worlds, you realize they're not two different worlds. It's just one continuous stream of thoughts and I think you get insights you wouldn't have otherwise gotten, and I hope we can continue to Be that.
Speaker 1:I would say the earlier question you mentioned about organizing Analytics functions. I think you're right in terms of having a career path for analysts. I would say I've never seen an analyst want to stay an analyst. Maybe it's just a bitter experience, as I had, but the two paths analysts always took were either to be a data scientist, because that's the sexy position. It's a sexy position that has external validity, that gets you a job at a tech company and Means that you don't have to be a data monkey.
Speaker 1:No one wants to be out cranking sequel dashboards or Tableau dashboards, whatever it may be. Or they became product managers Because they want to have control over the product, or they wanted to be able to have a vision for the product and they want to see it actually end up on the roadmap. You sometimes can feel helpless as an analyst to do that. You can feel pigeonholed into how a product manager wants to use you, and so those were always the two paths I saw. I think one of the challenges we have is how to make analytics in and of itself being an analyst, not being a data Scientist and not being a product manager Satisfying and rewarding and influential on its own right and I would say a lot of organizations. I don't know if they figured out how to do that yet.
Speaker 3:Okay, I agree very much. I agree very much. She wanted to, since we talked a lot about these Connecting points or also maybe the points where they can be in conflict between analytical creatives. I think one thing that, phil, you and I have talked about in a bunch and I think we also have somewhat divergent perspectives is Personization and how that can be done in games right, hold up.
Speaker 1:There's one thing I want to get I want to get out in front of this intro piece beforehand. So one of the things I find really interesting about you, julian, is that you have so many published people, so many published pieces In games, you guys you have so many published pieces in games going back a number of years.
Speaker 1:You were actually one of the first people to pop up on my saved Google scholar search for game economics with your churn prediction and jelly splash all those years ago and you helped me realize those shit. Someone is thinking about these questions in a live way. But if we were to think about the thread throughout your research, is there a takeaway, an overall takeaway from your research? If there's only one take, I know each paper has its own thesis, its own results, but is there a takeaway that you would give to the game industry if they were trying to look over your research? Is there something that they could, they could, summarize and take away? Is there a grand arc?
Speaker 3:Wow another tough one. I don't think I have that. There's a lot of application areas that that work Well, like you said, churn prediction and customer last time really prediction with really important inputs into both product and marketing. But, in honestly, like we're almost coming back to the same topic of, like analytics meeting creative endeavors you really get the most impactful if you'd start having deep conversations between game designers and the more analytical data science side of it. And I think, actually the first paper ever maybe I published in this page space, which is churn prediction for high-value players and chemical social games.
Speaker 3:I was 2014 at the conference on intelligence and games I think it's fault and we were actually nominated for the best paper award and where the second got in the second place there.
Speaker 3:And the thing is there we showed that you can do churn prediction for casual social games, which at that point wasn't obvious yet because it mostly had been done for the more Involved games with deep economies and deep immersive gameplay.
Speaker 3:And the experiments we ran, though, right we? We used that churn prediction system we built to run an experiment giving free currency to players before they churn, ahead of their churn, and that didn't do anything. It neither solved and served to re-engage them nor to make them Spend any more money, or with the only thing we saw is that our predictive system work in sense they were still more likely to click on what we sent to them through notifications, but from there sparked an internal effort that I can only maybe talk about some extent, like to spend more time between the data science team and game designers to think about what I actually the important aspect of the game that we should leverage to do something when people seem to be losing interest in it, and so I think my overarching thing is yeah, talk to each other, work into Disciplinarity and bring creation and analytics together. This is how you become successful.
Speaker 1:What? One more question before we get into the meat and potatoes Could you tell me what is something that has a Prior, that you've had, that's moved most significantly, based on your time in the industry, whether it be a surprising result or just something that you've intuitively gathered, I think what so far has surprised me the most is the success of freemium, the pricing model.
Speaker 3:Freemium or Free to play, what it's called in gaming. The success of that model has just done me no, and if you look at what it's done to the games industry now, more than half of all revenue is comes in from mobile games and it's only possible because of freemium, because we have that has enabled. This new publishing model has brought in completely new Audiences of players that before would maybe play board games or puzzle little word crosswords in the newspapers, and they suddenly started playing on their mobile phones, on their smart, and together with that, actually, there was recently an interesting Thread on Twitter where somebody said they don't know any gamers and then there was a little bit of discussion. How many gamers? A little bit of discussion?
Speaker 1:Oh, my, it was Benedict evidence man. All, all three of us went to town on that guy. Well, I wouldn't say good town.
Speaker 3:But I think this is so interesting, though this it's not known yet what a social phenomenon, cultural phenomenon gaming is. We have no more than three billion people in the world playing games Ditto games and that's just. That's the thing that I find most stunning how well Freedom has worked for games and how well smartphone proliferation has worked for gaming companies and how this sector is just growing and growing, and I think it's gonna be pretty interesting to see what we're where that's headed. I'm almost feeling like maybe we'll see a bit of a reversion back from premium to more premium or into publishing models over time. I wouldn't be super surprised by that, for example. Yeah, so far, this is the key thing that I signed, just like stunning still finds stunning sometimes.
Speaker 3:Do you have an explanation for it? I don't know that I do, but I think it just starts to show that humans love to play, right? I think it's something that is so built into us and if you give us the opportunity to engage and play, we do it, and I think it's also an amazing means to bring people together. There's a game I've worked on where we had people that met in the game and got married and realized what an amazing outcome is that, and so I think, yeah, games can have this function right. They give you a nice structure to engage on with other people, with yourselves, and humans just love to play.
Speaker 1:Chris or Eric, do you have any more questions, because we're grilling them.
Speaker 2:I'm excited to dig into the article. I got a bunch of questions there.
Speaker 4:Yeah, I feel like we definitely grilled them enough.
Speaker 2:Let's hop into the main topic Got a selection of good things on sale stranger.
Speaker 1:All right, the basis for our conversation today is a piece that was published on Mobile Dev Memo by Julian. We will have a link to it in the show notes. It is called Getting ML-Based App Personalization Engagement Engineering Framework. Julian, do you mind giving us an overview or a summary of the piece?
Speaker 3:Yeah, absolutely, and I might actually start with a piece straight away, but with a little bit more of an intro into how I think about personalization in apps and games. So I think there's two broad approaches, maybe three. The first one is bottom up. So if you think about ways that you can actually make personalization based on algorithms or smart segmentations, on data work, you could start by asking yourself what do my players and users spend most time doing? And then, in a puzzle game, this might be playing puzzles. Right In social media, it's looking at their new speed or some sort of feed and then from there ask yourself what is the most relevant preference? Heterogeneity, if you so will. In that, right In a puzzle game, it likely is like how skilled they are and how much they want to progress over how much time they put into it. In a social media news feed, it's going to be about who do they like to connect and other things, and so from there, you can come up with perspectives or at least identify opportunities in my experience, for where personalization might make most sense. Now, what I try to do with that piece, that we want to talk about, this and what I call the engagement engineering framework, is more of a top down approach. So if you don't maybe not know, you do not know yet what your users will spend most time on, or you just generally need some inspiration. Where can people start from?
Speaker 3:And what I do in the piece is I lean into what I believe is the most holistic theory of human motivation that's out there.
Speaker 3:It's a part of self determination theory by Edward Desi and Ryan Desi and Ryan and within that theory there's a sub theory called basic needs theory, which holds that to engage sustainably and with intrinsic motivation in an activity, humans need relatedness, competence and autonomy. You need to give them these three senses to make them engage with an activity like sustainably, and I think that's a useful perspective to start thinking about. How do I actually create good long term engagement with the app or game I'm offering and why I think this is so important? Or maybe to even give some cases, there was this at some point, facebook transitioned to what they called meaningful social interactions as one of the key goals for the platform, and that is the relatedness component. If you want to have people engage with your app or content sustainably, you want to give them a sense of relatedness that actually is meaningful. That can be the starting point for us to talk about this a little bit.
Speaker 1:So one of the things that you talk about, one of the examples that you start with, is the example of a puzzle game and personalizing a puzzle game and, from what I gather, the personalization of a puzzle game would get users closer to a flow state. Is that accurate?
Speaker 3:Yes, you can get more larger Shelby users into states that are similar to flow state.
Speaker 1:Yes, and could you describe what a flow state is?
Speaker 3:We are here at an interesting point between psychology and economics, between things that are very qualitative, maybe, and things that tend to be very tied down in economics. More so, flow state, broadly speaking, goes back to Mihai Tsigchen-Chun-Yai, who is like a psychologist, I think at UCLA, if I'm not mistaken, not fully short. He first came up with the idea of flow, and that is a state of intrinsic motivation where you have basically the right spot for yourself between challenge and boredom, and when you manage to put people into the spot, they are intrinsically motivated and staying intrinsically motivated, or do so more so than they would in the counter sexual world, where they're not there.
Speaker 1:And so that is the first part of the self termination theory. If I'm correct, that is the competence piece.
Speaker 3:Not fully, so it's more like right, it is a great question Because, once again, what I'm trying to do with this piece is give a top down perspective, like somewhat generically, across different apps and games.
Speaker 3:How could you think about ways that personization can make sense?
Speaker 3:And then, within the competence dimension there I think I'm mentioning the flow state aspect because it's something that many people have mentioned when it comes to game engagement and I think it's also it's just yeah, it serves to exemplify. You want to have the right level of challenge for an individual player and players, especially casual game players, can be very heterogeneous in their willingness to put up with difficult other levels. There are people who play your game in between, while commuting, while they have a moment free, and they probably first want to experience some positive, rewarding experience before they're getting challenged by this hard knuckle level that they got to be. On the other hand, you have players who play your game every day and are super hope and they want the challenges. Possibly and my point, I guess, is like data can allow us to at least identify to some extent like what kind of player you have here and then help them enjoy your game more by giving them the right experience or the right amount of reward compared to the effort they have to invest.
Speaker 4:Is the idea that a person who's in flow state, a customer who's in the flow state, is at their highest likelihood of spending, or something like that. Is that why you think it's some people? I've always talked about flow state, especially in games, but it's a totally different objective function. If you're talking about a premium game compared to a free to play game, I guess it could be just taken for granted that yeah, if you're in a flow state you're more likely to spend. But I wonder, has that been tested? I don't know if there's any research on that.
Speaker 3:It's a great question To me that's a secondary effect or an effect of secondary interest. At this stage I'm not even talking yet about how do the different pieces of this framework of playbook I'm trying to give come together. I wouldn't be aware of any concrete research that tells us if people at flow states I'm unlikely to spend. That's also not really the thing I'm trying to accomplish. So what I tried to do is show how can you engage players and users, or how can you think of ways that you use personalization specifically to engage players and users more. If and how you monetize that, then I think, is to me, as I said, like a secondary question and could be ad based, could be based on in their purchases.
Speaker 3:I'm sure there's a lot of interesting questions, but I also want to mention at this stage is like the autonomy dimension, because that's the other piece I dive into in that article into a little bit.
Speaker 3:Like the important thing is like why we do this is not about bringing people to flow states to then manipulate them into buying something. You still want to make sure that they get to make their own deliberate choices about what they want to do, and I think this is actually the crucial component for long term engagement Right. So I think what another thing that's super important to keep in mind when it comes to personalization, and which we as economists actually have the perspective is short versus long term consequences, and it's really easy to do things that are short term engagement maximizing or revenue maximizing, but then don't actually make people stick around with your product and build a trustful relationship to your brand, and so I think that's the other thing to keep in mind. I think that that's where the autonomy dimension is a powerful, because you don't want to use personalization systems, algorithmic systems, in a way that manipulates people or takes away from the deliberate free choice.
Speaker 2:I think this is a super interesting point that, like dynamic difficulty, I think has gotten a lot more popular in games these days where, to your point, like you, want to engineer the difficulty to match the user skill level and also the user's preference for the challenge. And one thing I think particularly interesting in this, as well as across the other topics covered, is that this is a problem that, prior to the advent of data and machine learning, designers basically were trying to solve themselves, often with opt in processes, opt in the game you know you're not going to get your difficulty level at the beginning of the game or games with self adjusting difficulty. A lot of RPGs are difficult, but then if you spend a bunch of time doing side quests, the game gets easier. And yeah, I think this data driven approach, especially in mobile games where there's less freedom of movement, less autonomy than the game, difficulty to match the players without having to create all this optional content, all these side quests.
Speaker 2:I think Breath of the Wild is probably the standard example in my brain where there's two main activities there's exploring and there's combat. And the way to get better at combat sets, increase your health, increase your armor and what have you, is actually through exploring, and so for players who are bad at combat and favor exploration, naturally they'll get better at combat by just doing the other thing, and I think you see the flip side in other games, like Skyrim, where by doing the same activity over and over, you get better at that one activity, but then it leads people down these rabbit holes of specialization where, like all I do is combat, I don't do any exploration, because that's where my stats are. I think it's interesting that these things are systems that game designers have tried to solve without data, and with data and becomes a lot more easy.
Speaker 3:And then the thing is if you have great design, you're going to facilitate a lot of this yourself selection, and I think that's particularly true for triple a premium games. But in the premium model and mobile gaming, I do actually think that the the main thing of the player base is just it's much larger because you have these casual players around as well that first of all, deserve to have an good game experience but also can generate revenue for your company, and I think this is where these kinds of systems can have even more value probably than that might be able to under triple a premium.
Speaker 1:So here be my question to both you, eric and Julian, because I think, eric, you make a really interesting point, which is that for a very long time and in many cases we still do the single player games you can choose your difficulty. You can choose it explicitly if you are a player. So my question to you, julian, and you, eric, is like why don't, why isn't that a solution here? Why do we have to build segmentation tools and offer players different difficulty? Why can't we just have them select whatever difficulty they want?
Speaker 2:The first one is that you show them easy, medium, hard. People feel patronized. Right, they're like, oh, I'm not a chump, I'm going to do the hard one. People want to feel like they're doing something hard, even if the the thing they're doing is actually easy. I think this is why games like Dark Souls are so polarizing because they are hard and they they're totally unforgiving.
Speaker 2:The other is that it can be fourth wall breaking or magic circle breaking whatever terminology you want to use. Where you're trying to overcome a challenge, in the back of your mind you're like, oh, I could just turn down the difficulty level rather than like actually getting better and accomplishing the thing I'm trying to do, and that kind of saps some of the fulfillment, the satisfaction of actually overcoming the challenge.
Speaker 3:That and I agree with what Eric said. And, by the way, phil, I believe you're truly excited about it, because this is probably not going to be visible in the podcast, but I could just see Phil almost bouncing up and down the screen here, yeah, so I think the other thing is that, again, like with casual games especially and people who don't have much time, it's just each step where you have to, where you have them, deliberate and think about hey, what do I want this to be? Is going to take away from your audience and you're going to have to drop off and see. I think this is all the way.
Speaker 3:The autonomy component is important because you want to hit the right trade off there. Right, you want to make sure people get to make deliberate choices where they really should and need to, but you also want to minimize the cognitive load for them in terms of onboarding with the game, getting into the game, and I think this is where some level of dynamic or personalized difficulty can just be super powerful, because you take this it's almost think about you having your booking a five star luxury experience. You want these people to read your mind. You do not want to be constantly be asked how would you want this? Would you want this and just to really make it a premium, weeks on like great experience? I think that's why it's so helpful to do that in a background, in an appropriate fashion.
Speaker 2:So one question here is like how do you separate the players skill level from their preference for challenge? So, for example, someone might lose the first level of your puzzle game, but how do you know if this person is bad and they want the game to be super easy, or this player is bad and they want the game to be hard?
Speaker 3:That's really tough. I think I wouldn't say that that I that I've seen this be solved in a very great fashion. I've built a bunch of trend prediction models and skill can surface as an important predictor, but it doesn't necessarily. Often and it's often very much is like the past was the best predictors for the future and the current engagement levels are just engagement levels of the future. I have a like when I built these systems, I do tend to take dimensions of skill into account as well, but this is really far away from a really clean and a bit both framing or solution at this stage and a lot of work to be done to really tie that down.
Speaker 1:Engineering relatedness. This is the second section. You talk about using personalization to match players with similar skill levels and engagement styles in social environments, and it looks like you have a field experiment on this. You were able to do this. You could increase. It seems like messages spent, rounds played and a little bit on the revenue side. Can you talk a little bit more about what you mean when you talk about relatedness and how you might be matching players in some way?
Speaker 3:Yeah, absolutely. I think. First of all, this is just a broad framework to start these conversations. There is far from any notion of like conclusive like. This is how you have to do everything here, right? Just to pre qualify that the field experiment you mentioned or alluded to is actually published as the papers called algorithmic assortment of matching on a digital social medium published in information systems research, which is the leading journal in information systems, the academic field of information.
Speaker 1:We'll put a link to it in the show notes as well.
Speaker 3:Yeah, great, and what we show there is and I think it's important to, as I mentioned earlier, to stay aware of the distinction between short and long term so what we show there is that, if you have players come into your game, into your free to play social game, by matching players with high propensity to engage in communities, or existing teams in the game that have high propensity to engage, you can produce more aggregate output in terms of engagement, messages sent and also revenue generated from the player base. What you also see, though, is that you get a bit of a bifurcation in terms of the communities and you could also call that reduction equity if you so will because you start having these super engaged communities that become ever more engaged, but you also have other communities that just fizzle out or drop off the map a little bit, and in a social game, admittedly, that's fine. It might even be the right way to do it, because people who don't engage with that game will find something that's probably more what they want to play in other social media, like dating apps or professional networking websites. This might actually be an issue with real world consequences. That would be non desirable and non equitable. The point is in terms of engineering relatedness by bringing players together, and this is honestly already implemented in a lot of the PVP matching systems. Right?
Speaker 3:Elo based matching, for example, is, in the end, the notion of assortativity on a different index, so we use the index.
Speaker 3:Usually when you do a sort of matching, you also need to agree on an index you use to create the assortative matches, and the index we use is propensity to engage. This is a free to play casual social game and we just match people together that I'm more likely to engage in. Either based ranking. The index would be skill, which is another index that is very commonly used in gaming, and the idea is just I see many social, casual social games rely on user self selection into social groups. I've played clash of plans and a lot of other games a lot, and usually it was like I had to search a little bit until I would find the right plans for myself, and the point here is by you can use machine learning, or at least the data sets you have, to help find players their way into the right community for themselves in a more efficient manner, again reducing that notion of cognitive load and making sure they get the right social environment more quickly.
Speaker 1:I think this one is a give me. We run experiments on this in forex and we found the exact same results. And sometimes it's just the basic shit like matching people who have the same language together. Of course, those things are going to increase engagement. Now I would certainly disagree with the relatedness as a potential explanatory power here to me. Let's go to monetization first. Let me go to monetization first before we crack into this one. Engineering monetization. Now, this one doesn't seem to be connected to the self determination theory that you were talking about. Maybe I'm wrong about that one, but you talk about using personalized offers and using personalization to give those personalized offers. Of course I'll edit this to make sense. Can you talk to me a little bit more about what you mean by that?
Speaker 3:Absolutely. So I think this does go back to a question or topic Chris brought up earlier right, would players in the flow state also monetize better? And again, I think at this stage, at least to me, that's a second order effect, like I'm really just trying to give a first starting point here, how we can help people build apps and games that are fun for the players and that make the companies revenue to be able to survive and with monetization, absolutely that's not part of the self determination theory, motivational piece of this, but it's crucial, right. So companies to survive, you got to make money somehow and targeted offers are just super crucial for you to play games. I think we're all aware of this and I think the paper the price promotions and premium app monetization paper I have we talked about earlier brings that point at home very much like how important that actually is to generating revenue in these settings.
Speaker 1:I'll edit this to kick this to the end only because I think that's the one where we have the most divergent views. Let's go on to engineering autonomy, if that's all right. Can you talk a little bit about what you mean when you say engineering autonomy as part of the self determination theory in games?
Speaker 3:Yeah, absolutely so. This is mostly a reminder that in setting up your systems, your personization systems that strive to maximize renail itness and competence on, strive to maximize engagement along these dimensions, keep in mind that you don't unduly manipulate the players into doing stuff that they maybe don't want. So one way that you can and I think I talk about that in the article is like if you have a system that adapts difficulty of the game for players and personalizes it, don't use that in the various way. Right. And the way to do that is, like you can say and that's actually what we did or I've done in the past when building these systems for the top 50% of most engaged players are top 70% we don't touch the game. They need to be able to compete in like on the same base level, otherwise it's unfair.
Speaker 3:But for the players who are a little less engaged, with less skill, you can almost think about it in a bit of an affirmative action frame For these players, we want to help them, to onboard them with a full game experience, because for some reason they have a hard time learning this game or they just don't have the time to devote to the game, to really onboard with it, and that's how we support them, to get them there. But then as soon as they hit like the skill levels and engagement levels of these core game players, so to say, we don't do any of that stuff anymore. And so the the autonomy dimension is really there to, after you build this system, that is pretty powerful to have another round of checks and balances to make sure it doesn't do stuff that is not how you want to treat your players or your customer base.
Speaker 4:So it's interesting you talked about affirmative action in this marginal person because, like affirmative action, one of the big kind of issues with it, at least in the economic literature, is that it's supposed to only impact the marginal person, is supposed to help them get into a better school or get a better job to try to make up for the socioeconomic disadvantage they had growing up, for example. Now there are some kind of unintended consequences. If you take like a statistical approach, the problem is, if you were the beneficiary of the policy, meaning you were marginal and you benefited from the affirmative action policy, it's difficult or impossible really to differentiate as a firm, for example, between someone who was marginal and somebody who wasn't marginal, somebody who was impacted by a policy and somebody who wasn't. So you almost end up with just this giant label. The question is twofold.
Speaker 4:Do we see anything like that in this context when we are focusing on this marginal person, which I really like this idea of let's focus on the person who's most likely to, who's on the edge, who's either going to churn out or they're going to keep playing. Let's focus on that person, that person who first of all has the highest return or, I guess, relative return to focusing on their decision. So is there a way for us to identify? I guess the first question is how do we identify that person, that marginal person? And then the second question is are there any unintended consequences of that person moving on into the world where we have either a selection issue or some sort of statistical discrimination?
Speaker 3:It makes me think that I guess in this setting it's a very dynamic definition of margin, and so maybe any comparisons to real world affirmative action are not fully appropriate. Maybe we should avoid them. But yeah, the idea is actually, the firm was pretty well able to observe, though, who was treated, exposed to what treatment. That's actually easy to do in the setting, and I guess the definition of marginal in a game setting is very endogenous, and if somebody would lose interest they might end up in that category and be onboarded again. I don't know, that might not be a great answer to your question.
Speaker 4:The question is are there any unintended consequences? So I'm picturing like there are many different ways you can change the quality of the game in order to make that person stay. Now at least what my interpretation? Something I read from the article was like you don't want to. You don't want to manipulate people. You want to give people autonomy. You don't necessarily want to make someone spend money that they otherwise wouldn't have wanted to spend. You don't want to trick people. I'm looking at this from this marginal person who maybe the counterfactual as they would have not spent money and then they end up spending money. Is that okay in under your framework? Under this, this like triple framework of competency or relatedness, is that something that you're okay with? Like this person having converted, because the model makes a lot of sense under the UA framework or the UA KPIs, but does it make sense under the revenue KPIs? They could.
Speaker 3:I guess in the end, that's the point of the autonomy dimension right to spend the time to make sure this doesn't have unintended consequences in this way, and I think it's okay if. Okay, actually, let's stick with the example we had. So in this case, I think it's okay if you see the revenue increases around people who before that before the system maybe, or without the system weren't able to onboard with the system, weren't able to develop the skills to engage with it, and you see an increase in converging and an increase in the roughly middle of your revenue distribution. I think that's an okay outcome, an desirable outcome.
Speaker 3:If you'd see, however and this is, I think, why this fact that we only know the system mostly, we try to make sure that it doesn't touch the base game for the most engaged players, the top 50 or 70%, that avoids that. But say you'd have a real the media system that tweaks the difficulty for everybody, what could happen, especially in loot box based games right, is that you start having, you start seeing increases along the very top end of the distribution, meaning your super hope players already spend a lot of money, keep spending and spend more. That's probably a little more problematic as an outcome. So I think to make sure you don't have unintended consequences from systems like this, you need to closely monitor your data, also being a dialogue with your player base, because if you do have these unintended consequences, most likely gamers will start to bring it up in foreign style voices, concerns about it and yeah. So make sure you do your due diligence on that to make sure you don't get these unintended consequences.
Speaker 4:Phil, I have a question for you. Do you think that it's okay to say let's say, we've got a game, we've got a user base, we make a change that makes somebody spend more money.
Speaker 1:Yes.
Speaker 4:Now I would. If I were in your, if I were putting myself in Phil's shoes, I would say it doesn't matter who you are, it doesn't matter how you did it if they spent more money. All it means is that we delivered them a product that they believed was worth spending the money on. So I'm curious. I want to get Phil's take on this, because I feel like he has a strong take, but I could be honest.
Speaker 1:So is your argument that, like price discrimination is wrong, or like people find out of price discrimination and they'll hate us?
Speaker 4:So I don't want to. I don't want to do injustice to Julian, but I feel like the this, this engineering autonomy, insinuates that there's something wrong with manipulating your players into spending more money. But I think that there's a question to be had or to be asked here Is there such thing as manipulating?
Speaker 1:people into spending their own money. I think manipulation is emotionally charged word. Of course we're optimizing products for engagement and LTV. Of course that's what we're doing. Of course that's what designers are doing. Of course that's what movie directors are doing. We're always trying to optimize for LTV and optimize for engagement.
Speaker 1:People use this word manipulation. This is why I hate a lot of the psychology literature because they try to use the scientism approach where they think that if they explain things with increasingly scientific terms, that it removes autonomy for people. If you explain that when I, when I do X, endorphins are released, my brain expects it, or whatever it may be, then it's this automatic bodily response and that you don't have autonomy over. And this, to me, is always what psychology does. When they try to pull the wool over you is okay. If we continue to explain things in these scientific terms, we remove autonomy for move autonomy. Then that provides a reason for third party actors to intervene and make changes so that they, we can control the better nature of our demons. And that's why I disagree with a lot of psychology, because I do think they start, they're starting to get into moral reasoning and I think a lot of those arguments are fallacious and they apply to almost every product and service. So yeah, like I disagree with the word manipulation, but yeah, of course we're optimizing.
Speaker 2:I don't mean does that? There is such a thing as bad spending? Right, if you promise a product but you deliver something much worse. Right?
Speaker 4:And they pay. So uncertainty plays a role here, and I think that's why loot boxes were so poorly received.
Speaker 1:Again, I don't necessarily think they were poorly received. I think they've been in games forever and I think they were magic the gathering forever. If you want to know my basic rules for monetization don't lie, don't cheat and don't steal. I think those are pretty good moral acts that have survived the test of time and again. They're revenue maximizing maxims. Of course there are cheaters and frauds that exist, but they don't last long. They get bad reviews. There's word of mouth. They remove from the app store. There's a lot of things that remove cheats and frauds, but those, to me, are the basic rules of the game Don't lie, don't cheat and don't steal. I know they're doing great in crypto. Oh man, if we had, if this was a two hour podcast. That's what we get to in the last hour. There's so much more that we could talk about.
Speaker 1:There's a more general thing.
Speaker 2:I think you mentioned your article, julian, which is to focus on the inclusivity over intensity or an econ jargon, focus on the extensive margin rather than the intensive margin. And you give some great examples where it can go bad, like political polarization or statistical discrimination or addiction. But on the flip side, I think there's a lot of games that have succeeded by being extremely niche and having a hardcore audience. I think Dark Souls comes to mind, or I even think about chess, which is a very difficult game that a few people are very into, and I think, yeah, are there criteria where we can say, hey, sometimes it is okay to focus on the intensive margin, but these are situations where you shouldn't, yeah, I think that's a great point and I think the examples you just mentioned probably, yeah, speak to that one to one.
Speaker 3:I think this is again like is more about if you build casual games right, that you want to appeal to a broad market, or you produce consumer ads, that you want to work for a broad market, in the end, like you want to increase your total addressable market, which you can probably do by using personalization in a smart way. And that's not to say that there isn't types of games where maybe you even don't want to use personalization systems of this sort. But I do think when you want to address a broad market, you want to give consideration at least to inclusion, equity, diversity, to make sure you give an opportunity to a broad as possible player or user based to onboard with your experience, and I think, especially in cases of apps or maybe also games that really support human well being outside of the core in-app experience. So, again, this is less likely in games, more likely in dating websites or health apps or other apps like that I think it's crucial to give a strong consideration to these aspects.
Speaker 1:Let's get into this personalization stuff, this promo discounting model. You have a paper about it that you mentioned at the top of the hour that you ran it for a very long period of time. I am extremely skeptical of personalized offers and just offers in general, because the times that I've seen them used mostly just amount to a sale. Congratulations, you ran a sale, you increased conversion. What else is new? Demand curve slope downward. We have the paper that Steve Levitt published at King Games where they ran this on three million players. They did quantity discounting. You'd go into Candy Crush and they would offer the amount of gold bars. They would increase the amount of gold bars at a given price point by as much as 80% to find that it was completely price elastic by changing the quantity discounting. They weren't able to even increase revenue 1%, and not only that, they were not able to even move non-payers to payers through quantity discounting alone by even 1%. It was actually shocking, when you look at this table, that they could not get a single non-payer to become a payer through this quantity discounting.
Speaker 1:You have a paper that you mentioned at the top of the hour which provides a different point of view here, but I guess I'm curious, because you've focused a lot on the sale argument. You've heard this before, that a lot of times. Personalization and bundles can just amount to sales and you push back against it. What's the story? What's going on here? Why do your results differ from this King paper? Yeah, I love that question.
Speaker 3:Are there different mechanics?
Speaker 1:here Are there different genres. There's a lot to unpack here.
Speaker 3:Yeah, there is, and you'll need to dig a little deep into the paper to understand what's going on, but it's actually been quite obvious. What I want to mention at this stage is that because I assume a lot of listeners won't have the time to read full academic papers Danford Insights actually did an interview with us as the authors of that paper and published a very accessible and nicely written piece called why Free to Play Apps Can Ignore the Old Rules About Price Cutting. We should also maybe link that in the comments. I think it's a very accessible digestion of that article.
Speaker 3:So now, coming to the Leavitt paper, it's a great paper, but let's think about what the treatment is that they use. They work with quantity discounts and there's a figure in the paper that shows actually right, I think they have four different quantity discount schemes that they test and let me ask you this when do you see the treatment starting? It is only for packs that have more than 100 gold bars. So they actually say in the paper what share of their payers not players, but payers do you think is buying packs of 100 gold bars or more, or how much revenue does the game make on those?
Speaker 1:Very few. Most of the revenues made at 20 plus dollars.
Speaker 3:Meaning the treatment that they have in the paper actually isn't effective or doesn't impact the largest part of the player base. So there you go why you wouldn't get strong effects on the outcome metrics.
Speaker 1:So is your understanding of the empirical method that they only did the quantity discounting on the lowest price skew, that they did not do it on their highest, higher price skew?
Speaker 3:No, it's actually the other way around. They only do it on the high price skews and that's why all the players where you actually can increase your expenses margin, where you can have additional conversion and more revenue generation, they're not impacted by this treatment.
Speaker 1:So when I'm looking at this chart, I'm seeing quantity discounting across all of their skews. They bend the curve for all of their skews. If you look at the dots, those are the skews and you can see the gold bars at each of them. So if we look at 1000 hard currency price package, yeah, 1000.
Speaker 3:Yeah, exactly. So very large gold bar packages, but if you look at below 100 gold bars, the dots are all on the same line. You can see the smallest skews. That are the first dots there, the first six dots or so. Yes they're all on the same line.
Speaker 1:The treatment or like.
Speaker 3:Yes, and so the treatment only starts kicking in for larger package sizes Because it's about quantity discount. So like you want to manipulate that, but you have to be then really aware in your analysis what you're actually doing here. You should probably only look at high pairs or something and maybe then you find effects. But if you look at the whole game player base, yeah, you're not going to see strong effects from this. But that doesn't tell us anything about, like, how effective our targeted offer is, how effective our startup packs in driving monetization in your games. And the thing is also startup packs. They are advertised through popups to players who would otherwise maybe not even enter your game store, and that's the other thing. With these quantity discounts. People would only even see them if they go into the game score and then scroll past the first small packages. So it's a great paper, it's super interesting, relevant, but it just doesn't. The treatment is so different from what we do in our paper that maybe it's easy to think they're similar, but the treatments are actually super different.
Speaker 4:Let's say, we data scientists and the PMs were not in constant communication. They didn't have the pipelines in place.
Speaker 1:So let's say we accept that they didn't vary the things that they should have. The things that they should have varied were actually the things that actually had the most purchase uptake. Right, because they're not varying the skews that you think they should be varying.
Speaker 3:That's all I'm saying. We need to be clear on the research question. So what they look into is like the effect of quantity discount schemes, and they changed quantity discounts for the larger packs in that game. What we look into is price promotions in free to play apps, and that's just, it's too very. I get that from a. If you look at it from a substantive or managerial point of view, they may seem similar, but on a research level these are two very different research questions.
Speaker 1:So I would push back against that. So, on one hand, the first thing that you're identifying between the King paper and some of the work that you've done is the UX effect. Right, they're just surfacing offers into your face and so there's just going to be more impressions from the offer than there might otherwise be. Totally seems reasonable to me that there could be some sort of UX effect here. So that to me would be one potential effect. Holding all this constant, what happens when I just put a pop up in front of you that you otherwise would have to self select into? Totally could be an effect there.
Speaker 1:But when I look at the second part, the reason I would argue or at least a lot of what the economic theory might lead me to, at least the neoclassical economic theory is, at the end of the day you're just pricing progression, we're to flatten most of the games, and again there's a lot of difference here in terms of doing discounting and promotional offers for a cosmetic based game, for something like a match game. So let's take match games, because that's what you've written a lot of your work on. I think it's a very interesting optimization box. But when I think about match games, what you're always doing is you're pricing a unit of progression, a unit of me being able to complete a level. So, whether it be extra moves or whether it be boosters, all those things are, to me are just a win probability, that's all they are.
Speaker 1:And so if you make a starter bundle, you can put different boosters in it, but to me, all it represents is win probability. Same thing with gold bars in Candy Crush. All you're doing is you're just giving me win probability and you're just pricing it at a given level. And there might be UX shortcuts. So you might create a promotional bundle that says has some gold bars and has, let's say, some boosters in it, some in round boosters, like a smashy thing that clears a row, or cannon that clears a column, but at the end of the day, that can all just be represented in win probability. And so that's why I think of these things as just the same, and I would argue that, like the UX effect and just flattening all these things, it all just becomes win probability, and my economic theory would lead me to say, okay, you're just discounting win probability in different ways. Do you think that's fair?
Speaker 3:It's an interesting point of view, but I don't. It still runs in the same issue to me as, like if you only treat like a small subset of your player base with this change in unit rewarding or unit like whatever you want to call it exactly, you're just not going to get affected. And so if you want to like, have strong facts, you need to roll this out and make sure it's accessible and impact a lot of people.
Speaker 1:So let me put it this way what is the theory of personalization, right? What is the theory of bundling or discounting when it comes to personalization? That would make this profitable in the long run for a firm Doing personalization all targeted offers. Do you think there's a difference between those two things?
Speaker 3:Well, obviously that's the point of my whole framework right that we need to think beyond just offer targeting or personalization when we want to really make sure we use machine learning based or data informed personalization effectively in casual social games or generally consumer apps. So, yeah, I think there's a big difference and the treatments matter a lot, and only if you have an effective treatment that can be effective on a large part of your player base can you actually think about personalizing that. Because if you personally something that almost nobody sees yet, and surely that's not going to do that much.
Speaker 4:Julian, I see the paper, or I see the concept, as like machine learning as a way to optimize personalization. It doesn't sound like you're saying that, though.
Speaker 3:No, it's less about optimization at this stage. So, as I said earlier, I think to me it's really about starting to give some sort of top down framework that you can use to start thinking about ways to personalize the experience. I wouldn't dare to say anything about optimization really at this stage. Yet it's more, they agree, if you want to stand out, if you want to start using either heuristic segmentation or a real machine learning model in the way that impacts how well your game does, just giving a starting point for that I think.
Speaker 2:To jump in on the personalized offers points, league of Legends use personalized offers very effectively and I think it comes down to heterogeneity. Julian, you mentioned the whole point of personalization is different people at different preferences, and the more heterogeneous it is, the more you can personalize. And in League of Legends there's 150 characters but most people only play one, maybe five or 10 at a time, and if you severely discount content for the characters they don't play, whereas you don't discount the content for the characters they do play, you can actually get much larger revenue lifts. But yeah, in a game like Candy Crush, if you're doing quantity discounts, I think there's a lot less heterogeneity there than you would on character preferences in a 150 character game.
Speaker 1:But I think that's a really important point, eric. Wouldn't the theory be that personalization is more profitable when a game has the ability for players to express more heterogeneous preferences? And when games are more homogeneous in the amount of preferences that people can express than personalization, there are less gains to personalization, and to me, that's the kind of example we're going back and forth on. To me, league of Legends has so many different ways that you can have heterogeneous preferences, but when I think about a game like Jelly Splash or a game like Candy Crush, it's always about win probability, and that, to me, is a testable hypothesis. It's always about progression how fast I can progress and what is the minimum amount of effort I can use, which is why I might argue that personalization has less gains in a game like Candy Crush versus a game like League of Legends.
Speaker 2:I think in the content you're buying, yes. But again to Julian's point about free-to-play games hitting a super wide audience. There's a lot of heterogeneity and skill and challenge interest, and likewise there's heterogeneity and willingness to pay for higher win probability. So you might be getting people on the very low end who are like, yeah, I'm not going to pay a dollar to beat the level, but maybe I'll pay 50 cents.
Speaker 1:I'm just guessing. I've never worked on. Doesn't that just get us to price discrimination now?
Speaker 2:Yeah, isn't price discrimination, personalization.
Speaker 3:That's an interesting way of doing it, but of course price discrimination would be a form of personalization I don't think a very promising or valuable one in setting those candle-social games, because you have pretty strong communities of people who don't want to be treated unfairly. I think that's why offer targeting is such a in my mind at least powerful way of doing it, because, again, you minimize the cognitive load for somebody to find what they want and just offer it to them. Plus, you make sure that you give a nice discount to everybody. So you try to actually minimize the any semblance of discrimination. You might even ideally make all the packs you use to target two players available in the shop for them to find and buy as they like to, but then you advertise it to them in the pop-up. You make it easy for them to sign it.
Speaker 3:You make sure that what you give to them gives them a great experience after in the game, and that's how you build, first of all, the experience for them. Hey, this was actually a good purchase that made sense for me. I really enjoyed this. I got the utility I wanted from it, and then people will come back and make another purchase. That's at least my experience and I've seen this across more than one data set. I believe.
Speaker 1:So let me ask you this, because I think the UX point of personalization is a cop-out, because to me, the control group here is not nothing. The control group here is a pop-up of a standard IEP offer. Shouldn't that be a true control group? And have you experimented with that, like, why can't I just offer someone a pop-up for a standard SKU that isn't a sale or isn't a limited time bundle? Isn't that a better control group? Again, it could be.
Speaker 3:It certainly could be an interesting experiment to run as more focused in on the attentional effects here, but I think it would be a follow-up study, I'd say, where we actually try to really isolate what is the effect of advertising our SKUs versus discounting them. And I think there's so many. Yeah, and this just goes to show, I think there's so much literature still missing in terms of what we could research here and what questions we could dive into. And, yeah, that'd be an interesting experiment to do in the next step.
Speaker 3:I think what we wanted to do with the paper, and I think also actually did is to show okay, we need to also pre-qualify this a little bit, because if you know the marketing literature, the academic marketing literature, there's a lot of pushback on the idea of running regular price cuts with the products you offer in the market, because there's very often in traditional settings this notion that you then get yourself into always having to run promotions and you can't actually ever sell anything at full price, that consumers expect this from you, and that it might actually also give adverse quality signals.
Speaker 3:Right, like usually the very good stuff isn't discounted as much or has high price points, and so what we wanted to really do here is hey, dear marketing scholars, we maybe need to revisit some of our scholarly thinking in terms of price promotions when we consider these new business models that are happening on the mobile and with premium pricing. And I think that's the main point of the paper and I think it achieves that one. And I agree, there's so many follow-up questions that I would also love to dive into Some of them. I am actually I have all-on research projects on some of this that I'm currently working on and, yeah, hopefully they'll also land in the mainstream journal. That'd be lovely.
Speaker 1:That's the one thing I want to make sure to get in the cast right, because one of the things this paper does is that you reject that idea that people get trained on the special offers and they start to expect it and that becomes their new price anchor. If I were to take a behavioralist approach, you reject that. The empiricism did not show that. Is there an intuitive reason why that was the case, like why wouldn't that happen?
Speaker 3:Yeah, so that's actually in the paper, so more in the discussion part of the paper. By the way, there's also a very key delineation from the Leavitt paper in our paper. I think it's on page well, let me look it up on page 117 in the paper.
Speaker 1:It's hard to get this PDF, man. Let me tell you Every time you go to every fucking website possible to get this thing. Science Gen.
Speaker 3:Library Gen. It should be on SSRN, to be honest, so there you should find it. And then it's in sexual three-point. We detail a little bit how our treatment differs from what Leavitt and L have done, by no means trying to discount what they've done. I think it's amazing paper, super important, but we just have a different research question and treatment.
Speaker 1:So what would be the intuitive reason that players don't get trained on special offers and start to expect them? That becomes their new price anchor.
Speaker 3:That might still be happening. We can't really speak to that. But what we do in the paper and support with some additional analysis but even here I think we need more study to be able to say anything conclusive is that it actually goes back to this freemium pricing model when you have the free version that players are used to using and have engagement with. Then they buy a premium upgrade at a discounted price, but that purchase makes the utility they derive from the free version at least from the combined premium and free makes that utility higher and turn increasing their engagement with the free version, making them stick around more and more likely in turn to buy something again. So we especially allude to this complementarity between the free and the premium bundle that make price promotions possibly have such different effects in these settings.
Speaker 2:I guess the argument is let's say you discount some bread, right, if someone buys more bread, there's still a certain amount of food they're going to eat every week, so it's not like they're going to stick around and continue going to that grocery store and buy more bread just because it was discounted. But if you discount these goods like, your elasticity of how much game you want to consume, how much jelly slash you want to play, might increase from getting these offers and that effect could outweigh the sales training.
Speaker 3:Yes plus also the fact that the base version is actually free and so even if you would stop buying anything, you could still always use that and I think that is probably the key component there as well, and the utility in that increased as well, and so you might come back and get another purchase later and then you buy it again. So I think it's less about what Phil asked about. There might still be a lot of deal-seeking that you induce by doing this in the setting, but I think it's a bit of a question of separate interest that we don't focus on here. It's more about this complementarity between the free and premium version and the increases in engagement you can also see, at least in the utility that players derive from the combination. That is what might make price promotion so have such different effects in this setting.
Speaker 4:I wanted to go back to this idea of price anchoring. I personally am not a huge fan of this. I think there are perhaps scenarios where price anchors are possible, but I really just see it as the equilibrium price going down. If the solution to raise revenue is to decrease prices through discounting and players don't buy stuff if you ink the prices, to me it's like the market price has just gone down for the content and I know that's the generalization because it assumes uniformity across the spending distribution. But I'm curious is this a price story or is this a quality story? I just don't know. Is there a way to fractionalize the content such that you can give it away to all participants and keep those whales spending up at the top of the distribution because you don't want to decrease the price for them? I think that's a very interesting question.
Speaker 3:This is also why I find premium as a topic or as a study topic for research so fascinating, because it just offers so many new perspectives. The first thing I might actually start with, since we're economists, we all know P equal to MC should hold in a competitive market. In a way you can say, digital goods have marginal cost of zero, and we actually saw freemium happening, which is P equal to zero for a good chunk of the product. I find that wild. To me it seems that P equal to MC actually holds there to some extent. But then what that also does in turn is I think where in traditional settings where you've got into the supermarket and you bought I don't know potato chips, the price information was probably the first thing you saw.
Speaker 3:It's the most salient piece of information. You have to inform any inferences you might make as to that product. Now in freemium you get to use the free version that might already comprise 70% of the functionality or a good chunk that actually, and you can do that forever. So a lot of inferences will no longer happen from price but from that experience with the product version, the free product version, and I think that ships dynamic so much that I think the adverse effects that we've seen from low prices on quality inferences and other settings again do not apply in the setting in the same way, which might be yet another fact why price emotions can be such a much more effective tool if you want to make the economics work.
Speaker 1:I want to spend another fucking hour on this man. I don't want to destroy too much of your time, Julian, Is there any more things we can do to advertise you?
Speaker 3:No, I think we covered a lot of good stuff and, yeah, really excited to have been able to talk with you. I also feel like this was more of a starting point, but that was also the point of that. But this was also the point of that blog article on Eric's log on mobile def memo was really more to get people started and thinking about this and start some conversations than any anything conclusive. And I think that red love is in the next step. Economist like us here, working and gaming, jump on that and actually maybe make it a little more analytically tractable and think about the optimization components that I hear all the second order effects that Chris also brought up and, yeah, maybe we can also chat about that again in a few years when we've all done more research on it.
Speaker 1:I think we really need to rip into that paper that you posted. I think we should definitely have a debate series. I think that'd be a lot of fun. I know Seaford would love that too, and I know these two gentlemen. I think we'd all have a fucking blast doing that.
Speaker 4:Awesome. This is a lot more exciting than I thought I was going to be. Very some passion, some love.
Speaker 2:Yeah, so I wouldn't get to dig into self determination theory or just like explanations of game utility.
Speaker 1:But yeah, topic for another time, episode 16 in the can. We should teach this to our children.
Speaker 2:Economics is major, major, major. Everyone has to major in economics. Number one for personal survival Economics is major.
