Game Economist Cast
What does the new wave of open economies mean for monetization? Will negative externalities overcome cosmetics economies in the long run? What exactly does a game economist do?
Game Economist Cast is a roundtable discussion of the latest developments in mobile, HD, and crypto games through a bunch of people figuring it out using the economic tool kit.
Game Economist Cast
E17: Regressions, Gin Rummy, and a VERY Special Guest (w/David Nelson)
Somehow, the cast wrings another guest, David Nelson, the former VP of Experimentation at King. Christopher Kaczmarczyk-Smith runs a regression on negative price, while Eric Guan explains the economic dynamics of rewarded video - bots and all. The crew debates how to solve the F-99 monetization problem and checks out what the NEW Experimentation Group is up to.
If you could talk to anyone, who would you want to talk to? And I was like anyone. And he was like anyone and I was like I seem to leave Number one, I seem to leave, without question, anti-fragile black swan, I seem to leave. And then I was like I don't know, I named like a ton Steve Levitt. And then I was like Obama, go find Obama.
Speaker 1:And this poor HR guy. The thing that sucked is he worked at London. In London, you do what you're told. And so he'd been told to go ask me. And then I gave him this list and he was like oh, and for me I just hate going and see what happens. Anyway, the dude reported back once a month. For three months he was like still nobody's answered. And then still nobody's answered. And then the third month he was like nobody's answered me and I was like OK, idiot. And then I just wrote to Levitt and I was like yo want to do something and he went back like yeah, let's do it. And I was like see, it's not hard, and that's how that thing got started. Let's start with utility.
Speaker 2:I don't understand what it even means.
Speaker 3:Everybody has some kind of utils in their head that they're calibrated.
Speaker 2:There's hardly anything that hasn't been used for money.
Speaker 4:In fact, there may be a fundamental problem in modeling you wouldn't want to model Game of Codems cast episode 17.
Speaker 2:And the fire did not stop last week. It is still burning. Somehow there's enough wood in the furnace. We have a guest with us today. I am happy to introduce Mr David Nelson, the CEO of Magnetic, which has some sort of description on your website that was generated by chat GDP. Yes, you mentioned that yourself. Former VP of experimentation at King David. How are you Doing great?
Speaker 1:Thanks to him, I'm beyond. This is him. I've got my game face on. I've been looking at Excel all day so that I'm ready.
Speaker 3:Now you need to get off of that soup. So you put on his game day glasses for us too.
Speaker 1:I did, I did, I learned, I did. I just put numbers in a sheet and then I used the sum thing, because the sum always feels. It feels Greek and cool. So I just summed them and then I was like I'm ready to talk to these guys.
Speaker 3:If you want to look like you're doing a cool math, you put a little sigma there. That's it, baby Summation.
Speaker 1:It's been a running joke for a long time with some people that I work with that are smart, but I still don't really understand what a regression is.
Speaker 4:Wait till we talk about p-values. When progress goes backwards, like you had a bigger house, but then there's a recession and you had a downside, so that's regressing, okay.
Speaker 1:But that's regressing. Right, that's regressing. I know a lot about that. I'm 48 years old. My body is regressing like nobody's business, but I've had dead serious people that have listed that they are economists in somewhere in their title sphere whether it's Twitter, linkedin or Myspace or wherever that said to me, I'll just put together a regression, and I was like, oh you, oh no, when I get it, will everything just be better? That's why I'm really excited to finally meet this legendary regression.
Speaker 2:Speaking of regression, I'm happy to introduce Chris. Chris, how are you? I don't like that. You should introduce Derek first.
Speaker 3:If you're talking about like oh, I'll ask regression. I'm doing very well. Couldn't be happier, couldn't be better New products coming out the left and right out of our ears at Star Atlas.
Speaker 2:Star Atlas taking up tweets these days, getting likes, getting retweeted by the CEO. Chris, yeah, oh me, yeah.
Speaker 3:No, I'm, I'm. I try to stay under the radar, but I just can't help but interact with communities. It's like it's like an addiction. It's not good at all. But no, we've been doing really well, seeing revenue going up and really strong. Arpdao stuff like this is really exciting to see as a company in the web three industry. That's just not a trend that you're used to seeing. So, feeling good, excited for this conversation today, David is, he's a. He's a Titan. So even though he doesn't know what a regression is, we'll forgive him that.
Speaker 2:And of course Eric is also here, head of economy design at Superlayer. I'm doing all right, yeah, but thinking about this.
Speaker 4:We launched a mobile app in Germany, I think a week ago, so I've been that's for all of my heads at it's all just sign up issues. At this point, it's all like you go in through and be like, oh, like, why did all these users disappearing? Oh, there's a bug. Okay, we got to fix that one. But yeah, it's all the app launch.
Speaker 3:Kind of craziness, is it like an is an experimental launch, or is this a full launch, just like a German app?
Speaker 4:It's like the small one where we were not blowing a ton of money on user acquisition yet. But I'm trying to iron out all the kinks. But yeah, I'll be talking about that a bit today. I think it's an interesting model.
Speaker 2:I think especially from an economics perspective Speaking of things we're going to be talking about today. We have two wonderful topics. Eric, what are you going to be talking about? Timeout trophy?
Speaker 4:It's this mobile game kind of advertising platform. I guess they call themselves loyalty, but really what it is taking money from game advertisers and trying to pass it through to players. Like mobile games spend tons of money on advertising. Can you get more players by passing some of that revenue through, and what are some of the adverse selection effects that happen?
Speaker 2:And for our second topic, we'll, of course, be speaking to David and man. We have so many things to talk about, from Steve Levitt's paper to the origin of the experimentation group, to what is a regression. Let's see how far we can get. Before we get to either of those, let's talk about what we've been playing.
Speaker 4:Got a selection of good things on sale. Stranger Eric, what have you been playing?
Speaker 2:I've been playing a ton of F-099.
Speaker 4:F-0 is this old racing game. It's set in the future. It's a play on formula one. It's like formula zero, it's even faster, but as an old game the franchise died. They took all their racing games and just put it all into Mario Kart. But F-099 is a battle royale version of the game. It's a battle royale version of this old Super Nintendo racing game and it's a lot of fun. It's fast, it's hectic, it's very high stakes and if you crash, your car goes bouncing all over the place and you explode and you die, which is great fun in a battle royale format.
Speaker 4:One thing I think is particularly interesting, though, is that this game has no monetization at all, and it seems like they made it with the plan that it would be short lived. They did something similar with Mario 35, which was like a Mario battle royale, and these battle royale games they tend to die out pretty quickly. If you're going to sell in-app stuff, there's an expectation. You maintain it and support it for a long time, and if you go under, if there's a small indie studio going under, people accept it. But if Nintendo takes away all the skins you bought or whatever, that's a bigger deal. So yeah, they made the interesting choice to make it free in the online subscription and I'm pretty sure they're going to shut it down in three to six months. But yeah, but having a lot of fun while it lasts.
Speaker 3:Are there in-game microtransactions?
Speaker 4:No, none, oh, okay, so we were talking about this on Discord.
Speaker 2:What do you do with this game? Like, how do you monetize the love in this sort of model if you're not going to have IEPs? Like, how do you get that depth of engagement? That, to me, has always been the frustration. How do you align, or at least how do you correlate the marginal benefit games that can provide thousands of, hundreds of thousands of hours of gameplay to provide with some sort of cost profile and MTX, to me, was the solution to that. Or even coin arcades Putting in a coin into the arcade was the answer to that. That's how you correlate those two things. Can you even do that in a subscription? It doesn't a subscription, it's flat pricing, right. You can't correlate it. It's naturally going to be uncorrelated, but you could go with coins, though.
Speaker 1:right, Like you could do the. If you think about all the free to play models, if you make it when you make a purchase, the subscription could give you 50 tokens to play the games and then if you love it you could put in gets more tokens.
Speaker 2:Literally go back to the arcade machine model.
Speaker 1:Yeah, how different is the arcade? If you think you're playing Royal Match, you're getting gold coins all the time, so you're putting the coins in when you want to extra moves and then sometimes you run out and you go to the cashier and you get some more. They're exceeding you the whole time. It steps between the arcade and the casino. It's on middle ground with no cashier.
Speaker 4:I heard rumors that Game Pass Xbox was thinking about something like that as well.
Speaker 2:Oh, how would that work? So basically you would pay a flat fee and get like a couple tokens a month, yeah, and each game might have a token price to play.
Speaker 4:I don't think they were committed to this. It was just an idea that I heard the run around.
Speaker 2:Oh interesting.
Speaker 1:Is that really that bad an idea? I mean, like consider the Netflix stuff also. You could do the same thing. Your subscription gets you like this much.
Speaker 4:What about the spending aversion issue? The same reason Disneyland doesn't charge prices for their rides is because they don't want people to think, oh, I only got 10 tickets left, maybe I don't want to go on this ride. And that decision analysis affects your enjoyment of the experience.
Speaker 2:But I see the thing is that some park experiences do that. So when the carneys come to town you are back to a ticketing machine and that seems to make sense for economists. You have a ride, it's more popular. You need to ration what the wait times are. It has a certain amount of capacity. Ok, you can raise the amount of tickets that it costs to go on the machine. So it's not just I want to maximize as much revenue that the ride gets. It's also like the functioning the rationing mechanic of hey, something costs more, people are going to do less of it. The man curves slope downward.
Speaker 4:But Disney doesn't do this. In those carneys, though, the boots and the rides are competing with each other, whereas in Disneyland it's all owned by one thing, and that they're indifferent what ride you go on.
Speaker 2:When you say the boots and rides are competing, what do you mean by that? They're literally separate companies. They're separate revenue streams, I think so I'm not sure.
Speaker 4:I've never worked at a car. But I assume so.
Speaker 1:You know the CFO. I think it's still the same CFO. It's the Spence Newman for Netflix. He was the CFO for parks and rides or whatever. For Disney, he was our doctoration for a while and then went on.
Speaker 4:Maybe that's why he likes the bundle model so much.
Speaker 3:I saw Netflix stores are opening up soon. I don't know if that was like a fake news article that I saw, but maybe that's his brainchild.
Speaker 2:No, that's true. Regression right there. I think that's what we were talking about.
Speaker 3:We've identified the regression. So to me, though this token model is, it's a totally different monetization strategy. In that capacity it is like pay for play or pay per viewer. To me it's almost a premium model. Like I know I'm going to have this fixed amount, presumably there's some way to spend more. So my marginal all of a sudden my marginal spending is now a part of my decision again. To me that's just premium or free to play or whatever. Free to play, but with a subscription, like basically pay to play premium games with microtransactions, which we know players don't really like that much, especially the types of be signing up for Xbox live, these like types of services. So I wonder how that would change the. To me it's more. This would impact user retention, user acquisition. You might see higher spend paying player, but maybe you see the aggregate across your entire player base. You maybe you see a reduction in players or something like that.
Speaker 2:I never think we got the shareware model. There was supposed to be middle ground, I think, transition period between a full box practice product and like a true free to player MTX model, and Blizzard did a little bit of this. Right, you could get to level 10 World of Warcraft and that's before the subscription would kick in. I think they started to experiment with this on almost all their products. It was like a very mild shareware and no one really figured out how to make it work in the long run. But I wonder if there's something there for the type of games you're talking about, eric, like FF 099.
Speaker 4:You play for free and then at some point you pay money. But how do you deal with the expiration date of the game? You know this game is going to be dead in a year.
Speaker 2:I think you could cap. I think you could at least. That's just a retention problem, though, just for the entire product, but I still think the shareware gets you the on ramp and at least can maintain some sort of user base. See, rainbow Six Siege did something like this too, where they slowed down progression for players on a particular version. Now there was an entry fee to still get the version that was slow is about $5. And then what they did is that if you bought the full box price product that you would progress at the normal speed. So I thought there might be something to that model too.
Speaker 2:So there have been real small experiments, but just because it failed once, I wish someone else would take it swing at it. Easy for me to say, but I just I wonder if there's something. Maybe you can get to level 25 and F099 and then you have to pay tokens. Maybe there's something there. There's got to be something that, like you're looking for something that you can layer onto any game for these types of subscription services, right, you don't want anything that's specific to the in-game contents, like boosters and Candy Crush or extra moves or something of that nature.
Speaker 3:At the end of the day, this is like pretty antithetical to not to Nintendo micro transactions. They want to provide a pretty fun little product at a very low marginal cost for their paying players to try and keep them on board. That's the way I see this, this type of game, like I don't think Nintendo has to put as much. I could be wrong, I don't know how much a game like this costs to produce, but I don't think that it takes as much to put this game out as it does to put something out that, like, the Xbox crowd would be interested in playing. It was Tunic, oh, too Micah.
Speaker 2:David, have you been playing anything? We'd like to give you the opportunity here. What would you like to be a guy like us with? What have you been up to? What do your gaming habit we're going to like really?
Speaker 1:go in the other direction. I was a professional poker player for a couple of years and when I was eight years old I learned how to play Gin Rummy for money my grandmother so. She'd tell my dad to go borrow 20 bucks from my dad. I'd borrow 20 bucks, she'd beat me for all of it, and then we'd go out and she'd buy me like a $5 toy that's my scream and pocket the rest. So she was so early on. She sat down and was like look, I'm going to show you why this is just math and it's not that hard. And then I played Gin Rummy for money through college and I played some poker, but not that much. And then when the poker boom happened, I was working in a factory in Sweden and I started playing online poker. I played online poker for two years before starting to work in the poker industry.
Speaker 1:I've always enjoyed card games. I think they're really cool. And I was talking to one of the founders of King and they told me about Solitaire Cash or whatever. But I mean they were like, dude, they have real money, apple Pay, inside of the app. And I was like no way they have Apple Pay inside of the app. And he showed me and I was like this is stunning. So I downloaded it but it wouldn't process any of the payments. I tried to make it a challenge. I've written a customer service. I don't know doesn't work and I was really wanted to find out. But then the problem is they only seed you with six bucks or something like that, and I want to get into one of the actual tournaments with Apple Six crowns.
Speaker 4:And it's like a.
Speaker 1:Solitaire competition. It's real money Solitaire U versus another player. Actually it's like a six person tournament. Top three get paid or whatever. You break it very hard, but half of it disappears in regular. I wanted to see if I could free roll my way all the way up to the bigger cash ones. So I just got a couple of days with grinding gems when I have a free moment in a player to player Solitaire. So up from 300 gems to 1180. And I just need to get to 1200 and I can free roll. And what is this called Solitaire cash?
Speaker 3:I think I've seen ads for this. I was like, oh, this is a scam, I'm not going to click on.
Speaker 2:That Is this where that cat game that King made came from. There was some sort of cat run Everyone thought it was super interesting and then it died like why?
Speaker 1:it wasn't super interesting. I was so mad about that. That was so mad about that. They could call it shuffle cats and it was a gin rummy game. I'm like you know what gin rummy people want to do. They want to play gin rummy. You don't need a cat, you just need a really good gin rummy game.
Speaker 3:Damn it. This is one example of not being able to just paste a picture of a cat over top of something and get more money for it.
Speaker 1:It doesn't work. It doesn't work, especially with gin rummy players. Just interview a few of them. None of them want cats in the game, they just want cards. I'm going to play that and then I still I'm way deep into Marvel's map. But also card game. But I love the complexity of it. I've got an old gray.
Speaker 2:Thanos deck. Are you bored? There isn't a lot of variation in the core gameplay, though. It's like your lanes and then your decks fairly stable, though, right.
Speaker 1:No, I don't know. My son and I have been playing it, so we've. He made this Thanos deck for me. For a while I was like there's no way it's going to work. And then it worked really well for a while. And then I switched over to this Mr Negative and Jane Foster Thor and I love that combo and I was like crushing people got up to infinite and was like I got pretty high with that one. You gotta put in the hour, like you gotta put in some time on it, but it was super fun. I always get fun when you can sometimes feel the vibe. You're like I'm definitely planning to get a 12 year old and then you're like loser.
Speaker 3:No, I don't. I don't know a lot about the poker industry or the kind of gambling, online gambling industry. I'm assuming there's a bunch of regulatory stuff around letting people win prizes and I'm curious why doesn't a game like Marvel Snap, but for money, exist? Why doesn't a gambling version of Marvel Snap exist? Is there a reason for that? Is it just the wrong? There's no overlap in the customer.
Speaker 1:If there's gonna be any payout you have to do. There's legislation, so most countries you need to get a license. I don't know what the I knew it really well in 10 or whatever. I don't know it anymore, but there used to be like here in Europe there's Malta, gibraltar and there's a few other places you could get a license. You could also get these sketchy international licenses and, like Curacao and there was Isle of man and all these different things.
Speaker 1:One reason poker was a bit weird was because a lot of people that were doing poker were like this is a skilled game. It is definitely a skilled game like good players win. This isn't gambling. And then there was a whole longer story for some other wonderful podcast where we can talk about the poker stars and full-tip poker, more of these big companies, and they had lawyers that were sure that told them that we're all good, this is not, you're not gonna go down, you're not gonna be in the wire after. Everything's to be fine. And then, yeah, I see, as I was working for full-tip poker and a friend of mine called on a Friday and was like, hey, you need to go to the website, man, the FBI just took it and I was like what, and you go ahead and log on to fulltippokercom and there was just the stamp of FBI and it was like this domain has been seized by the FBI.
Speaker 3:So that's really interesting. I guess my question was like why doesn't a very gamey, exciting version of log of legends exist, with a kind of gambling or poker or payout back end and it's just legal, like that's the reason and it's not?
Speaker 1:only that. I would say probably these are rough numbers, but something like 5% of people are willing to wage a real money to win real money. And you can get more people to do it if they're in Vegas for some reason or another, but on a regular basis. The general population's uninterested in it, so it ended up with something very niche. And then I think in general also, I never wanted to work in the casino side of it. Poker was cards and I enjoyed it, but the casino side for me just gets dark.
Speaker 1:It's like trying to convince somebody that you're like no, it's okay. Two minus four is like well, about one, I'm not it doesn't feel okay.
Speaker 3:I would love to have an episode about casinos and, just like the whole entire industry.
Speaker 2:We will. We'll bring on Matt's, who is also a game economist. He works at W Games in Toronto. He works on social casino, just like a really misunderstood genre that I don't think a lot of people understand that they generally say it's just a bunch of degenerates, which really isn't the case on mobile social either, because there's gonna be no payout, it's all trapped within the system. I think it's just things people haven't understood about how these games work and I think Matt will be able to illuminate us. Look forward to that in a couple of sets.
Speaker 1:Just a real quick. Another one of these like inside. On the inside, we're sitting working in online poker and Zynga poker comes out and we're all like there's no pouts, nobody's ever gonna play this. This is silly, it's gonna go away. And then it was like Dow's going up, revenue's going up and everyone's just gonna blow up. Everyone will stop soon, but you're actually. I think it's an interesting question. It's the opposite of your league. It's a legend's question. The game of poker has a certain. It's fun in its way and, as long as you believe that having a big pile of chips is good, it doesn't need to be cash for that to be a fun way to spend your time. I think the same thing happened with the social casino. People are like no way. They're incredibly viable businesses.
Speaker 2:Let's dive into articles. Eric, do you want to hit us up?
Speaker 4:So I mentioned where we launched this app called Trophy. It's a we call it a mobile games loyalty platform, really what it is play advertised games and earn some money to do it, and so there's some backstory here. David Phil, feel free to chime in because you guys have been in mobile games longer than me, but user acquisition is really important for mobile games for a couple of reasons. One is that the tracking and attribution is way better and therefore you can see your return on ad spend and optimize that much more effectively. You've got your devices everything from click watching a video ad to clicking on it to installing is all tracked right, whereas maybe in the traditional PC box game, you have to run a big advertising billboard and you have no idea what the impact of that was right Back in the day. The other is that there's big discovery issues with mobile. The app store is notoriously difficult to search, and so you have to get direct to users somehow. And yeah, mobile games spend a ton on user acquisition. A lot of this is in video ads. If you've played any mobile game, you've seen video heads for other mobile games, and another technique that's often used is called what's called an offer wall, or there's these affiliate marketing networks where it's like, rather than just paying to put a Candy Crush video in front of somebody and giving them 10 cents every time someone clicks on it, you might say, hey, for every user you get, whatever platform you are a third party to get to play Candy Crush to level 100, we will give you a dollar. And so these often will create offer walls where it's like hey, play Candy Crush to level 100. They'll try to attract users who are willing to play that far in order to get that money, and sometimes these affiliate marketers pass that revenue back to the player, so they might earn a dollar for the user who gets to level 100, and they might have an app that says we'll give you 50 cents if you install this game and get to level 100. That's affiliate marketing in a nutshell. A lot of these offer walls struggle from fraud because, first off, if you're paying people to do stuff, they'll figure out ways to bought it and cheat it. And so there's this big adverse selection effect where, like the people you're getting from the offer walls probably are not as good as the people who clicked on a video ad of their own volition because they opted into the experience.
Speaker 4:Traditionally you think of when you sell a product, your first users, your first customers are the ones with the highest willingness to pay the highest utility. Sometimes it's called the golden cohort. You know, the first month of users you get stick around way longer than anyone else, whereas when you start paying people you gotta get the opposite effect. And this adverse selection issue you see it in insurance. But it's this tricky thing where if you pay too little then you're only gonna get shitty users, but if you start paying more then you're like you're losing more money. So this is like a very tricky pricing question to balance. But yeah, so anyway.
Speaker 4:So what Trophy does is there's the offer walls that might say play Candy Crush to level 100, get a dollar. And what we're trying to do is try to gamify that process a little. So an offer might, for example, might require a user to play for 10 hours. But the user sees that and they say I don't know if I'm gonna play this game for 10 hours, I don't know if I'm gonna get all the way to level 100. And so what Trophy is trying to do is try to gamify that. It's almost like amortizing a loan or offering insurance where we're saying we'll pay you 50 cents when you get to level 100. And we'll also pay you maybe five cents an hour along the way. Try to breadcrumb it a little bit and then we take some haircut off the top to make it the offer more attractive for someone who might play for an hour and be like, okay, I'm out of here or something. Or they might get an hour in and say, oh, actually this is fun and I do want to play all the way to that level 100 goal.
Speaker 4:And yeah, so Trophy is doing this.
Speaker 4:There's a couple of misplay is the I think, the biggest platform that's doing this currently, and it's all time-based rewards where it's just play this game and for every hour you earn a certain amount of dollars.
Speaker 4:But yeah, it's an interesting, it's a big pricing exercise because you say, okay, the candy crush will pay us a dollar if they get to this point. If we pay five cents an hour along the way, what does our retention curve look like If we pay slightly more or slightly less? It'll affect the drop off rate, but it also affects how much money we're making. And there's a ton of opportunity for personalization here as well, where, if you can tell, someone plays a game for an hour and quits, then you might try to offer them a lot more of these short or like front loaded games, whereas if you know someone will find an RPG and play it for a super long time like how can you target them and how can you offer the right amount of payout, where it's high enough to get them to the goal, but not more than you need to offer to get them there. But yeah, anyway, that's affiliate marketing and Trophy in a nutshell.
Speaker 1:It's an app, then right. So I download the app and then I go in and there's some offers for me. So you got a bunch of first party data. In other words, if I'm picking this one and this one, you don't offer me this one and this one and you can start tailoring what you offer me.
Speaker 4:That's right. It looks like a mini app store where it's like a game store, where there's only like 10 games on there and they're all rewarded.
Speaker 3:How's the equilibrium here? Not to pay the players so that the payoff for the game is just Epsilon, if you've got, and correct me if I'm misunderstanding this but so the affiliate advertising? You've got some game and they are paying a third party. That's gonna say we're gonna bring in players who are gonna play your game and they're gonna be exposed to a bunch of advertisements. Those advertisements are going to pay you. Let's say Candy Crush, they're gonna pick King a bunch of money through advertising because they got to level 100. So King has made $100 from advertising. They're willing to pay you guys X in order to for those customers. How is X not 100 minus Epsilon? How is it not like this tiny amount? Because anything any additional users through this platform is good. So I guess, like, what's the equilibrium? What does the equilibrium look like for these wages for these players? If they're producing let's say they're producing 10 cents an hour, are they getting like nine cents an hour?
Speaker 4:To be honest, I don't actually we don't actually know how much the game, in this case Candy Crush, is making from these users. What we have seen is that for games with very reliable revenue, for example ad-funded mobile games, they pay a lot more, or they're willing to pay because they know okay, in order to get to level 100, the user has to watch 100 ads, which will earn us X dollars, and we can give them X minus Epsilon dollars. But I'm sure they've market power effects here as well where, even like they don't have to pay X minus Epsilon, they could pay X minus a much larger amount because they know we'll still feed them users regardless.
Speaker 3:That. I guess that's like. My question is once a competitor comes in, what's stopping them from like driving that wage? Oh, I see, like for competing offer walls, yeah, I guess the equilibrium would be where the players are actually getting paid Epsilon, which is just this tiny amount. There's a really complicated problem once you have competition in this market.
Speaker 4:Yeah, so there's a bunch of these offer wall companies and since they all use the same affiliate networks, they tend to have the same things. Like all of them will have Candy Crush and all of them will say we pay you on level 100. I think where we're trying to differentiate, I don't know, and they are very commoditized and the users will shop around and see okay, I see the same offers, but this one pays me more. We're trying to differentiate is what I mentioned with the bread crumbing of trying to pull the rewards forward and potentially eat some loss on it in order to make the experience more attractive for a user who might not get all the way to the end. But yeah, I think in and there's some platform lock-in effects. If you can get better games or platform exclusive games, higher quality games, then you might attract people. But yeah, there's definitely a competition on that front.
Speaker 2:I'm just struck by how brilliant this idea is. Of course, this makes sense, right? User acquisition costs have skyrocketed. What's the natural evolution of free of play? Pay to play, not to take something out of the web threes playbook here, like this, is much more straightforward, much more controllable than a lot of web three things. But I guess what I'm interested in is it becomes inbound acquisition rather than outbound, like normally.
Speaker 2:Acquisition happens on these platforms when you create a really valuable service and then you just slap ads on it, like Facebook. There's nothing natural about ads on Facebook. It's a social media network and they just slapped ads on it because they have eyeballs and they also have data which lets them have targeted and personalized ads and it fits naturally into the feed. So I guess, maybe, perhaps there is something natural there. I set out to be a social network and ads just happen to make sense. But there is something to like. Hey, this is clearly an ad platform and if you're interested in finding games, I guess you'd want to go there. There's a conglomeration effect, like I'd want to run to this app.
Speaker 2:I guess the personalization piece is more interesting, because you don't know who the winners or losers are going to be. When you have an ad on Facebook, you don't know who's actually going to be the person who monetizes Like. When we say cost per install, it's actually average cost per install and some of those users are going to subsidize the other ones, right? Some users can be worth $10,000 and that's what subsidizes all the zeros. The average here is doing a lot of work, like. The median is almost always zero. But if you had a platform in which you knew that you could serve an ad to someone who was verifiably someone who had paid beforehand, you could give them pretty significant payout, right? You could give them a much better rebate on their in-game spend. You know that they're valuable for it. This is the whole thing with Facebook post backs, right? Is that Facebook started to know who the spenders and who the spenders weren't?
Speaker 1:It's funny that we talked about gambling beforehand. There's two things that this reminds me of. In online poker, there was something called rakeback. Do you guys know about rakeback? Is this like a common?
Speaker 1:Basically, a couple of companies built this poker platform and they had a brand on it and then they realized, hey, I can like white label a bunch of brands so other people can use this same platform, because having a big base of players mattered. Each little brand couldn't set it up its own. Liquidity matter. You had to poke a player, right. So basically, people started using. Then also, you had eight brands using the same poker network and playing. So what happened is the first ones are doing marketing to bring in people, but then the smaller ones start and they just start going into forums and offering rakeback.
Speaker 1:So rake is how much you take out of the pot each time. It's the revenue that you make. And what they would do is they would say look, if you play a hand and rake gets pulled out, some of that rake gets assigned to you as a player. And then if you came from brand X, maybe brand X is offering like I'll give you 50% rakeback or 10% rakeback or whatever, and you get some cash put back into your account. So like this is similar to that where you maybe as a player, you start thinking I'm playing and putting engagement here and spending on it and I'm gonna get some kickback.
Speaker 1:So wouldn't surprise me if, especially games that are high spend oriented, if it went that way.
Speaker 1:I think it's harder to work with things like Candy Crush because the relationship to spend I don't think people are even like aware that much about how much they spend or they don't think about it that way. I'm sure there's a few people, but for the most part I'd say if you went and looked at Clash of Clans, there's probably a bunch of people that, like I, spend money, I love this game, I spend a thousand bucks a month. I think it's gonna be really hard to find those players in smaller casual games. So probably the actual balance is what is acquisition costs. So if you know that there's a person who's like super active on a game, play in mid-core games and spends a lot of money, then maybe Clash of Clans is super sales willing to say I'll put this much in to get that player, though I can't see SuperSales doing that because it's gonna be harder with you Cause it's gonna be against the UA cost that you're really balancing not how much the player spends, it's the market that decides how much it costs you.
Speaker 4:We're all trying to tweak all the payout rates and seeing how retention changes, but how do people respond to changes in wages in games?
Speaker 2:How elastic are they?
Speaker 4:Yeah, so generally paying more causes higher retention. Whatever the highest paying game is attracts all the farmers. So there's a bunch of like people who just idle in the games to abuse it. So we always have a honeypot game which is the highest paying, and all the fake users go there, cause when the games actually look at the stats they're like, hey, these players have high retaining but they're not doing anything in the game, so we don't want these users, so we've got a honeypot game. But there's also that selection effect where the lower paying games have higher. When you increase a game's payout sometimes, or feature it sometimes, the retention goes down because the players who opted in at a low wage rate are the players who like playing the game the most. And when you increase the wage rate you attract Players who don't like, say, card games as much, and so they turn faster. It's actually been not obvious to me how to optimize these things.
Speaker 2:And I guess that really strikes at the point about personalization you were saying is, if you know the bots have zero spend on the platform and you're able to track that We'll assign them zero wage rate and that the people who are valuable on the platform pay them higher wages. They're more valuable like you can price discriminate here. I know we say personalization, that's like the nice word for price discrimination these days.
Speaker 3:Yeah, so they're paying for those users, for those views, because those views convert into downloads and those downloads convert into spend. If you've got all the sudden, if you've got like this in group of players, if you add in k-bots that's the same size as in, all the sudden and their average spend is zero let's say their average spend is zero, like just zero across the board all the sudden that lowers the value of the entire pool of players, which should lower the payout of that the advertisers willing to spend in the game. So there's this like big giant IO problem going on where, if you let these Contaminants or these bots get in and contaminate the sample or the population like destroys everything. I'm almost like that's right, okay sorry.
Speaker 4:Yeah, no, that's totally right. And these affiliate marketing networks, like I mentioned, they're open to a bunch of third parties and there is a ton of fraud, like exactly like you said, and they turn a blind eye, cuz like they're just feeding installs to these game companies. But part of what our goal is and these affiliate marketing networks, first off, the games don't pay as much because they know there's fraud and also the network itself takes a pretty big I want to say 40% cut, and so part of our goal is to make direct deals with these games by showing hey, we actually have high quality users that aren't fraud fraud in you, and if we can establish that, then the games will be willing to pay us directly and we get a much bigger cut because of that. But yeah, the whole system operates on like low trust and everyone knows everyone's fronting and so that's why all the wages are lower.
Speaker 3:So do you have any? I know it's probably the answer is no because of like QDPR and stuff, but do you have any idea what the what the users look like? Are they from very low-income areas? Are they just spot farms? Are they like? I'm picturing the guy with a thousand phones on his bike running around with Pokemon go going. We, we definitely run into.
Speaker 4:Multiac. There is a dude who made a hundred accounts on one device. There's people who have like multi devices, yeah, so there's definitely a lot of that stuff happening, but anyway, it's funny to me that you guys nobody reacted this.
Speaker 1:The fact that you tell somebody how much they get paid for it and then the lower number gets higher retention, the more you pay the retention drops off is such a fascinating discussion for like how do you motivate people for things and how we relate the money and where is our?
Speaker 2:motivation really. But I don't think that's what Eric was suggesting in the way that you're talking about it. It's not that those same users, but I mean, but maybe it's a trick of the numbers, you just in low users that drop the average higher numbers and you get a bunch of yeah, exactly, but I wonder if it's not also related to it.
Speaker 1:If you show me a game that I would be intrinsically interested in but it costs like right now I'm gonna pay ten bucks for playing I whatever changes to my relationship to it, I wonder if there isn't some weird now get that's like shit out of here.
Speaker 3:Hey, hey, I'll bring up a paper I did in my undergrad on zero prices and I, so this is a horrible topic to bring, or a horrible paper to bring up, because it's by Ariely.
Speaker 1:So I know Ariely is, I know about the fraud and I know all sorts of wonderful.
Speaker 3:It's great Okay so I didn't commit fraud. Ariely committed fraud, probably on this paper too, but basically papers that shows that. So we all know slow the demand slopes downward. So as something becomes cheaper, we consume more of it because of our budget constraints. And then he did this little cute experiment where he was offering people, I think, candies at the end of a checkout line at a cafeteria. Like an experiment that would just not like, wouldn't pass the sniff test today. But he's handing out these treats for a ten cents a pop and people are buying them. The second he offers them for free, for zero.
Speaker 3:All the sudden the demand, the quantity demand drops, and so his big thing was like oh, there's a moral kind of cost associated with Purchasing. Actually, my argument in the paper that I wrote in undergrad was um, there's a moral cost to Getting something for free. Now, this is like super behavioral, phil's gonna throw up if I keep, if I talk too much longer. So I did this and I then I explored negative prices because it's like okay, if zero cost has a moral cost to it, negative must have an even bigger moral cost. So what if we?
Speaker 2:give them 15 cents. This is where psych fucking fails. Chris, Thank you for this. They don't take it to the next level. Yeah, you got zero. Why not go negative? That's what I did, baby.
Speaker 3:Yes yes, I still have my sign that says chocolate experiment and I'm like this, like pudgy faced undergrad. I think I was a sophomore anyway, and demand did, quantity demand did keep going down for those negative prices. But I think, david, I could be completely Misinterpreting what you're saying, but that's what I'm thinking You're hearing is like do's do my just at least the result of my Utility function. My budget constraint does something get funny when I'm getting paid to play this game as opposed to having to play it.
Speaker 4:There's a completely different social. People took less chocolate when you paid them to take it.
Speaker 3:Yeah, the more I paid them, the less. Yeah, but it's weird, right that's good, that's exactly what I'm saying.
Speaker 1:Okay, phil, yo, I'm going out with some friends of mine you want to come with. It's gonna be fun. We're going to get dinner at this place. Like it's probably 20 bucks. It's gonna cost you 20 bucks for dinner. We're gonna, and you're like I don't know, maybe it'll be to shoot people. And then I say to you, phil, you got to come and I know my friends, don't worry, I'll cover dinner. Dude, I'll give you 50 bucks.
Speaker 2:It's gonna be like really great, they don't have the same effect, those Okay, I will happily concede that there are social norms and that there are heuristics we associate with certain activities, like paying someone to do something can change the guys of the activity. I will totally concede all those things. I would just say that those results which tend to be on the fringes of the 1% of human activity in human life usually get blown up by psychologists and Used to destroy the entire theory of the in trial model. And they never end up being able to do that. They just end up describing a very fringe part of the human experience which still deserves to be described, but what they think it proves is much smaller than what it actually proves. Demand curves do sound, lord.
Speaker 1:Yeah, but we could just go back down to this app now with the affiliate thing and ask ourselves Is this gonna be? The reason I'm asking this is because I look at like retention numbers for UA and what we're running ads for people. A lot Of times the retention for when you've, when you've gotten people through UA, through ads in games and Facebook and stuff, their retention is better often than organic installs people that have searched and found the game. So why? There's a bunch of different things going on but it's valued differently, right? The other one, somebody something said this is a thing and I value it higher than a thing that I stumbled on and I don't really know. This is a serious thing and on, I try not. There is how you're introduced to. Something matters Is and I just wonder what happens when you hike the prices. Maybe it does affect actually not only that, but like the relationship.
Speaker 3:I'll argue the fill thing here and that I think that's just a selection bias. I think that's just types of people who are coming from those. I haven't necessarily. There's no counterfactual there, right? You can't know whether? Did Chris's brain change when he got a chocolate for free? No, it's not worth running an experiment on it.
Speaker 1:Really, it was just interesting. I just wondered if they'd be able to see it in the natural data, like when we change this, because you're tracking people right, so you could say it turns out the same players are being paid this. There be paid one to play this game. Nine to play this game. Their retention is actually better the lower the prices, no matter which player. You should be able to see that.
Speaker 2:It was in sight. Speaking of interesting David, yeah, you're here with us. You have more citations than I think all of us combined. You've outside at us, I think. If we wanted to Google scholar, your profiles more filled out than any of us. You published a paper quantity discounting Connie. Massive funny discounts on a virtual good, the results of a massive pricing experiment at King digital entertainment. It is published with Steve Levitt of Freakinomics fame. John. List of field experiments fame. God pray that man gets a Nobel Prize in the next 10 years.
Speaker 1:That's what mr Levitt said to me. By the way, mr Levitt said to me when we did it. He said look like it's fun that you'll be on paper. We'll be on a paper together, but nobody will care about this. But when John list gets his Nobel Prize, you're gonna be so happy that you're on a paper with him.
Speaker 2:And I was like thanks, steve got to go to a meeting by. I really hope he does. He needs probably just 10 more years. There's like an age limit or age floor and you, david Nelson, you're also on this paper. We've referenced it multiple times on the podcast here. It's how we met. I remember this getting published. It was his published in July 2016. You ran a massive discount with Steve Levitt and Candy Crush on the number of gold bars that you could get in a particular. How did this come together? What's what's the origin story of working with Levitt, with King? What can you tell us?
Speaker 3:and your or origin story as well, david.
Speaker 1:Yeah, yeah you know little town and Rhode Island a long time ago.
Speaker 1:Actually, I think my parents were actually in Martha's vignette, anyway. No, but I joined King in 2013 and At the time the company was like blowing up and when I joined, I started working with a guy named Loshy Arnov who was known as a sort of maverick in the company. He'd he dragged the company on the Facebook and then dragged the company onto mobile and then it started working on growth topics. So the first thing I did was actually go to start working with getting candy crush onto Kakao talk, which was big social media app in Korea, and the next thing I started working on in like September 2013 was I actually went and met Twitter.
Speaker 1:I went to Twitter and I had this theory I don't know if you guys like a sock. Do you know a saga? Candy crush, approximately right, there's a saga map and there used to be these things called like collaboration locks, where you'd play 15. At the end of an episode you had to get three friends to send you would send out onto Facebook help me and three people that have to respond to unlock the next levels or you could pay a buck.
Speaker 2:Oh, you hardgated it.
Speaker 3:Holy shit, that's insane, it's what it was like those notifications are both super fucking annoying yeah.
Speaker 1:Yeah, it was awesome because this is one of the some of the most amazing conversations. This is like a serious conversation side king at the time about cutting down on the amount of notifications that are going out, but we've got people in marketing saying we have to cut these down. They're pissing people off and we've got a back-end developer very senior but back-end developer good, maybe it's the best marketing we have and I just lose is so much fun with. Like this made no sense. It should be the other way around anyway.
Speaker 1:But so I got this idea that what if we did this, a weird hack with Twitter, where what happened was you, if you logged in using Twitter instead of Facebook, we checked how many followers you had and then we hardgated some content based on how many followers you had. So if you've got I don't know five followers on Twitter, you can just one person or maybe even for a longer period of time. It's no big deal. But if you're Tom Cruise, you need 10,000 people to help you and I just thought it'd just be such. It'd be mayhem. It's just so much fun and I like these. There's a bunch of celebrities on Twitter. They don't have anything to talk about anyway. Nobody really wants to talk to them about anything.
Speaker 1:Now They've got something to tell them stuck on level 574 and I remember the King leadership team was like go ask him. So I flew there and met with the Twitter people and I just met with some engineers and some product guy and the guy was like You're hacking our graph.
Speaker 1:I was, I wouldn't do that. But and then in the middle, like basically at that time, king had a data science unit that was wrong on its own, that kind of sat as they just sat with the teams, but they're really their own functional unit. And then there were like product managers and then producers and people running Creation and somebody again in Marcus Jacobs had an idea to put these two things together with the data people and product managers, really put them into one, one group and work closely with the game teams. And so in the middle of me coming up with these crazy ideas, then somebody was like, hey, you want to be involved in the start of this. And I was like, yeah, sure, let's do it.
Speaker 1:So I went down to Malmö, just the southern part of Sweden, and started working with Pet Rescue, which was the second biggest, second biggest game at the time, and then I did that for a while, until the 2014,. I thought we weren't taking enough risks and I'd started working, doing more innovation stuff, testing more things. I was harassing the leadership team all the time. I'll give you this is what you do it because the game economy stuff, this is really fun. So we've translated Candy Crush into Japanese and we've got a request to translate Pet Rescue into Japanese. And I said to the leadership team I'm just going to say a random number, so it's not the. I don't remember what the number was I said what are you going to do with that $537 man? And the CEO of the company said David, how do you know? It's f*****g. And I said ratios, ratios, japan, revenue over global revenue, x over global revenue.
Speaker 3:Anyway, is that how you can summarize all of economics ratios, baby? Why run a regression analysis when you can use a ratio?
Speaker 1:Exactly. Is it a regression? Seriously, I'm not even joking.
Speaker 3:No, it literally is. If you really get into it, it's just what, is it RSS over TSS or something s**t like that?
Speaker 1:So then we were also putting games on Kindle and stuff. And I was like, look, any, why are we doing this stuff? This is peanuts. And then somebody said, what should we be doing? Then David and I said literally anything would be better than this. And I got a hold of Steve Levitt and Vincent Ductham and run a project with TGG. So it was Steve Levitt and Daniel Cohnamon's company the greatest good and I sent over for consultants and Steve also came and went around on tour with the place and talked about our results and stuff and he proposed a bunch of experiments. He went and looked through all the data and, like with the consults, we looked through the data and stuff, trying to figure out like what we can do. And this was one experiment that really like that we got out.
Speaker 1:That was big enough and at the time you really were working inside of King each game team. It was a really important thing to King that each game team decided for themselves what they were going to do. Of course, you're working at a big company, so the COO tells you to do something you're probably going to do. You're probably not going to say no, but in theory each backlog was owned by the game team itself, but it meant that the way that things were really working is like Candy Crush would do something and then it would work and because it was big if they would just tell everyone else what to do and everyone else would just do what Candy Crush was going to tell them.
Speaker 1:So at the end of this Steve Levitt thing and he'd done this paper we'd gone around. I started trying to convince people that we should do is we should do all the crazy stuff in the small games, take all the risks there, and if something works in the small games, then we can take it up to Candy Crush where it'll have a huge multiplier. And I remember having like serious conversations with NBA types and they were like they're never going to convert from the small games to the big one. And I was like, yeah, but I was like, but they convert from the big games to the small and they're like, hmm, not the same thing.
Speaker 2:Pretty sure it's exact.
Speaker 1:And then like those same people, like only six months later, once we'd had a couple big hits, we're just like you know why this works, david, because you're. And then they're like this lecture on optimizing your opportunity cost for the big games and then put it in a PowerPoint, send it around. Sounds good. But then we just started running these experiments and the original chart we had like the requirements I had with the game teams was it needs to be as impactful as possible, because your games are too small. So if you do something, it's 1%. We're not going to see it.
Speaker 1:So you need to be like 10% or over is the only place. So, like I had this one arrow, this way, it needs to be 10% or old and then it needs to be as generalizable as possible. So if you're working on like a clicker or pyramid saw it was a solitaire game. If you've solved something in solitaire, yay. But like we can't do much with it, right. If you give me something at the time I use this example if you give me something that if Twitter found out about this, they'd be like oh shit, that's great. So over 10% and extremely generalizable, anything that you can max out on those two is on the table do some damage, and that's how we kicked off the experimentation group.
Speaker 2:And so you run this paper. You run this experiment quality discounting. You're holding the price of gold bars constant. You're increasing the amount of gold bars you get at each of this priced skews and the results come back. And I remember I met Steve Leavitt. I talked about this paper. I went out to Chicago to talk to Leavitt about this paper and he considers it one of his few failures that he likes to open his when he has people outside of academia. He likes to open the. Hey, if you come to me, I can make you more money. People don't believe in economists. I've done it. Look at my tracker, except for one, and he talks about this paper and how he failed. And you are not able to increase revenue even 1% by doing massive quantity discounting.
Speaker 2:And not only that. The result that always blew my mind that gets very little conversation in the paper is that there was no conversion of non-payers to payers in each of the experiments. So you would change the amount of gold bars you would get at, let's say, $5 by as much as 60%. You would get that much more value, and you couldn't even convert 1% of people who are non-payers to payers on coin discounting alone. That, to me, was a mind blowing result. He considers it a failure. Do you think of it as a failure? Was that a failed paper, a failed experiment?
Speaker 1:No, it opened up for at least for me and I think probably for all the people who worked with 10 failures only crappy if you didn't, if you already knew what the outcome was. If you fail because you're trying something and it's way too marginal, like it's not going to have a big enough effect, that is a proper failure. You should like punish yourself and then, because then you knew that you weren't really having any serious. I think as long as you're actually doing something that can you think is going to have a huge impact, then the failure is just more data input to the next iteration. So I don't consider it a failure.
Speaker 1:I think one of the fascinating things with this is at the time we thought this would be interesting. And now, later now you just look at piggy bank, which and battle pass and you realize because this is fascinating, right, when we put in piggy bank in the game conversions like through the roof, just like, oh, we're like, holy shit, tell us what a piggy bank is. Piggy bank is like mini, mini battle pass, right, you're like playing and you earn gold coins that go into the piggy bank and then you pay five bucks and you get. That's what the mechanic is. It was originally in a game. Oh shoot, what was it called? Like an egg game, egg and senator or something like that. An article came out on one of the game websites and everyone read it and we had this system at the time where you could put in everybody in the company could put in requests into the into the. We called it the hot list was like a list of what we want might do, and that article came out literally. I think five people put in that we should do this test to test piggy bank. And so the fascinating, the really interesting thing it's easier to look at it from battle pass.
Speaker 1:If you want to get conversions, you just mess with prices the people that think it's okay to pay in the game you mess with prices. You might get more, you might get less. It's the same as we were talking about with the app, like it might change a little bit how much I pay or whatever, but it's not going to open new payers. But all of a sudden, if I feel like I've earned stuff, it changes the way that I relate to this particular package. It's no longer like just cheating, it's this other thing. I've earned these things and these earning mechanics. They work without question and they change. They change the entire like relationship for paying for a game.
Speaker 3:They're stakes, they feel invested. I feel like I'm already 90% of the way there. David, I wanted to ask a question. We had Julian Runech on the podcast last episode. It's incredible that we have you here, because we talked about this paper and that episode and I wanted to provide his critique.
Speaker 3:I don't know if it's a critique, but at least the reason why, he says his paper is different and it comes down to we're talking about conversion, we're talking about people on the margin, and his big argument is the effect or the sample that you guys were actually, I guess, the intent to treat or whatever. If you were in a labor economics class, you would talk about the treatment effect on the treated, the treatment effect on the if there are any labor economists I'm sorry like it's been a long time since I've looked at all those terms, but basically the idea is you have the whole population or the sample who might be getting the treatment. There's only a select few of those individuals for whom the treatment is actually relevant. These are typically the people on the margin and the argument is that your skew, that you didn't even start testing this lower skews, the skews that would perhaps be the entry point price tag.
Speaker 3:So yeah, so I think you started 100 gold or something like that, but the lowest skew isn't actually. There's no discounting there, and I think from a mathematical point of view, that was probably like, for that was probably for methodological reasons, like you needed a solid control or something like that.
Speaker 1:No, it was just technical. I don't think we had. You couldn't lower the price because then I think the lowest price point was an actual price of a five extra moves or whatever. So there's no. I guess you could have given them two or something for it, but it was like at the time it was like we were like that's the bottom, that's the bottom one.
Speaker 1:I think they're naming me a technical reason, we couldn't do, or whatever. But I also met, he bought me or we went out to drinks here in Malmö, and so I've heard this critique.
Speaker 3:Oh nice, nice, nice Physical waves hit me yes. Yes, and I've got, so 10% of the purchases that happen are for that, greater than 99 gold bars, and so that was where you were testing. Was that it, one could argue, and Julian argues like that a top 10% tiles? What about all these other people who aren't being impacted, and how big of a difference do you think that would have had in your results if you had been able to somehow access that bottom 90% percent?
Speaker 2:I completely agree. I feel like this is such a bizarre critique from by him because it's we're talking about what is the relevant group to be treated and the people who are making those extremely small purchases compose such a small percentage of total revenue that the people that you really want this experiment to get some juice out of is the people who spend a lot of money, and those people tend to spend far more on average on a particular skew, like they're buying on a regular basis $25 skews or $50. They're buying more price point. They're prying higher price points. So to me, it's far more important that we understand what's happening to those users rather than the people spending $1 or $2. That's not interesting and free to play context, in my view.
Speaker 3:Phil, you're talking about intensive margin versus extensive and I think that's like, I think, to criticize this paper and say, oh, it's no good because it's not exploring the extensive margin. We don't know who's converting. That's not what the paper's about. The paper's about, like you're saying, the intensive margin. For these people who are spending. Can we get them to spend more? And it looks like no.
Speaker 1:It's. I think that we know way more about this now. So we've done a lot more tests and this was nowhere near as nuanced. It was pretty. This test must have happened in two, three years or so. Much better Maybe we. I think we did it then I don't remember the exact date of it.
Speaker 1:Another just side story one of my favorite conversations with one of the leadership people also. They said, david, you called me up and they were like this paper that you're writing, are you sure it's okay to publish it? I'm like I'm sure it doesn't give away any of our secret sauce? And I'm like, if secret sauce is failing miserably, yes, otherwise don't worry, dude, no one's gonna guess, is not? This is academically interesting. It is not commercially interesting, don't worry about it. It's only commercially interesting for the gaps that it leaves. And the reality is that the big opportunity was still at that point a small amount of people paid. We know that actually their pain habits. They're not improved by discounts. It's not driven by discounts. Most of the pain habits are driven by your experience with it, your short term, midterm, long term goals of the game and how you relate to them and how you intrinsically value it, and those things can be affected a lot more by the way the game is structured than the pricing stuff.
Speaker 1:And at the time the game was underpriced. I can tell you that without question.
Speaker 2:So there's another. We can get into the actual experiments. I don't want to do that. That's not what we're here.
Speaker 3:Just do what you want, so follow your heart.
Speaker 2:What is the learning that you take away from this in terms of external validity? Is it just that, hey, we can't do quality discounting or hey, we don't want to fuck with prices? How much are you willing to? What is the scope of learning that you take away from some of these experiments?
Speaker 1:You actually. I think it was on deconstructor fun, right, it wasn't on this podcast that you were talking about the Door one, door two and door three.
Speaker 2:So just to set that up for listeners again, fortnite made some changes in its pricing structure. What they chose to do was hold the amount of hard currency you would get constant and instead increase the USD price of the SKUs. So it's the exact opposite of what you did in this experiment, where, in this experiment, you held the USD price constant and you increased the amount of gold bars. There's actually one more scenario you can consider to change price, which is to hold the SKU prices in real world currency constant, as well as the hard currency that you would get at those SKUs constant and instead change the virtual currency prices. So there's three different levers, all with different costs and benefits that you could pull to fuck with prices in a digital game.
Speaker 1:Okay, wait, don't say anything else. This is really exciting. We've got some fresh blood here. Chris and Eric, Okay, so you've got those doors to walk through. Which one do you want to walk through?
Speaker 4:I'm 100% in. Door one the one for night shows you change the dollar to V bucks, increase the dollar amount, leave the V bucks quantity sizes the same, because you've tuned your skin bundle sizes accordingly. And, very importantly, this allows for flexibility across countries. If you want to Brazil, seeing hyperinflation or whatever, you can adjust the price there or not anywhere else. You don't have to worry about weird discrete effects with the price sizes and it lets your in game pricing, the people who price V bucks to content, lets them be basically independent of what you're the decisions are making on the pricing side.
Speaker 3:That's fair. I think, like for me it would be. Typically, when I'm doing an exercise like this, I always think about, like, where is the worst in the player's mind, the final sale? Is it when they purchased the SKU or is it when they go to buy battle pass for that season? And I think, especially in the case of Fortnite, I'd be more upset if I found out that the cost of battle pass had gone up. So that's door three, correct, phil, where the actual costs in the game are shifting around. I think I'd be most frustrated by that because I have a huge, I have already a mental model of how many V bucks do I get each season if I play through the battle pass and how much is it going to cost me next season. But there's two different purchases, there's two different points of sale here where I'm buying the SKU and then I'm also buying the battle pass, so I don't have a super strong. I think. Definitely the second door sounds like the worst to me.
Speaker 2:I agree with you, chris. I think I'm going to walk through door number one with Eric. I guess I'd also add that I think the effects between all these different experiments if you actually got all of the equivalents right, so it was a truly clean experiment between all the different variants I would argue that probably the differences between all of them would be again, that's some pretty vague terminology. I would say I'm very skeptical of doing anything that creases the friction of a player having to make a real world purchase or having to go back and make a real world purchase again and again. If you have to do that multiple times over a given period, that to me is points of friction. Your credit card could be out of date.
Speaker 2:Going through the first party SKU process is rather abhorrent. It's not as easy as it is spending hard currency in a game. And to me, when you have door number one, when you're raising the price of a SKU but you're holding all those other things constant, the schedule with which you purchase SKUs can also be the same as it was previously, because you're getting the amount of hard currencies the same. And so let's just I gave this example in Deconstruct your Fund. But let's say you buy one cosmetic a month.
Speaker 2:If I'm just raising the price of a SKU but holding prices and hard currency and the amount of hard currency constant, then the amount of times I need to refill on hard currency is going to be the same, whereas if you fuck with door number two or door number three I might have to go back to the first party SKU again and make another purchase. So in door number three, if the hard currency prices increase, that means I have to go back to the first party SKU potentially two times to be able to buy that cosmetic or potentially what I need to do. If the hard currency is less in door number two, then I also need to go back to the first party SKUs. So this introduces pain. The only reason I'm skeptical of door number one is because I think people are more sensitive to prices that are in world money than prices on hard currency. I think they have more heuristics around them and more comparative pieces of information.
Speaker 3:but I would still probably narrowly give it to door number one for the pattern In addition to that final point you're making, perhaps in favor of door number three, I would add this final piece to door number three, and it's that people are terrible at discounting into the future. So I think I probably.
Speaker 2:Oh, hyperbolic discounting. I'll give it to the behaviorist, for that one Exactly, I'll give it to the behavioralist?
Speaker 3:No, no, no, no, not hyperbolic discounting. This is pure risk aversion. Enter temporal discounting into the future. I just there's some ambiguity with the future. I discount future value at some interest rate basically, and maybe it's 3% or something like that. So I see that future virtual purchase that I have to make it's less impactful to me than money now. So money now is more valuable than money later, and that's why people require interest when they lend people money. So to me it's pure theory. You don't have to go behavioral for that.
Speaker 1:I think they Interesting. So for me it's much more in the game world. If you change the price of a thing like if a gun or five extra moves cost this arbitrary fake thing they cost 10 of these fake things for and now they cost 12. I feel like there is something. It's like I've been cheated, there's something that doesn't. It's like this is actually like people get mad when the price of milk goes up and it gets discussed like the price of milk.
Speaker 1:I think that there is more of an emotional bond of the thing that I'm buying and then I think the actual, my relationship to purchasing the soft goods or whatever it is, outside what I'm using, real money. I think that is much more of a budget question what's okay for me to spend, and I think this is in like grand numbers. This might be more casual than it is mid-core, but I've seen, if you look at payer behaviors over huge numbers, there's a lot of people that just pay five bucks, five bucks, five bucks, five bucks, five bucks and that's what they pay. And I think that you'd think I've looked at this data and been like just buy the 50-pack. But I think if you interviewed them you'd hear people saying I only want to spend five bucks on this game and nobody says yes, but how many times and how often? That's not the I spent five bucks. So I think that there are some sort of like methods going on in, like how I think of my money, and if you change what's in the bundle, the best thing to do is make the bundle still look good, even if the actual thing that's like the currency or whatever that you get for it is less. But there's some other bonus things that you know have less utility or whatever that'll work. But I think that there's a difference in relation, emotional relationship to these two things and how I see it.
Speaker 1:Also, coming from online poker, I knew a lot of, so there's a while where I could see the data about a bunch of players. I switched from one company to another one and I knew a lot of people that played on the platform, and so I knew what their outcomes were like. I knew whether they'd won or lost money and how many deposits they made. And I asked you guys up or down, how's it going? And I can tell you every single one of them had themselves higher, closer to positive or positive, even though they were negative. Nobody hit the estimate right and certainly nobody underestimated, and I think it's a similar thing you purchase a few times. If you ask somebody how much have you spent on this game, they'll be like a hundred bucks.
Speaker 2:So let's adopt this. Let's let's play in David's world for a little bit. Let's call it the budget hypothesis. If we assume the budget hypothesis is true, I think there's some testable implications or some things that should be true empirically, one of which should be that there's little difference in spend velocity for a given spender. So if we were to, if someone had to query a database and they were looking at the rate at which they make purchases, they should be fairly steady. It's not like they're going to drop a hundred dollars or an increase. Even I might actually throw a little bit of cold water. I think there's some information. I might doubt that theory. I think there's going to be some information gathering. People are doing as they're playing games, I think their favorite they're dating. They don't know if they want to get married yet and I think that relationship changes and at some point, like they do decide to get married. There's some information gathering they're doing very early on. So I might throw that.
Speaker 1:But it's testable. It's a testable hypothesis.
Speaker 2:We have ways to. What do you mean?
Speaker 1:that they would spend like a little bit of money to and then see if I like the game before I start moving off and spend more.
Speaker 2:Yeah, I don't think they understand their time horizons yet. And so when you get into a game, if we think about some things that you would purchase in some of the virtual worlds, the costs or, excuse me, the benefit of that item that you could purchase is amortized over your lifetime of playing the game. And so the longer I play a game for some items, the more the value of that item increases, like a battle pass for instance. That thing increases in value the more I play. And so I think players might be uncertain about their time horizons when they download a free to play game. And I think they figure out very quickly because we determine who's a spender and non-spender. Usually within the first two weeks you can figure out like 95%. So I think they figure out their time horizons very quickly.
Speaker 2:But I don't know if it happens on day zero, but I still think there's probably something to your point. But if that were the case, if it's really a budget decision, if budget is almost exogenous, the decision to spend a given amount of money is almost determined outside of the game. So there are casual players and they're mid-core players and they have different budgets, then really the goal of free to play games for us is to maximize retention, because we can't really fuck with anything else. We just need to fuck with retention and ultimately extend those budgets over a longer period of time, rather than trying to get people to change them. That is our easiest lever to pull.
Speaker 1:And also if you buy in a package and it has less gold bars in it, at least in one of these games or whatever, and product costs has been the same. So my package usually when I spend 10 bucks I'm getting less stuff for it. I run out of money quicker so I might actually spend more often now, so the spend might go up. I just think that people don't have very good. They're not keeping track of how often that spend goes up or goes down. If you ask some people just go and ask 100 people how often do you buy candy bars? Do you think there how many of them you think nailed the number or like underestimate, or maybe they undressed them?
Speaker 2:They go like a few times or whatever but would you stand behind your budget theory for both cosmetic based economies and more mid-core games like 4x, or do you think this is specific to something like casual?
Speaker 1:I don't know I wouldn't say I don't mean budget, a reasonable, reasonable amount, like I'm saying, if you like the rain it's more.
Speaker 1:It's more like that. If you said to me like how much is reasonable to spend on this, I might be like five bucks. This is no, I must spend. I'm not gonna spend the $20 one, but I might buy the $5 one a couple times as I hit pressure and walk the thing to move forward when somebody else might be like I'm happy to spend in 2000 bucks in this game because I'm not. So. I think that there's like a. It's like a how much I'm willing to put into the machine without feeling shame? I think it's more like that, not budget necessarily.
Speaker 4:Tell me about the experimentation group. I love what you guys are doing and I'm not politically very active, but one thing I really care about is I wish games and tech companies would share more data for the sake of science and Advancing our understanding. I think we as an industry can make better games if we collaborate and share more data and teach each other things. So I love what you guys are doing with experimentation group.
Speaker 2:What's your address? Where do I need to spend the 20? The check, eric. What a fucking layup man. It was really join us, had his brand marketing.
Speaker 4:Wow, yes, that's my contribution to the movement.
Speaker 1:I think I think there's a whole bunch of stuff that's interesting and I think that you guys I'd love to have minds like yours looking at things that are going on in games and Actually get more ideas, because I think there's way more left to do in mobile games. There's way more interesting things to test, there's way more to try. A lot of people are just in the same, like lane, doing the same stuff, but they've always done just borrowing from some other things and plugging them in, and mobile games Is it's the quickest place in the world to go from like an idea to a thing in front of like a million people. Why aren't we just doing a bunch of interesting things and trying to find out what happens? No platform ever has been able to deliver something to end customers at this size, at this speed, with data ever. It's insane. So let's start actually figuring out what people give or shit about and really discover what really matters to our players.
Speaker 2:Yes, that's our tagline. We need to get that in, david.
Speaker 3:Yeah, that's what it is. Yeah, yeah, I was gonna say I Does this lead into what you're doing now with?
Speaker 2:so for listeners who don't know, by the time this podcast drops, david, myself and Tom store, who is the product lead over at King with David, will be launching the Experimentation group, very similar to what David had at King, also called the experimentation group. You'll find a link to the experimentation Group in the show notes. We are trying to get companies to run experiments with us and we will do the experiments with the company for free. We will help grow your game. We think that we can use the science and the experimentation playbook to grow games and ultimately make products more profitable, and we will be doing it for free. That is right, for free. But the thing that we are going to do, the twist here that I think everyone listening to the game economist cast is gonna Love is that we want to do it out in public. We want to do it out in the open, as long as you're willing to publicly share Some of the experiments that we set up when we run them, the results. We hope to get a public conversation going about what we're doing and how we're doing it. And so I know all you game economist castellers, I know all you game economist discord members You're gonna be in those comments telling us we're wrong, and I honestly, god hope you do, honestly, god hope you do, because that's how science happens, and every time I hear supercell doing something in Finland, it's like they're sharing their dashboards. Like, I'm pretty sure, if you have a finished ID card, you get like access to Supercells dashboards. They're extremely public and open about what they're doing and I feel like there's so little of that in the game industry and it's time to show everyone not only how you do science, but Also how you have a public conversation about games, and we've been lacking that for so long and I'm really excited to get this started with David and Tom. And if you know someone who has a casual game and has over 10k DAU and is able to run an experiment, let's get in contact, let's talk about it.
Speaker 2:I'm sure they're gonna have a million questions. What does it mean? Things have to be public. What are you talking about? How much has to be public? Do we have to release the Excel file? There's a lot to. There's a lot to talk about and we're here to talk about it. So reach out. We'll have a Calendly link as well if you want to book some time with us. All right, sorry, go ahead, david now.
Speaker 1:I got the pitch in. No, I think it's a really interesting reason for companies to do this, for game teams to do this as an Opportunity to run a test. Get together with get together with us will propose a test. The results will be public. But as soon as the results are public, amazing people like Eric and Chris, you guys, are gonna be able to look at them and be like, wait, why didn't you think of this? Very? Or, if that's true, what about this? So it means that the single experiment you ran that you want to keep in your little house I'm not telling anyone about as soon as you let it out, you have a world of smart people. They're gonna start talking about what they, what else you could do and other ways to think about it. So I think it's gonna be a big talent magnet for teams that want to try this.
Speaker 2:And I think the other thing we were talking about earlier today is that their knowledge spillovers in games, even if they're not public. Employees go from place to place and they take knowledge with them and then they spread that knowledge at those other firms and then the people at those firms spread that knowledge elsewhere. That's how you end up having all these agglomation effects, different parts of the world when you make games. And I know, david, a lot of the learnings that came out of the experimentation group at King are already in all of these companies and a lot Of King employees have not gone to these companies yet. Somehow they figured out about them. It's those knowledge spillovers, so people are concerned about doing these things in public. I guarantee you it's already happening. It's already happening, it just hasn't been accessible to people and it's time to make that accessible for all of us. This is a rising tide that will lift all boats.
Speaker 3:I'm gonna, I'm gonna be the greedy monetization person here. Are you guys off-setting? Are you pulling an Uber and off-setting the costs right now for like an Inevitable price hike in the future? Is this just pure like what?
Speaker 1:yes, we're on an economy. What are you talking about?
Speaker 3:Is this? No, it's not about, it's never about science. We're. This isn't a cop, as we cannot possibly. Okay, I mean, I can.
Speaker 1:I can say this Chris that I really like disruption and I have sacrificed in my life cash For the opportunity to disrupt things on a regular basis and it's not a problem for me and I think I got a track. I can show you pay slips if you'd like to prove it, and mispromotions and whatnot. And I know actually that Phil this is like indeed, from the first moment of contact with Phil, he was okay, this is a conversation with Phil. I don't know if this is how I remember it, phil, our first phone call. You're like so wait, how are you doing these experiments? I'm like we just make the change and release experiment. Okay, okay, but how big are these? Like these treatment groups, is it like you got five percent? It's in the treatment or whatever, and not 50, 50, what at 50? 50, otherwise you don't really get to see the signal. Wait, you just put 50 people, 50% in the treatment. How big of these games? Are they? Like 10,000 or something like millions of Dow? It feels just like this.
Speaker 2:Oh, and it just melted. It was a glass of ice water to someone in hell. It was not running a B tests, it nothing. Yet they had nothing. It wasn't Zynga at scope they. There was so few a B test being right. In fact, I don't even think we were at a single a B test in the time I was at scope Lee, and to hear that someone was doing this was incredible.
Speaker 1:I'm just as a pitch, for Phil really cares about the science in this. I think that you've you're running into zealots here. These are not normal economic actors, sorry.
Speaker 2:Chris, trust me, I have a lot of ways I waste my time. I can tell you this podcast ain't making money. Yes, that's definitely true.
Speaker 3:If it is, we don't know. No, I think this is really interesting, this radical movement. I definitely like it. One thing I want to touch on, since we've, in this conversation We've mentioned John lists name. He recently released a book called the voltage effect and it's all about how to make sure that ideas are scaling properly. So the idea that, hey, this looks like a really good idea, or even our a B test show that this is going to be great. But once we implemented, all the sudden it actually shifts the prior, it shifts the entire distribution in a way that, in the long run, isn't necessarily good for the. You know, I think about if you were to run a promotion but you get a result that said, hey, the promotion brings in more money, but you can't run the promotion forever. It doesn't. That's not how it works. Promotions aren't scalable, most promotions aren't scalable. How do are we worried about that? Do we care about that? Are we just throwing caution to the wind? Are we saying, fuck the voltage effect and just?
Speaker 2:let me double that, david. Have you ever replicated an experiment? Have you ever run a replication test? Have you?
Speaker 1:what do you mean? Do you mean like we ran a test and then ran it again other places? You mean in other games? Yeah, and all the time, everything that we did, every I mean we're passing it. We're passing it. No, we were passing it between games, of course. So everybody did it and we have extreme arguments on the different impacts. But so replication was constant. The battles were constant the what is the? How was it done? What's implementation like? What is the rank? You know, but how often is it shown? They took a lot of our stuff and made it live ops and say I'm always on, and then there was a battle about why Got it? So same replication is trigger age.
Speaker 2:So hold on, I was rude. I was rude about Chris's question. He is right about the voltage effect question. I don't know if you've read John Liss book. I actually it's quite an insightful piece. You should actually check it out, it's think what percent is really interesting?
Speaker 4:I love the first half. Yeah, not gonna lie, they had me so this is an economy by? Oh no, I wasn't joking, but I see the joke. Okay, the thing you said about game designers wanting all players to have the same experience, I think is a really I don't want to call it like a moral debate and what counts as the same experience and all that. Yeah, that's what I think a great, and there's more how to convince your designers to let you run a beta, and not only that.
Speaker 2:I mean David, and one of the things David had mentioned to me a while back is that game designers make the best Experimenters, and I found that to be true. Once I've started to talk about experimentation, you have to flip them to your point. David, you also mentioned this too. But when I've gotten them to like really sit down and think about these type of models, they understand the player experience in a way that no one else does, and I think like they get at the heart very quickly about what if I change this. Like they're extremely intuitive about this.
Speaker 1:If you guys, I'll join anytime and you can even just have me out, bring me off 15 minutes and cut me off so you can talk about intelligence stuff or whatever. But just a really interesting what we had these whatever new game design people would join the company. They would come and they would. So we had this process. We could cover this amount of time. Also, we had a hot list. So the idea was you needed to keep game teams autonomy, not just because of the political game teams need to choose themselves, but a game team that wants to do something does it way better than a game team that's forced to do something. So basically, what we had is we had like a peak. We have 10 game teams or something like that, working from Seattle and Berlin, london and all of Barcelona, all over the place.
Speaker 1:And so we'd have this list, like we called the hot list, and it was like 10 ideas or something. Sometimes it was 20 and it was like these ideas we think are interesting. We had another process which was like how to get onto the hot list. Anybody in the company could put in ideas and we would we'd look at it and sometimes it was something we'd already tested or that wasn't part of our strategic priority priorities at the time or whatever. So, like we'd have an open meeting, people would come, we tell them how we chose and so far, the amazing thing is, every time Creative people showed up and it was like it got to the point where it just pained me. They'd be like you know what I think we should do? If you take pricing, and I was like price, jesus, like you're a game designer, where's the creative? And they're like, yeah, price, if we can get them to convert.
Speaker 1:And then you'd get these people from the NBA would show up, these, these people that have been to a Boston Consulting group, and they'd show up and they'd be like what if Game board pieces had faces? And you're like why are you bitching these creative ideas? It was just the strangest thing. It was just such a bizarre. It was like they just crossed wires. But just like you're saying, phil, every time you got one of the game designers to want to find something out about what really matters to your players, do you want to find something out? Then go ahead and pitch something drastic. What if all their friends on the map were gone? Would they care? And we're like I don't know, would they? We don't know? And then we tested stuff like that and it was just fascinating. So every time you flipped one of them, they'd start to ask tough questions themselves.
Speaker 2:Let's wrap. I wish we had another hour. Yeah, I think we'll definitely have you back. I love you guys. It's been really cool.
Speaker 1:It's great. I think you guys do great.
Speaker 2:I love your podcast.
Speaker 1:Thanks for doing it, guys.
Speaker 2:Yeah, and the rolling product of you guys. Listen to that too. You had a great episode on monopoly go, and you've covered social networks and app stuff, which no one else is talking about. I got in to be real because of you guys.
Speaker 2:No failure and it's an ability to grow. Yes, and I was. I actually be real before we came on in the podcast and I'm like, oh wow, I really get what they were talking about and I would have never been exposed to that if I hadn't listened to rolling product All right, cool beans. We should teach this to our children. Economics.
Speaker 1:Everyone has to major in economics. Number one for personal survival by economics.
