Game Economist Cast
What does the new wave of open economies mean for monetization? Will negative externalities overcome cosmetics economies in the long run? What exactly does a game economist do?
Game Economist Cast is a roundtable discussion of the latest developments in mobile, HD, and crypto games through a bunch of people figuring it out using the economic tool kit.
Game Economist Cast
E30: The Economics of Game Development
Eric develops an economic model to explore or exploit game development decisions, while Phil wants a block-grant style gate process to align incentives. Chris is back at Marvel Snap, and boogies with some new social casino mechanics.
The team reviews a new Call of Duty matchmaking paper with some surprising and revealing data...
Don't say that on air.
Speaker 2:I don't think you're allowed to say that. I think it's trademarked or some shit. Let's get ready to rumble. Or maybe it's only you're not allowed to say it in context or something.
Speaker 3:Is that WWE Send the YouTube censors after us? Let's start with utility. I don't understand what it even means.
Speaker 1:Everybody has some kind of utils in their head that they're calibrating.
Speaker 3:There's hardly anything that hasn't been used for money. In fact, there may be a fundamental problem in modeling that wouldn't want to model. It's episode 31,. Game Economist cast semi-regular, as always. Hi, it's Phil from Game Economist Consulting. I'm joined today by my 200 wonderful co-hosts. Eric from Second.
Speaker 2:Den. Hey, how are you Doing good?
Speaker 3:And Chris from Star Atlas how are you second day? How are you doing good? And chris from star atlas how are you?
Speaker 1:oh, I'm doing good too you coming to europe soon too. Yes, I think I'm gonna have a like 12 hour debut in gamescom and then I'm gonna leave, so I'm gonna do a couple of coffees with people finally made the decision we were going to do three days in luxembourg. We decided to cut our three days short and we're gonna go to the cologne, but I'm not getting a ticket, so I'm just going to chat with people in the city you're going to have one of those little beers they have in cologne they have little beers in cologne?
Speaker 3:oh it's, they're very cute. You'll see them everywhere. They're adorable. They and the waitresses carry them the all the same glasses. They're all these little sizes and they have a very specific serving. It's imagine having six hot dogs being ketchuped at once. That's how they they fill the fear classes hot dogs being ketchuped it's kind of like an udder, a beer udder we're gonna.
Speaker 1:We're gonna be coming from belgium, so I'll have, I'll be very hopefully I won't be sick of beer by then impossible, see, impossible.
Speaker 3:We have three, three wonderful topics. No, no, we have two. No, we have three. We have three wonderful topics. No, no, we have two, no, we have three. We have three wonderful topics to talk about today, and one of them is an article we're sticking to it or a paper of sorts. I'm going to be talking a little bit about Lilith Games and one of their very interesting employee incentive models where they spin up a subsidiary when they launch a new game and that subsidiary, the employees own 20% of it. And, eric, what are?
Speaker 2:you going to be talking about? Yes, marvel Snap just released a new game mode called Deadpool's Diner that draws on social casino mechanics, so this is my first exposure to them as a player, so I want to talk about that. Yeah, it's pretty compelling Number. Go up.
Speaker 3:And then for our final paper and topic, we'll be talking about a new. It's not peer-reviewed research, but it's a white paper from the Call of Duty team. Actually, we know an economist that worked on this paper, or at least we think we at least know one who we talked to. They have been doing a matchmaking series and publishing their research publicly, which has been fantastic to see, and it is a treasure trove of information. And then, of course, there's an AB test that's in it as well on the role of skill matchmaking, so we'll be taking a dive into that.
Speaker 3:Before we do talk about what we've been playing. Got a selection of good things on sale.
Speaker 2:Stranger, evo happened, the big fighting game tournament, about a month ago. It's like the Super Bowl, I don't know. It's more like the Olympics of fighting games. They've got a whole bunch of different games there and I've been playing a bunch of Street Fighter Third Strike, not Street Fighter 6, not the new one. Third Strike is the arcade game that came out in 1999 and they featured it at EVO this year. It was a throwback game. They do that every once in a while. It was super gesture to the crowd and do all this crazy stuff after every time he won. But yeah, I've been playing third strike. It's fun.
Speaker 2:There's this emulation platform you can play on, called fightcade, where you just it downloads the emulators, downloads the roms and you play against other people on the internet. It's cool to see technology bringing these old games back and there's like a. They're simpler, they're more pared down. It was a simpler time in 1999. I've been enjoying it is evo in japan. They have an evo japan, but the one in las vegas is like the big one this number just doesn't seem to grow.
Speaker 3:I'm trying to find some evo viewership numbers and it looks like it's been largely stagnant for a long period of time, which is my view of fighting. It's the same as racing it like just sells the same numbers every single year. Yeah, that sounds about right.
Speaker 2:I know the number of entrants hit a new record, but it's not growing like dramatically, it's just like gently moving up. I think street fighter 6, the new release last year, generated a lot of excitement.
Speaker 1:But yeah, it's like you said, it's a very hardened audience, I would say so, like new games do have a small effect, but not like a huge one yeah, and they don't grow the fighting game genre that much.
Speaker 2:It's mostly the existing audience that moves between games.
Speaker 3:This is why I don't understand X2KO, the Riot fighting game I remember you've been talking about it beforehand where it's meant to be played with a keyboard rather than necessarily having would you call them an arcade stick controller. But I still don't understand. Okay, they're going after a small pie. That seems like a bad idea for a company that needs to generate probably nine figures worth of revenue.
Speaker 2:To be honest, I think it's mostly a prestige play. This was before they were vetting games for their market potential. It was more just like what games are prestigious Magic, the Gathering and Street Fighter? Let's make our own versions of those. Their hope is that it will massively increase the audience somehow, maybe by accessing China, which has a large gamer audience who does not play fighting games. They're hoping by being a PC first, like you said, keyboard first, fighting game online first fighting game. That also might expand the audience. Pretty skeptical People I know who've played it say it's very grimy. It's like one of those old fighting games where you can do just all sorts of messed up stuff and just put your opponent in block for 20 seconds on end if you know how to do special tricks and combo setups and stuff which is not very new player friendly. But yeah, that's their shot at it.
Speaker 3:Why have we never gotten a hot take from you, Mr Super Smash Brother, about multiverses? I expect you to come in here with hot takes.
Speaker 2:I don't know what do you want me to say. Smash Bros is the king. People thought that, okay. People looked at platform fighters and they were like, oh, smash Bros is big, but there's only been one incumbent. And normally when there's one incumbent for a long time they get fat, they get lazy and it's like an opportunity for someone new to come in. But Mashihiro Sakurai does not slack on the wheel. All right, that guy is a workaholic. That guy makes the best, most polished games. All right, and every Smash Bros has been a phenomenal game. They are not getting fat, they're not getting lazy. Smash Brothers is king for a reason.
Speaker 2:It's not just the IP. You can't just stick a bunch of Warner Brothers characters. You can't stick Arya Stark and Bugs Bunny and think people will play your game. The game has to be good, and multiverses is not good. It doesn't feel good to play, it doesn't hit. Good to play. There's good to play like this, just it. Yeah, there's a lot that goes into making a good game besides ip and smash bros knows what they're doing, so you did have a hot take.
Speaker 3:Wow, just ask, eric, about platform fighters. There was the sony one. Remember? Sony tried to make a super smash game playstation all-stars.
Speaker 2:Yeah, game was hot garbage. There was the nickelodeon one. It was like all right, there's another one coming of that too.
Speaker 3:There's another nickelodeon game they announced I was surprised it was a paid game too.
Speaker 2:It wasn't free to play that one was made by an indie studio who you know. The game was fun, but it's. You're dealing with triple, a polish, by a very dedicated game developer who's known for hyper polishing and stuffing loads and loads of content into his games with the the best ip from video games you could imagine. Yeah, pretty hard to compete with that. And switch has a huge install base. Like what you're gonna say oh, my game's free, it's not gonna get you that many players so what have you been playing?
Speaker 3:indie darling. What's what's on the tap these days?
Speaker 1:no board games, although I've given a surprising number of board game recommendations this week. But I've been playing marvel snap quite a bit. I played originally over the span of the first month when it released, put, put a good number of hours into it, but then I fell off. It was around the same time I was playing Brawl Stars and Brawl Stars just took the limelight. So I came back to it because I was like OK, I really want to dig into this game. I have friends who still play it, still enjoy it, and then the whole Deadpool thing. So I've been playing this shit out of it probably hour or two a day.
Speaker 1:And the one thing that I still am a little bit frustrated with is just the collection levels. I know you can technically buy collection levels by buying boosters which allow you to level up your cards, which allow you to gain collection levels, but it's just something that's super annoying. Like I want to own the cards, I want to be able to put together a deck and I want to be able to play against opponents and start winning. I feel really crippled with the cards that I have. I'm sure that there's some fundamental game design and maybe Eric can even comment on this. I'm sure there's some sort of fundamental game design on why they do the collection levels, but I just can't think about how well it works for Magic Arena to do the packs and their whole drafting thing. Why can't they do something like that? What is the fundamental reason why that's not possible? Why can't I go buy a $3 Marvel Snap Pack?
Speaker 3:Because Steven Jarrett gave a whole talk at GDC. He gave a whole talk about this at GDC and he talks about card shards and they rejected them. And I still didn't really understand, I wasn't really convinced. But like, how are you going to do packs?
Speaker 1:There's a bunch of issues. I understand there's a bunch of issues. I understand there's probably a fundamental problem with doing packs. I know it's a limited, limited size pool. There's a limited number of cards, especially when you compare it to something like magic gathering, there's like a tiny number of cards. I think a deck is what?
Speaker 1:10 cards, five turns like very small 12, 12 cards, like a very small pool of cards to select from. You can't do like a five seven card pack. You buy three, four packs. You've got like almost the entire collections and then like they would need to come up with a dust system, I assume. But it's just.
Speaker 1:It seems like a missed opportunity to allow players to just spend their money, because I'm the type of player who would like probably spend the money to get the cards. I'm assuming the best way to do this is through some sort of drafting, create some sort of soft currency or I guess it would be a hard currency, the same way that they have gems in magic gathering you have some sort of gem that gets rewarded and then you get rewarded with packs or something like this. But definitely it's always been a pain point for me with marvel snap. It's just like gosh. It's so slow. And then I play against a player who has the same collection level but they're playing with mostly the same cards. But then they have this baller ass card that, oh, this one quadruples everything and fucks my entire side of the field. There's moments like that where I'm like, jeez, I wish I could buy that card I'm sure.
Speaker 2:Yeah, I think what you're saying is a big and common complaint, and we even see it in the monetization, where the revenue is very backloaded compared to most mobile games. I think a lot of that is you can't buy the things you want to buy, and then there's all these layers of obfuscation with the collection level and upgrading and oh, I have to do a and then b and then c to get more cards. I can't just press one button and get more cards.
Speaker 3:Why is it like this production function like that's one way to look at is like I can't just mix my capital to get output, I have to mix my labor, and the coefficient on labor and marvel snap is really large, right, because I buy credits. That, essentially, is an opportunity for me to fuse my credits with my labor, which is boosters. So remember, you get a card, then you have to level up the card by getting enough card boosters, which is earned by playing. So I have to play a bunch and then I have the opportunity to spend credits which move me up my collection level. I can't just unleash capital to progress, I have to unleash capital and labor.
Speaker 2:It's a really good call out, right Is that? Yeah, it's the money and the you used to have to grind to get boosters. They changed it so you can just quick upgrade without any boosters in the shop because of that exact problem you're talking about, where it was like blocking players from acquiring cards. But yeah, there's just like layers and layers of obfuscation between things and I think it was in attempt to be less greedy. You're not selling things as directly, but the downside is we get a lot of early player frustration, like chris is describing about I want this card, how do I get it? There's no path, and we see that reflected in the revenue as well because there's no way to directly buy only boosters, correct?
Speaker 1:like you almost always have to buy a pack with that has the card, the avatar and then the boosters. And yeah, they always bundle it with some other shit. Yeah, I like tried to calculate, okay, how much is like each booster worth, and it was like they're varied by the different packs and I was like, anyway, gave up.
Speaker 3:It's weird to layer on not necessarily vertical progression, but like kind of this time basis to what ultimately is like horizontal collection, like kind of this time basis to what ultimately is like horizontal collection. Like I can't assemble my strategy until I've invested so much time because not only is there an RNG element, but some of that RNG is controlled. There's the series notion, which is how Marvel Snap works, not even on the back end anymore, but there's these drop rates based on what series you're in and cards can move between different series, which affects their drop rates, if I'm correct.
Speaker 2:Yeah, basically you work your way through the starter set, you get like a core set of cards and then after that it's yeah, it's a giant pool.
Speaker 3:I like that theory for the horizontal progression based game is you're going to master these different parts of kind of the strategy map or strategy nucleus. However we want to think about it. That's fun to give progression designers that level of control, Because normally, like you play Magic, it's all just random, right, All the CCGs are random. There's no progression in CCGs.
Speaker 2:There's a notion of sets in most CCGs where this set has this mechanic where you bring stuff back from the dead and this set has this mechanic, and so you focus on a specific set of mechanics how to play around those month to month. Snap does not have any notion of sets or seasons or anything that, and I think that's a scaling problem that we're gonna have to confront eventually just the fact that there's like a limited number of if you think about magic, there's like pretty much a limitless number of cards.
Speaker 1:You can play infinite, indefinite number of turns, tons of different colors, different combinations, versus snap, which is a much more constrained, smaller strategy space, or I guess not a strategy space, but design space. Is that kind of what you mean?
Speaker 2:in magic sets rotate out so you don't have all the cards you want from this old set. It's fine, here's the new set, right? Everyone's progression gets reset a little bit and you only have to think about oh, this set has this combo that I'm trying to assemble. But, as in snap, the way it works is the pool is just one giant pool and that that pool is getting bigger and bigger. And let's say there's a three-card combo you want from that pool. As the pool gets bigger and bigger, it's harder and harder to obtain that specific combination. The number of things you have to think about it gets bigger and bigger because the number of potential decks your opponents have gets wider, which seems fine for our elder, mature, engaged players, but it definitely seems like a problem for anyone trying to get into it I am a little overwhelmed.
Speaker 3:It would be interesting to see if this theory actually holds up, because I totally agree with you that on paper, the number of cards increases, the number of combinations increases, and so you have to prepare against all these things. That makes sense to me, but when we think about what the effective meta is, does the number of cards that are introduced, the raw number of cards, as that continues to accumulate and accumulate? Does the number of cards that are introduced, the raw number of cards, as that continues to accumulate and accumulate? Does the meta become increasingly diverse with a wider set of distinct cards, or is it still a stable set? That is ultimately the meta. The meta revolves around the same 15 cards, although those 15 cards may change irrespective of new cards being introduced, if that makes the designers do put their thumb on the scale.
Speaker 2:If there's some dominant strategy, they'll nerf it and buff other things, so they do keep things shaken up. But yeah, there's definitely like cards that get introduced that stop being relevant and yeah which is fine.
Speaker 1:Like I'm totally down for set rotation I. I think that would be probably good for it. But yeah, been enjoying it a lot though, and I was surprised, like how it seems like it's still a very active game, like when on the subreddit there seems to be posts like tons, tons of people, tons of activity.
Speaker 2:So two years later, right, three years later- I don't even know when it released a year and a half 22 I've been playing.
Speaker 3:I've been playing deadlock, which is valve's game which is in. I don't know, it's like a I don't want to say it's an open secret, but they've been testing this game, it looks like, for a little while now. It is an open invite in the sense that once you have a key invite, I think an unlimited number of people and I played it it against Concord, their beta, which is a first person shooter out of Firewalk Studios in London, which is coming to consoles, and it was pretty shocking to play both of these. Concord is a first person sci-fi shooter in the vein of Overwatch and it is bad. It's just this game is going to absolutely bomb. You. Look at the Steam numbers. I think they only around a thousand people play during their thousand max ccu, during their during their beta, which was on playstation plus. They need to have way more users for this to sustain it's 40 multiplayer only. So they're gonna. They're gonna have ccu problems off the bat and we know that can.
Speaker 3:That can result in like self-defeating spiral, because as soon as okay, matchmaking times are five minutes, then people leave, then they're six minutes. That triggers more people leaving. There's huge tipping problem. So they're in a bad place. But they really struggle to like nail this ip which is which kind of has a sci-fi glean, almost star warsy, like star wars outlaws, smuggler, of that nature. And then you go play a deadlock, which is valve's game, which is steampunk themed, and you just realize how much people are able to bring IP and like a feeling or a vibe to a game and Valve just absolutely nails bringing this kind of sci-fi, steampunk element. So to give you an example of how deeply they integrate the feeling, there are electrical lines that you tag onto when you go to different parts of the map. This is how they like speed players to different parts rather than you having a mount or having to walk it, and just feels very steampunky. All of the different characters have a very strong steampunky theme. One has a robot like, with funky different parts. Other characters mirror what you might think of as like a Victorian woman. They have a lot of the dress. That's the same. The sounds feel like it would be from a Victorian steampunk era.
Speaker 3:It's just a game that seems to get all of those little pieces and I've had an absolute blast playing it. I haven't played a game like this for so long. I remember I played a little bit of Paladins, which is another attempt to try to take this third-person MOBA idea and run with it. There was also Smite. Those games have done okay. They're from this developer in Atlanta called Hi called high res studios, which is one of the only I would argue like true free-to-play middle-class developers I know in the west. There's a couple in the east but they've hummed along at like maybe 500 headcount and they have all of these different free-to-play titles that exist mostly on steam. They have one that's third person goes after csgo. They have smite, which we talked about. They have paladin. They have they might have one, one or two others, but they're very interesting company so yeah, deadlock is really good.
Speaker 3:I don't know if there's a market for it. Um, I just hate how good valve is at making games. It feels so effortless for them to do this. There's the recent leak from a lawsuit on how much they pay their people and what their headcount is. Their entire games division is under 200 people, so they're supporting Dota CSGO somewhat of TF2. They pumped out that Half-Life VR game and they have this in development for 180 people in AAA.
Speaker 2:That's unheard of their games released per year is super duper low though.
Speaker 3:But they got live service right. They're live servicing Dota and CSGO. That alone should be way more than 200 people.
Speaker 2:Talk to anyone in those communities that complain when are the updates? We don't get shit. They outsource the cosmetics to the community. That is true.
Speaker 3:They got a system down pat.
Speaker 1:So this is going to be free to play game, you think?
Speaker 3:Of course, for Conker Deadlock. Deadlock is easily going to be a character cosmetic based game. They know this playbook. They might go with Dota, where they give all the characters away for free and just strictly do cosmetics and rely on the marketplace. They basically have solved all their problems, their monetization problems. They'll just run the playbook, as you just mentioned. As Eric mentioned, they'll probably do some loot boxes Then, because there's an auction house, the prices of the loot boxes or the amount of loot boxes people will open will be bid up to whatever the selling price is. So they'll sell a shit ton of them. Then they'll get the Steam Workshop going, they'll outsource everything to the community and they'll take their rev share and they'll get happy and fat making content. They're like 10 steps ahead of everyone.
Speaker 3:The first piece we are covering is a article from Pocket Gamer called Inside Mobile Hitmaker Lilith Games. If you're not familiar with Lilith Games, they've sometimes been called the supercell of China. They do incredibly innovative titles. They are one of the developers behind AFK Arena, as well as Call of Dragons, which is a 4X game. They make billions upon billions of dollars per year. They just released AFK Journey, which is a very, I would argue, high quality game that is based on the AFK Arena IP. It's actually struggling right now. They are headquartered in China, but they have around 2300 staff, which is actually more than I thought, and they actually have a North America studio which they just opened, and they just announced that they have a life sim game in development. But all of their games are incredibly high quality and I truly admire them.
Speaker 3:And the article is pretty interesting because it hits on two points we want to talk about. The first is how they organize their teams internally and how they choose to incentivize and pay them. So they have this thing called the Aladdin Plan, which is supposed to promote team loyalty and initiative. So the interview is with Vincent Wu, who is the head of strategy and investment and one of the pieces of the aladdin plan. The second plank is that when they launch a new game, they establish a standalone subsidiary company and core members of the development team collectively hold a 20 stake and, once the project alive, profits are calculated annually and distributed according to their share.
Speaker 3:And he says that as long as employees stay in the firm, even if they're on another project, their shares will be maintained and they'll receive dividends. They also talk about the fact that they don't have non-competes at Loaf Games and non-competes are very common in China, which I thought was interesting. And the other piece of this article that was interesting was that he says they break with the typical mantra over the last decade that building mobile games requires developings to kill their darlings or to fail fast, so very much a supercell approach. He says they don't do that and he says that the lowest project strategy is to ensure that all game projects are planned for global release from the very beginning of the project. They have a destiny and that the key metric that they focus on is user attention and that we will not easily cancel a project just because this metric is not met in the short term. We typically give development team enough time to continually test and optimize.
Speaker 2:This don't kill your darlings thing is pretty different than what we've heard a lot about Supercell. They're famous for soft launching games and killing them repeatedly if their metrics aren't high enough, and whereas it sounds like, at least based on the interview, that Lilith is much more willing to invest in and iterate and improve the games until they have high enough retention for that global launch. And I think this brings to a classic problem, which is the explore-exploit trade-off or the stopping at a time problem. Basically, you've got this game, you're working on it. How long do you spend improving it before you decide, hey, let's kill this one and move on to another project, versus. Oh no, there's still potential here. If we work on it for another year, it'll hit the targets.
Speaker 2:And it's a really hard problem because you don't know, like, once you kill a project, you don't know what it could have been right. Or if you spend more time investing in a project, you don't know what you could have accomplished by diverting resources. And yeah, we see different companies have very different approaches to this explore-exploit trade-off. Supercell famously very quick to explore, very quick to throw away old stuff. I think Riot had the opposite problem, where they spent too much time investing in projects that were doomed to fail. Lilith, at least from their success, seems to have found a good balance.
Speaker 3:So it's interesting. It feels strange to give a game a destiny, though, that this game is going to launch and we're going to make this idea work, no matter what the capital costs are. So, at the end of the day, what are you doing? You're giving yourself constraints, and if you're willing to kill ideas, it feels like you're willing to give yourself less constraints, and constraints are at least on paper and economy model. If you give yourself constraints, that does lower your total output. Why would you want to give yourself that constraint? Maybe the idea isn't working? We were just talking about AFK Journey, which doesn't appear to be revved at a great rate. Why shouldn't that idea be killed?
Speaker 3:Supercell seems to be on one side of the aisle. Super Supercell actually gives things a couple number of goes, and when they redo something in a big way Clash Mini, which is one of their games which is in development for two years, had all these crazy big updates where they radically changed parts of the game they give you like one example they made it such that you couldn't move the pieces in between rounds. So when you place the piece in Clash Mini, it was frozen. Remember, this is an auto-chess game, and one of the core components of auto-chess games is acting and reacting to your opponent between rounds and repositioning your units, and they just completely removed that because they felt like they didn't have the KPIs that they expected in soft launch. They'll do these radical relaunches, but this is like saying that this game's going global no matter what that feels to me on a really unhealthy path I'm sure the interview is like exaggerating for the sake of making a point, but that's like.
Speaker 2:I'm sure they don't.
Speaker 1:I'm sure they have killed projects, right, I'm sure, like not everything is kept alive forever you think there would be kpis you could track that would let you know whether this was a late bloomer or something you should kill right off the bat. And I'm trying to figure out, like what would those be? How would you know whether this is something that's worth trying to exploit or whether it or something you should kill right off the bat? And I'm trying to figure out, like what would those be? How would you know whether this is something that's worth trying to exploit or whether it's something you should definitely kill? Because we can all think of games where there's a small little change and that's all the difference.
Speaker 3:At some point you can start to get validation of your idea. You can take this from a Bayesian approach. Isn't that the right? Isn't the game? Economist approach to development is kind of Bayesianism. We are constantly updating your priors, but you have control over that the rate at which new information gets incorporated into your model. You can put this thing in a soft launch, you can put it into testing, you can give it to streamers. There's all these different ways you can accelerate the learning process, and each of those have different abilities to move your priors, or you should have them that way. But isn't that the game? Economist approach to development is to be a good Bayesian update your priors and get new information along the way and then make relative investment decisions based on that. It's the gate model, right? This is normally how AAA production works, right? You go through gates and each gate is reviewed by executives and then they choose to fund you or not fund you additional amounts of money.
Speaker 2:Exactly. You got to gather information, know whether it's good enough to move on or not, but you know that that didn't write the executives holding those gates right it's. It's not obvious when at the point of decision whether it was the right one or not to pass or fail a game.
Speaker 3:So what do you think about an alternative model where you have gates and basically you get a pool of money at each gate and that's basically the amount of ammo you get to overcome that gate, which has more clear guidelines, although I would argue you want the guidelines to be less, less material and strict in the early parts of the funnel and you want to be more subjective in the early parts of the funnel, but as you get later on, there's look at, if you don't hit 40 d1 retention, like we're going to stop this project, like you have 10 million dollars to overcome 30 d1 retention, if you drain that money, the idea is dead. Don't you think that's more of a startup idea? They spend the 10 million.
Speaker 2:They make a game, it doesn't hit the D1 retention target. Everyone's like, oh, this game's interesting, this game's fun. And then the lead designers, hey, I insist that, look, we just need a few more months for this to cook and we'll get there. And it's always hard to kill. Something has to make that judgment of call of okay, the designer says they need another six months to cook. Do I think they can actually get there? And you've got this whole sunk cost component to it and the team morale component to it and the emotional component of these are coworkers that you're going to be killing their dreams. I don't think it's that simple, as here's a pool of money. You either made it or you didn't, and then you move on.
Speaker 3:It feels like with this and with the dividends, what people want is they want a mini market. These corporations are getting so large and the connection between your input and, ultimately, the company's outcomes gets so distant that, if you can break down these incentives and give people more ownership, or that you can mirror market incentives okay, I have limited budget, as a startup would be. You have seeds, you have series A, series B, series C, and at each of those different series you have to go out and convince people that you've done enough to raise money, and I feel like we're trying to mimic that model inside of game building. That's what the gate process is and I think that's what Lilith is at least doing with the share amounts, right, with giving people literally 20% of the game, 20% of the subsidiary of the stock.
Speaker 1:So the 20% is it like? How does that work? Is it like a share of profit or is it like, literally like shares? Is it like rs?
Speaker 3:users. So they say they spin up a subsidiary so you would own shares in the subsidiary but who's it sold to?
Speaker 1:I might not know. It says once a project goes live.
Speaker 2:Profits are calculated annually and distributed according to their share, so I guess it's basically a profit share.
Speaker 3:So I think it's important that this isn't just a profit share model and this differs from a profit share model.
Speaker 3:So let's say they have AFK Journey, which is a game development.
Speaker 3:So they're going to spin up a subsidiary called AFK Journey Inc and all of the employees are going to work at AFK Journey Inc, the people who work on the game are going to work there.
Speaker 3:And let's say they make 100 shares and so the team, the core development team, is going to get 20 of those 100 shares, and Lilith Games itself, the parent company, gets 80% of the shares and so you're able to take a dividend based on how the game performs or how any company performs, and those are taxed at special capital gains rates too, when you get dividends. But they also own a share, which is why they're saying when they move projects, they still own that share. So if someone who's on AFK Journey goes to AFK Arena or to one of their other games of development, like Dislight, then they will still get those dividend payments based on the shares and how the game is performing and the dividends you get is a profit. So the game has to make profit and then, when the game makes profit, dividends get paid out. So you get dividends based on the number of shares you own.
Speaker 2:So, phil, you seem really surprised by this, but I feel like this kind of incentive pay seems pretty common, right, like you've got equity pay, you've got revenue share. I know, like Capcom, for example, there's franchise revenue share where if you work on Street Fighter, you get a share of Street Fighter profits. If you work on like Monster Hunter, you get a share of Monster Hunter profits. Yeah, I don't know why, it seems pretty standard and common today.
Speaker 3:Like you said, big companies need to essentially create a market for their games and their talent. So, to me, the underlying thing all these mechanics are trying to solve is the principal agent problem, which goes back to the infamous paper of theory of a firm management behavior, agency costs and ownership structure. You have agents that are employed by principals. Agents sometimes shirk, so now we create all these elaborate incentive mechanics to minimize the amount of shirking that employees do and ultimately raise the amount of output they have per dollar that we spend on the agent. And the agent in this case are employees and the principal in these cases are employers. And so what I was surprised, and so those are all different approaches and they all are trying to minimize the shirking Delta, and what I thought was interesting about this is that owning a share is materially different than any of those schemes that you mentioned and I think get you closer to market, like outcomes that we're trying to mirror, like that incentive of having dividends is much stronger to me, like how is that different?
Speaker 3:than the capcom system. So on one hand, you own dividends, which means that you can make more long-term investments in the game. So we were talking about sales previously, that a lot of product managers sometimes run sales to goose to get to quarterly numbers, which is sometimes based on executives and sometimes based on their own bonuses because they're incentivized in that way, whereas if you own dividends, because you retain ownership if you move to other projects, you can make a long-term investment in the game and then you still get back pay based on it.
Speaker 2:I see that long term as opposed to oh, I was on the project for a year, I got the payout for a year and then I left.
Speaker 3:You get time horizon benefits and I think that leads to better franchise management, because you should treat it as a long-term asset that pays out money over time, rather than something you squeeze every quarter. I don't know if this shrinks the shirking delta the most. That's how I'd at least evaluate. That is when we think about these different. If you think about imaginary B test, we can imagine AFK journey With this kind of schema. We can imagine AFK journey with another type of schema. Let's call it the quarterly franchise payout bonus and we can see which one actually has the better aggregate output and shrinks the shirking delta.
Speaker 3:But I think this is an interesting model. I'd like to see more of this. It's like you literally get ownership. I would say there's definitely like a million questions I have about this. For instance, how do they deal with dilution? Can they issue new shares? Who ends up deciding who gets shares and who doesn't? Do the employees get to decide who gets shares and who doesn't? I would say the other thing here is it's a tax benefit. The other thing is it's just nuts and bolts, Capital gains. In the United States, long-term capital gains is 15%, which is incredible. That's a radically different schema than most employees are used to. Usually they're paying top marginal brackets in games. California looks like Sweden these days. You're like what? 55% top marginal bracket? Imagine 15% instead.
Speaker 2:Isn't this you mentioned? Ownership Equity pay is super common in tech companies.
Speaker 3:That's true, but usually that's just cash. That's just cash because you end up flipping it in the same quarter and investing it into your S&P 500. If you're a good economist, you don't hold on to it.
Speaker 2:Or a startup. Like I'm at second dinner, I get a certain amount of options and those options are effectively incentive pay.
Speaker 3:Yep, that's the same here, but that's still.
Speaker 2:That's still direct, that's the same here, but that's still, that's still direct. That's that startup equity that potentially has a value. Right, you don't have a market value right now. Yeah, I guess the thing I'm I am surprised. You seem surprised that they were giving such a large revenue share or profit share, and to me it seems pretty standard for tech companies to do some kind of incentive pay.
Speaker 3:I agree. It's just it's issuing shares and giving employees literal ownership is really surprising to me. That just feels like a very bold move to me and I agree you can get similar mechanics with other bonus based pay systems. But this just literal shares just carry so many differences with it legally, financially and I would just say, spiritually. Owning a share to me is very different than getting a bonus payout beyond the incentives we talked about.
Speaker 1:It's like paying. It's like the way CEOs are paid, except for every single employee is paid like a CEO.
Speaker 3:Yeah, you've got real stakes.
Speaker 1:Do you think that this is like a different, like a new, revolutionary way to pay employees? Everybody gets like a small royalty from every single product they put out. But I am really curious your point, phil, like how do they determine who gets what? I'm assuming the most senior people end up getting or not even the most senior, but the most like high ranking employees get the highest share.
Speaker 3:I mean, it's not only that it's okay when new people join. What happens? Yeah, can you sell your shares internally? Can I sell my shares? It's a stream of dividend payments. Can I sell my shares? It's a stream of dividend payments. Can I speculate internally? Can I buy some AFK journey shares? Because now you get an internal market right and that's also interesting dynamics right. That would cause a whole bunch of problems.
Speaker 3:When I posted this on LinkedIn, it's interesting, and I see this natural reaction on many platforms in real life. The first thing when you post an interesting model like this is everyone comes up with a million reasons why it won't work or why it doesn't work. For instance, one of the pushbacks here was like oh, teams won't share information because they want their game to succeed and they may not have shares in the other game. And the first question is never like oh, this exists empirically, does this happen? Do people not share? They imagine a situation and when this thing won't happen, I think, do people share at Lilith? I't know. I don't know if they share on the margin, more or less because of this system, but I at least know we have some empirical evidence on whether or not it's happening and it doesn't appear to be the case, there's no reason to, there's no reason to believe that this mechanism is any different than typical compensation in terms of your incentive to help an employee, a fellow employee.
Speaker 1:It's not like them. Succeeding makes you do worse.
Speaker 3:Trade shares If Eric's on Marvel Snap and I'm on a second dinner game in development. I don't know, eric, what do you think? Do you want some of my shares?
Speaker 2:I'll trade you for some, I was going to say crypto's. Doing this, I wouldn't say it's been terribly successful.
Speaker 1:It is a very crypto type of concept. That's like early liquidity.
Speaker 3:We don't pay dividends, though, except for Luvium 2 or whatever the fuck. That token is from Australia.
Speaker 1:No, they don't. That's different. That's completely different. That's the players getting paid. But so, like the Tomica model is like you produce a suite of products, each product pays out a dividend. So, for example, like the Galactic Marketplace, there's a 6% fee. Every single time there's a trade, we get a six percent fee. So we are effectively getting this royalty from that entire product. So you create that product and then you hand it off to the players, maintain it and you move on to this next thing. So it's actually very similar and you can imagine a scenario where that's distributed across the entire firm or at least across the workers.
Speaker 2:So definitely crypto-y, and I would say that model hasn't been a silver bullet to make development successful.
Speaker 1:No, it doesn't really help, especially since the tokens have value before the product has value, which is a whole entire different topic, and one of the fundamental problems with the crypto company incentive model is the thing that represents. The product shouldn't have value before the product itself has value.
Speaker 1:The product shouldn't have value before the product itself has value, Like in this type of a model, like it would make more sense for the dividends to pay out once the product is making money, like the profit sharing. I think it's cool. I don't think there's any. You were talking about a principal agent problem, Phil. Like why is why?
Speaker 2:does this. I think the idea is imagine you're working on one of these games. You could put in extra crunch time one weekend to make the game make $100,000 more. But you're like, eh, fuck it, what do I get from it? Let me just sleep in. But under this you might do that, because you'll get paid an extra grand or whatever.
Speaker 1:So you're saying it's a solution to the shirking problem?
Speaker 2:Yeah, so Marvel Snap. So the Deadpool and Wolverine movie just released. What Great commercial and maybe critical acclaim, I don't know. First movie I saw, first super movie I've seen in eight years or something. I thought it was mad. People seem to like it. So who am I to say? But Marvel Snap as a tie-in. Marvel Snap as a tie-in had a game mode called Deadpool's Diner, which it's all Deadpool themed and it's got a bunch of social casino mechanics.
Speaker 2:So the idea is you start off wagering a mode-specific currency called bubs. I think they're called bubs because that's like Wolverine's catchphrase At the first table you wage one bub to play. At the next table the minimum bet is two, at the next table it's four, et cetera. It goes up exponentially right All the way up to like millions of bubs as your minimum bet. It's like poker where they're like super low stakes and super high stakes, tables and tables, and the idea is you play your way up. If you win big, you can like easily double, triple, a thousand, x your bub balance in one play session. But these bubs it's not real money, it's like funny money If you the game encourages you to go big or go broke, and when you do go broke. If you lose all your money betting, it refills and within five hours or something you're. You've got a nice balance to work with again and just see how far you can take it again. And yeah, it's been pretty effective.
Speaker 2:Personally, I found it pretty compelling the whole pushing your luck and playing more aggressive and oh, even if I bust out, it's fine because I'll just come back in a few hours. It created a nice engagement loop and, yeah, this was my kind of first exposure to social casino mechanics as a player and I found them compelling. I wrote a whole thing about, oh yeah, this is great, amazing, quote-unquote skill games adopting social mechanics mechanics. But I got a bunch of pushback with people saying, oh no, people have been doing this for ages, so I don't know, am I wrong? Like phil, you probably know more here. Is this common in other games? This?
Speaker 3:kind of what the fuck are those people talking about?
Speaker 2:no, it's not this isn't common at all high stakes.
Speaker 3:high stakes betting is not common in games outside of social casino Very rarely, by definition almost, but I've rarely seen this. Maybe a loot box open is not something you can really wager.
Speaker 2:I've rarely seen wagering mechanics. Okay. So Calum was talking a bunch of shit and I was like, oh, maybe I'm off base here. He said Archero and Pokemon Go did this.
Speaker 3:High stakes wagering. Yeah, what the fuck.
Speaker 2:I've never Okay all right, we can cut that part, but yeah, I think it's really cool. And to Phil's point, you want to get on your soapbox about this. You're always talking about how, hey, there's actually all these innovations that people aren't copying over.
Speaker 3:No, I do think Social Casino has a lot of very interesting innovations. All the constraints that comes with social casino and it's it's not just expected value. There's a little bit more that goes into. Okay, what should be the ev of this? There's a couple more things that you have, a couple more levers you have to pull, but they build so much interesting live ops around that simple mechanic that I think many people ignore how they could fit these similar types of mechanics in other games, because social casino gross. But they're compelling and it doesn't always have to be connected monetization. Sometimes you're just. We want people to have a good time and these are fun. They're fun.
Speaker 2:Mechanics like stakes are fun even if that experience, where you play and you win and then the stakes get higher and the tension mounts and mounts up to a climax where you like, oh, I lost it all, oh shit, it's a very compelling experience, I think when you do have a game like mode like this.
Speaker 3:It's good that it's a limited time Because, to your point, eric, the fact that players are refreshing bubs every five minutes and the fact that the winning player gets the losing players bubs, means that the supply of bubs is constantly increasing as people start to play, and so how many bubs you need to be, like, let's say, on a leaderboard is inflationary, and so there's a lot of tricky things when, let's say, you're setting up a purchase package and you have to fix the amount of bubs you get. Okay, that may not be the value like the dollar per bub ratio. It's constantly going down, despite the fact that your prices are fixed. So there's like a ton of baggage that comes with these type of mechanics too, especially in a PVP environment.
Speaker 1:Right, Well, the great part is it's like a disposable mechanic. It'll be done soon.
Speaker 2:It's just an event, right, that's brilliant. Yeah, phil's point, we actually had to do that. So, based on the highest table you've reached, the amount you refill and the amount of bubs you get per dollar spent goes up. Because you don't want to reset all the way to ground zero, right, like you want to just go a few tables back and work your way up. Yeah, definitely super unsustainable, hyperinflationary and I think temporary game mode is the way to go.
Speaker 3:We can't do price discrimination either. So you can't, because if you sell one person, lower buffs and I guess you could do some trade. But if matchmaking is random, you really can't do transfers.
Speaker 2:then so, interestingly, Snap had a previous mode.
Speaker 2:It's called Conquest where, if you squint really hard, they look the same right? It's a mode where you play. Every time you win, the stakes go up, exponentially increasing, and the higher up you go that goes from five points to 200 points to a thousand points, with rewards, and if you bust out, you can start again at the bottom table for free. So, if you squint, conquest looks the same as Deadpool's Diner, but had nowhere near the same level of engagement, nowhere near the same level of enthusiasm. And I think, for me at least, one of the big issues with Conquest is that it's very.
Speaker 2:Conquest was framed as the super competitive mode. It was like you play against the same person repeatedly so you get to learn their decks and learn to counter their strategies. Everything around it is like this is serious for try hard people, whereas deadpool diner it's. It's very goofy, it's fun, it's light-hearted, it like really encourage you. Yeah, have a go at it. Sure, it doesn't matter. And yeah, it's not just the sketch of the system, but also like how it's implemented and executed do you think snap will eventually have permanent game modes?
Speaker 1:because right now I think it's just the main ladder that you're just climbing.
Speaker 2:The hope is that Deadpool's Diner is reusable and we'll recycle it in the future and maybe have maybe four game modes that we cycle in and out like a LiveOps playbook. A LiveOps playbook that Snap was sorely missing.
Speaker 1:And it will be. It won't be like a different branding of Deadpool.
Speaker 2:It won't be a different skin. It'll be Deadpool's diner. Yeah, I don't know. Maybe we'll skin it, maybe it'll be like a fucking Wolverine diner, I don't know.
Speaker 1:Because I'm curious how many different ways are there to use the betting mechanic? Because the betting mechanic, in my opinion, is one of the probably the most, the single, most interesting mechanics in all of Snap in terms of like stuff that I've never seen before. It's just a mechanic you do not see in video games. It's really common in board games, not as common in video games being able to like, bet and put your money down I think we should think of this as wagering.
Speaker 3:I think wagering is a better way to look at it. I can set wagers and I think the other component is you have to have autonomy over the wager. Like conquest, you don't have autonomy over the wager, if I'm correct yeah, that's right, it's you're at a level and that's it I think that's that, to me, is actually a very big line.
Speaker 3:I would draw between conquest and other social casino mechanics is your. It's the autonomy over the wager. Because what's interesting is the conquest mechanic, by the way, has been pitched me on every single fucking battle royale I've been on. Everyone has pitched this idea of okay, if you come in first in a battle royale tournament, you get a ticket to go to another mode, which is another battle royale game which everyone else has also earned that ticket. If you win that one, you get another ticket. And everyone has looked for comps for this and I actually have started to point them to this conquest mode, but none of them have taken off, which I find interesting because it feels it's like a battle royale on top of a battle royale.
Speaker 3:It's like a babushka doll no to your point.
Speaker 2:Every version of this I've seen has not been successful. Riot tried one on league. They're like a pseudo mode also not very successful.
Speaker 1:It just sounds like it should work like like the draft pool in magic arena, like everybody who's, I think I'm almost certain. If you win your first game, you verse other people who have won their first game. If you win your second game, you verse other people who won their second game. What's the difference between that and what you guys are explaining? Because that's very successful. I guess it is the same.
Speaker 3:I would argue it's really just ux to your point, chris, it's really just ux of. You're right, it is a battle royale essentially when you're doing the draft stuff, but I would argue it's I would argue that the conquest and the battle royale ideas they feel like they have a really big exponential slope in terms of rewards.
Speaker 3:And the Battle Royale ideas they feel like they have a really big exponential slope in terms of rewards. And the Magic the Gathering Arena. Rewards for the N plus 1 win do increase, but I don't think it's that sharp of a curve.
Speaker 1:Yeah, the sharpness of the curve actually happens around the third win. Usually that's the point when you make more than what you spent on joining.
Speaker 2:I think it is exponential. I made a chart of this one time. It is exponential. I made a chart of this one time it is like exponential decay. For how many people Do you?
Speaker 1:mean in terms of the gems rewarded? Yeah, because it's definitely linear in terms of the packs. Maybe it is exponential in terms of gems. That'd be cool.
Speaker 2:Do you think there's other mechanics from Social Casino that developers should be?
Speaker 3:looking at oh, absolutely no-transcript. I just I think that the bound should be bigger, like they were doing it for it was a five dollar skew and instead I wanted a much bigger rebound at a hundred dollars the screen share working a hundred dollars skew in a mobile game.
Speaker 1:You can do $10,000 now, oh, is that the cap?
Speaker 3:Yes.
Speaker 1:Okay, so it is pretty linear and then it looks like Hearthstone is exponential.
Speaker 2:This x-axis is percentile, so each step up is one incremental win. So if you put this on a linear scale, it would be, but yeah, it gets dramatically higher for every tier.
Speaker 1:Damn. How many games are there in a?
Speaker 3:hearthstone match so hearthstone has a very exponential payout at the end, so they're stake heavy. I guess you could put it like you really there isn't stakes, but like you have a lot to gain if you get to a very high level performance and the drafts an arena.
Speaker 2:It's a little bit flatter yeah, it's like a 40 multiplicative bump for every win. Roughly, you win two in a row, you double, you win two more, you double again. Anyway, this is cool. I remember this is in one of your pieces I forgot about this.
Speaker 3:I got charts on deck you were. Did you show any of this in your second dinner interview? No, just like someone asked you a question about drafting, you just like whip out this chart. You can see here the reflection point. So very interesting paper we were talking about from Call of Duty, from the Call of Duty team. One of our Arcanum's friends worked on it In early 2024, they ran the deprioritized skill test in Call of Duty Modern Warfare 3, where they used their A-B test framework to loosen the constraints on matchmaking.
Speaker 3:It's important to note that it was not removed entirely from the matching algorithm, but the level of skill that was used in the matching algorithm went down and they tested this on quite a few number of users in North America. I believe they said it was around 50% of the users in North America. That's a crazy sample size. They don't specify the specific end, but I think we can assume that it was quite large. The thing that I want to start with, before we get to the results of the paper, was that they actually just spent a lot of time talking about how they do matchmaking and it's as you'd expect and I still think it's really interesting to walk through.
Speaker 3:So one of the charts they have in here is called their heuristic selection process, and so what they do is they take a given player and they try to rank them on a number of dimensions.
Speaker 3:So geographical distance is one of them, because you remember, ping matters here.
Speaker 3:There's skill, which has a weight associated, there's control scheme, so whether or not you have a mouse and keyboard or controller, and so they have all these different parameters that they weight in different ways, and it'll be interesting to see like those weights are. Still, I wonder how those weights are chosen, whether or not they're designer driven or whether or not there's a deeper algorithm here. And what they do is they try to minimize the cert, the distance between an individual player and across all of the aggregate sum of all these metrics and other players, and that's the core basis of the formula. So they have geolocation it's stored as longitude and latitude, skill is between zero and one, which is the average skill percentile, and then the control screen. They can add and drop all these different dimensions. It felt like a really clever and well thought out system. This is exactly how you want to think about matchmaking systems and also integrates into all their sub cases like they have a lobby and they need to fill one player versus matchmaking altogether.
Speaker 3:I just I love seeing this just spelled out explicitly.
Speaker 1:And, interestingly, the skill metric is one of the lowest weighted metrics. The highest weighted metrics that they talk about is ping. That's like the most important metric, which I thought was really interesting. So, as people are like, oh, skill based matchmaking is a really small part of the actual matching.
Speaker 3:We don't disclose the weights though did they?
Speaker 1:They didn't disclose the exact weights, but they do say at the beginning ping is most heavily weighted, Skill is lower weighted. So we don't know. Is it like 10 percent, 20 percent?
Speaker 2:And I. That was a takeaway I had as well, because just the feeling of playing, if the game is choppy and the game feels broken, that matters way more than my opponents. Better than me, you see the same dynamic appear in fighting games, where people strongly prefer a good connection over a close match.
Speaker 3:I think there's actually an interesting insight which is I wonder if this has external validity to streaming websites getting the right movie for you versus the quality of the film. This seems to translate more like here's the thing ping ping matters a lot in pvp games simultaneous real-time pvp, twitch-based games. Yeah, that makes a lot of sense to me and we. What we don't know is we don't know the relationship between how ping changes and then how your subsequent outcome changes, because I think that's could that could be the case, chris, is if you have higher ping going into a match, then you're going to have a much worse kd ratio, which would affect your subsequent retention. There could be be a skill ping relationship.
Speaker 1:What was super interesting is like the effect size of reducing just slightly. Reducing the weight of so like the AB test is literally like the weight on skill in the matchmaking algorithm is decreased. That's it. It's not. There's like nothing fancy, and I thought the effect sizes were actually pretty substantial, especially for a game like Call of Duty. It was like 1.25% or something like that.
Speaker 3:Let's get there. Let's just talk about. I think it was interesting how because they do report what the search time is to find a match by skill bucket, and so it was around 15 to maybe 20 seconds. But what I thought was remarkable about how they've done their matchmaking is that the ping or, excuse me, the search time is relatively constant across all the skill buckets. I would assume it'd be sometimes hard to fill matches as skill increases is there because the distribution of skill is it's not constant across each of these, it's not uniform, but there's, it's like any other normal distribution, and despite this they're still able to maintain constant search times actually one of the interesting things they did was they do a percentile normalization on and they said, specifically for that problem you just described, which is to keep the matchmaking time and any like skill parameters, consistent cross buckets brilliant right
Speaker 2:yeah, and the other weird thing was about their skill thing is they didn't use like a classic elo style skill.
Speaker 3:They like it's a bunch of of heuristics about kd and then run through the percentile and the other thing is we see the same thing in ping, like they're able to maintain pretty constant ping. In fact there's actually a slight decrease in ping as the skill bucket increases. I actually wonder if they set weights at a skill percentile level and so they actually start to favor lower ping a little bit more in favor of a little bit more search time as skill percentile increases, which would make sense to me.
Speaker 2:My guess is the inverse, which is that higher skill players just have good connections and you don't play until you get high skill if you have a bad connection.
Speaker 3:That makes a lot of sense, but would you still, as a designer, want to accept longer wait times for better skilled matches and lower bang.
Speaker 2:Cod says no, that's fair enough, fair enough. But these cods a bit more casual than I don't know, like a, maybe like a counter-strike tournament game or something well, and they were.
Speaker 1:I don't know if you're going to talk about this, phil, but this idea of this cyclical process where the more attrition there is, the higher the skill level becomes. So you have this vicious cycle where, if you match too little on skill, what you end up with is these blowouts, and if you are on the receiving end of a blowout, you're going to from the game and then you end up with just a higher skill. So it's a really interesting like this idea of if you don't match on skill, you actually end up with just super high skilled player.
Speaker 3:Let's talk about the tipping problem because someone, an economist want to know about prize on that too. The only other thing that I thought was interesting is that they established that their team skill metric actually does correlate to win rates. It it felt like this was also a good establishment graph. So when you look at the difference in team skill and you look at what is the average win rate, if there's a zero difference in kill, zero difference in team skill, the win rate is about 50%. So at least we can have some trust in the system. Which I thought was interesting, though, is that the win rate seems to cap out at around 95% when the team skill difference is both 0.2 and 0.3 different from one another, which I thought was weird. You don't expect to see more of a linear relationship, so maybe they're having tough times assessing skills at very top end distributions, or maybe there's just so much stomping going on that there's almost so much stomping you can do. Like there are also backstop mechanics in the game, like they start to add if you die so many times, you get a death streak mechanic, which I don't know if that's still in the game, but they have some safety mechanics so you can't get completely wrapped.
Speaker 3:Okay, let's look at the actual results. This is a big chart right here. This is the paper Figure three. This is the difference in players returning within 14 days of the deprioritize skill test, and so this is represented for each of the skill buckets, and so what we can see is from the zeroth percentile to it looks like the 90th percentile, there is negative treatment effects, so less players returned within 14 days for 90% of the population, and yet there was actually a small increase in players returning if they were at the top end of the skill distribution. So when skill based matchmaking was turned off, that means that people who suck are going to get killed more and people who are good are going to kill more. And yeah, I don't know what were your thoughts on this, guys we're talking about. The most negative group only had a 0.75% difference from the treatment group. That is a really small effect. Size 1.7. Yeah, thank you, that's really fucking small.
Speaker 1:It's not that small for a company of this scale. Right, and this is extremely accurate.
Speaker 2:Wait is the denominator here, 1.75% in day turn, is huge.
Speaker 1:Is it a percentage point?
Speaker 2:My guess is they're like average is probably, let's say, 60,. You know you're losing 1.4% of your single week. This is the thing that's.
Speaker 3:Here's the thing that sucks, eric, is we don't know the actual N, so we don't know if. So if you go from one to two, you know it's 100% increase. We go from 65 to 66, that's a much smaller percent increase, and this is percent, not percentage point, which is driving me crazy sure, what's the difference?
Speaker 1:between percent. Going from 65 to 66 is a one percentage point change, but it's a I don't know what it is like a 2.5 percent change. Okay, and is this the former or the latter?
Speaker 3:This is not percentage point, this is just percent.
Speaker 1:So we don't know what the D14 number was.
Speaker 3:I don't know if it was a big number or a small number. I think it was a small number and if it was a small number, that would mean that the effect size is pretty big, similar to what you were mentioning, Eric.
Speaker 2:I just it just shrugged. Call it negative 0.5%, compounding every two weeks over the course of a year. So taken to the 26th power, that's something.
Speaker 1:We also don't know how much they slackened up the skill right, they didn't remove it, they just all they did was de-weight it a little bit. So that's another thing. It's not like. This is the difference between no skill matchmaking and all skill matchmaking. And it's funny because I think almost every gamer you ask is skill-based matchmaking good? They're like no, it's bad, it makes me verse a bunch of sweats and then I have to and I don't like playing the game, which I think is so funny, because this is obviously.
Speaker 3:That's obviously not true well, at least for the call. I think we have an even better explanation, chris, which is that if you're a high skill player, you gain you stand to lose the most with skill-based matchmaking, because you can't stomp you can't stomp a bunch of people, and so for the people who tend to go on the internet and complain about this, I'd argue it's people in the, it's the interest groups. What's the lobbyist from the interest groups? This is the gun lobby of gamers. Is spmm is so.
Speaker 2:So for context for the listener, this new to me. But skill-based matchmaking was controversial in some of these shooter games, especially these console shooter games like COD and Halo, which historically did not have skill-based matchmaking. And, like Chris and Phil are saying, a lot of influencers, top players, were saying, oh, skill-based matchmaking is bad because it messes with the experience, blah, blah, blah. It's funny because I had the opposite opinion coming in, because if you look at people from Starcraft, people from fighting game, people from League, they all love skill based matchmaking. They're like, yeah, I want an even match. This game sucked if it's not even. It was funny hearing the opposite community perspective here. But yeah, to your point, that that little bucket, the little green bar at the far end, that that's the vocal minority.
Speaker 3:I think the other thing that was interesting here is exactly the point you mentioned, eric, and you were mentioning earlier, chris is the tipping problem, which is that the churn rates, with skill-based matchmaking turned off, are not uniform across each of the skill buckets. The players who tend to be the worst and suffer the most in terms of KD ratio churn the most, and so, ultimately, if they keep moving the population week over week, you have the tipping problem where the skill distribution for the median player keeps increasing and it adds this problem that this problem gets worse and worse because the players at the bottom of the skill distribution will keep leaving. Although we don't know, we don't know if that was the effect here. We don't have a time series. I wish they showed us a time series of this happening, but I guess that's at least what they speculate. I agree with that.
Speaker 1:I love how simple it is. I love how straightforward the approach is. You've got the ELO system, which has this fancy equation that says here's how you calculate how much the change is. I just love that. They're like here's the metric we care about. Here's this algorithm that is composed of a bunch of different weights, straightforward weights this is 10%, this is 20%. It made me really%. The whole thing made me really happy.
Speaker 3:I just love that they're putting this paper out there, and the interesting part about the tipping problem is that it's reflected in figure four here, which is the difference in quit rate from the deprioritized skill test, from the experiment that they ran. And so it is that the quit rate actually increased for lower skilled players and actually dramatically decreased for higher skilled players, which is exactly the opposite of what you'd expect. But they get this result because they have the tipping problem, which is that you had all these low skilled players who kept left leaving because the distribution kept shrinking and so the quit rate was much higher for them I wonder how, I wonder if the standard errors got impacted throughout the experiment.
Speaker 1:I'm assuming they don't really care about standard errors but, like you could imagine, one of those like diff and diff crazy econometric guys in on crypto or on econ twitter, that's oh, you can't really estimate this parameter because the sample was like trimmed throughout the period or something like that. I don't care about that kind of stuff kyle's got buckets of players.
Speaker 3:I'm sure they're fine the other part that was interesting here was figure six. This is the difference in kills per minute from the deep right skill test. So the number of kills per minute decreased when you turned off skill page matchmaking for low skill players, which is exactly what you expect, and it got a lot higher for players who were highly skilled. If you were in the lowest skill bucket, you were in the top. You were in the lowest skill bucket, you were in the top. You were in the bottom 10% of skill. You saw about 10% less kills per minute. That's a pretty dramatic change in gameplay.
Speaker 1:I'm honestly surprised. It's interesting to see this because this means that players were having similar, especially in the middle deciles. They were having a similar experience. They were still getting the same metric, kpm, but they were leaving nonetheless, which I think is really interesting. I'm assuming it has to do with the results, like you could still get a, you could have a maybe two kills per minute and you lose, versus two kills per minute and you win.
Speaker 3:But there are some mode data in there. There could be some differences in modes because different modes might have different amounts of kills, and so they look at score too. There's. There can be a couple different ways to look at it. But yeah, it's interesting. Chris, I'm not quite sure about that one either.
Speaker 2:All right, so I got to jump fellas. Yeah, all right, let's call it episode 31 in the can Boom. Yay, we should teach this to our children.
Speaker 1:Economics is major. Everyone has to major in economics. Number one for
