E37: Is Gaming Better Than Everyone at Experimentation? (w/Dr.Julian Runge)

The best tech firm experimentation seems to offer thousands of button color experiments. Dr.Runge has a better approach, which changes at every game development stage. We debate gaming's broken relationship with science, the proper experimentation framework, and how much you'd bet on yourself to complete Cousera assignments.

Read Dr.Runge's new paper NOW!

Showlinks:
Julian Runge
Gaming Companies Run Thousands of Experiments a Year
Game Data Pros
How to use games to build relationships with your customers

Transcript

Eguan: 0:00

Aren't they already monetizing on people's failures?

Phillip Black: 0:02

Let me, yeah think that's true. What would you pay, Julian, if this were the case to insure this? How much would you put up as a deposit?

Julian: 0:09

Me.

Phillip Black: 0:10

Yeah, Much do you, would you really pay for this

Julian: 0:13

for what? To ensure I don't fail on coursera.

Phillip Black: 0:16

pretty much

Julian: 0:17

I,

Phillip Black: 0:17

it's a commitment device to learn game design theory. How much are you willing to pay for that? How much are you willing to like.

Julian: 0:22

Think you have to pay me I guess not to fail wait. Oh, which way around?

Phillip Black: 0:45

37. Returning guest. Returning

Julian: 0:48

Woo.

Phillip Black: 0:49

I think are, is this our only guest? I think we've had right?

Chris: 0:53

No. this is the first one. Wow. you. must be a special man.

Phillip Black: 0:58

You're back in business. You've got a new paper we're gonna talk about today. Hot takes everywhere.

Julian: 1:03

Okay. Yeah. Good, good.

Phillip Black: 1:05

What have been up to we saw you?

Julian: 1:07

Not much. Same old doing research, teaching, little bit of advisory work.

Eguan: 1:12

You up to GDC next month?

Julian: 1:13

Yes I am. We're gonna organize a mini summit on revenue optimization in games March 19.

Phillip Black: 1:19

You had some celebrities. You had yo

Julian: 1:21

yeah. He is gonna be here again. He's gonna again.

Phillip Black: 1:24

he was there. Brett Novak, I work with, whose really smart about product, was there talking about some theories that have like really held up over the last year. ask if there's and a section, you can heckle, you gave a talk. Are you

Julian: 1:37

Yeah. Yeah. I'm going to actually give a joint talk with Yost.

Chris: 1:39

Oh,

Phillip Black: 1:40

that, Little collaboration work. Little co-publishing going on back

Julian: 1:44

Yeah, so I don't know if you saw we published Business Review article together November on games as media to relationships customers.

Chris: 1:53

I vaguely remember seeing that.

Phillip Black: 1:54

Yes. We'll post a, how about we post a link to it notes?

Julian: 1:57

good. Yeah, let's do it. So started writing on that piece after GDC last year actually, and then November it finally came out. And that, yeah, we would just, we're gonna give a talk together this year, actually, drawing on that that article quite a bit.

Phillip Black: 2:10

Is academia buying any of this shit? Since you published all your papers, are people buying into the price experiment results? Do they care about it in the context of games? Do they think it has no external validity? Has the agenda needle moved at all since we last chatted?

Julian: 2:25

That's a good question. What's the agenda needle exactly?

Phillip Black: 2:29

I don't

Julian: 2:29

My, my own agenda or

Phillip Black: 2:31

there's Mario at one end and there's like macroeconomic inflation

Julian: 2:35

okay.

Phillip Black: 2:36

at the other,

Julian: 2:37

Okay. Yeah,

Phillip Black: 2:38

closer to Mario these days? What's going on?

Julian: 2:40

probably more inflationary, macroeconomics takes no do, but I'd say I'm pretty excited actually. I think marketing, which is my academic field slowly waking up to games and there's more and more work being published. Both empirical and theoretical in management science and also in the top marketing journals. Pretty exciting stuff. Actually. I'm also working with one of Chris Ryan he's at UBC Solder. School of business and he has done some phenomenal theoretical work that came out in management science, on level design and the likes so I'm actually really excited for it. Shall see if it's gonna move the needle. When it comes to the paper we discussed so much last time, I think the external validity point is always a difficult one, but you have to convince reviewers of and editors. But this mechanism we pinpoint, this com complementarity between the free version and the premium version, and that, that leads to this virtual cycle which makes price promotions quite profitable in the long run counter to existing academic wisdom. I think that's convincing people, and I've actually yeah, also talked to academics about that repeatedly.

Chris: 3:43

Is the narrative for marketers like Is it like for me, the narrative when I'm trying to sell games to economists has always been like experimentation, oh, you can set a human behavior in these online environments. Is it similar in marketing or is there a different angle

Julian: 3:58

Good questions. I think it's in some ways. Definitely consumer behavior and understanding that interesting. the the marketing mix and actually understanding how strategically what firms can learn on a strategy level, like in of to run games and what are, like, even applying the marketing mix to games is something that hasn't really been done actually in the piece we wanna discuss today. I'm trying to start doing that a little bit. For example, I just we just had a paper accepted at the International Journal of Research and Marketing that actually describes a dynamic difficulty adaptation system in a puzzle game. There we also lean a little bit into the marketing mix. So it goes beyond like looking at consumer behavior and trying to understand what's actually possible on the product dimension of the marketing mix. And actually, sorry, I should start by quickly saying what the marketing mix is. That's a really long standing in marketing scholarship of in the 1950s by Neil Borden.

Phillip Black: 4:55

we'll have a your you had a podcast with Eric Seifer recently as well. That was a good one.

Julian: 5:00

Yeah, indeed. Yeah, always enjoyed going on the podcast. And there was one where I went with Kun. Powells also, right? The professor from Northeastern. Yeah.

Phillip Black: 5:06

memo.

Julian: 5:07

Yeah.

Phillip Black: 5:08

Payment

Julian: 5:08

anyway, so the marketing mix longstanding concept introduced by Neil Borden, who was a professor at Harvard Business School. And it has in its canonical version four Ps price, promotion, product and place. Even just spelling means in the context of games and then understanding how does a dynamic difficulty, the adaptation system, which is clearly on the product dimension of the marketing mix, but how does it impact your ability to price and to run promotions, et cetera. I think there's so much work to be done that goes also beyond Consum behavior.

Chris: 5:37

You use the four P's in the paper we're going to talk about today.

Julian: 5:39

Yeah. I'm trying some marketing to guess

Chris: 5:43

now.

Eguan: 5:44

earlier, did you say?

Julian: 5:45

it.

Eguan: 5:47

management science and level design?

Julian: 5:49

Yeah. that's. interesting.

Eguan: 5:51

Did Yeah. me What? What's that?

Julian: 5:53

that in the context of Chris Ryan, who's a UBC Solder And He's done some really interesting theoretical think two or three of his papers were recently published in, over the last two years, I believe in management

Eguan: 6:06

I.

Julian: 6:07

One is on level design and really building the theory for why many games have this mini boss, big boss design, right? Where you have a difficulty peak in the middle of the levels, roughly, and then another one in the very end. And really trying to build up a theory behind that. Why is that? How games are designed? can we learn from that for world design more really interesting that space.

Eguan: 6:29

Is the implication that we should design, like how we in a similar way, like help me connect the dots here. Yeah. Between the game and the.

Julian: 6:38

No, not at all actually. It's not that. This is really interesting'cause this is, I think the differences between academia and practice in some ways that, may lead to that misunderstanding. Chris is an operations research scholar, right? And so do, in, in, in all kinds of business settings. And so he's really developing theories games work from a business point of not really even about learning anything about how to compensate workers or anything of the sort. It's really more a theory piece squarely focused on games.

Eguan: 7:10

I've often thought about that. like why do we create these always three acts of a play or whatever, the theoretical basis why? When you should have difficulty spikes and whatnot. have you been what have you been playing? Are you playing any games lately? What? you talk about your work. What do you do in your free time?

Julian: 7:36

Actually I'm, I've been moving around a lot. So skews pretty heavily mobile at the moment. Both move. I just moved to Chicago to start a new job at Northwestern Madill Yeah. SKU's heavily mobile. I, though just downloaded Steam a Windows but haven't done with that yet. one thing I did is I started playing Legendary Game of Heroes again recently. That's a game I worked on a lot, actually, a few years ago.

Phillip Black: 8:02

At network.

Julian: 8:03

yeah, that's right, exactly.

Phillip Black: 8:05

Did you work with Ethan

Julian: 8:06

Yes I did. Yep.

Eguan: 8:09

What's this game?

Julian: 8:10

Legendary Game of Heroes.

Phillip Black: 8:12

Puzzle. RPG, medic Squat RPG with a match mechanic

Eguan: 8:16

Oh, cool. Reminds me of puzzle and Dragons.

Chris: 8:19

the legend of the legendary heroes.

Julian: 8:21

Yep. It's I think it's a phenomenal game. And anyways, I just downloaded and played it again from the beginning just to see how it evolved. I think that's pretty cool when you've worked on a game like, whatever, three to five years ago, and then seeing what it has developed into I'd say a lot is actually quite similar, but there's new nuances and overall, more mature experience.

Eguan: 8:40

How do you feel looking back at your baby? Do you look at it and be like, oh, like I can't believe they changed this thing I loved, or is it more oh, look, it's grown up and become its own thing,

Julian: 8:48

so I wouldn't say it is not my baby, right? It's the game designer's baby.

Eguan: 8:52

Your nephew, I, don't know.

Julian: 8:53

yeah, I was doing data science so were more the offer targeting and those tho sorts. It's cool to see. And yeah, it just feels, it's still super fun. Some other games I'm playing is Agent Hunt. Nobody will know that. It's just a pretty shooter where you sniper and yeah, it's fun sometimes when you have a few minutes and blow off some steam.

Eguan: 9:14

Agent Hunt.

Julian: 9:15

Agent Hunt, yes,

Eguan: 9:17

Oh, cool.

Julian: 9:17

a great game,

Eguan: 9:18

Hey, yo,

Julian: 9:18

Check it out. Check it out.

Eguan: 9:20

If it's fun, it's a good game.

Julian: 9:21

Yeah. Fair. Fair. And then Fruit Ninja. Actually I'm playing Fruit first mobile game I've ever played back in the day, or in depth I

Chris: 9:29

That's Oh, gee,

Julian: 9:31

And it's so fun. I was like actually thinking that when if there would be an opportunity maybe to pick the game mechanic back up maybe in a different, combined with a different meta game. I've also started a class actually on game design on Coursera recently. I realized I've been working in games for so many years and I've never actually taken a class in game design. So I felt like that was a little overdue.

Phillip Black: 9:51

gotta back to us. I want to know what they're teaching us. Like the theory of game design. Like I've always imagined it's just a hodgepodge of like social science like then some conjecture is like my understanding. But I love to be proven wrong. don't think I've

Chris: 10:06

there's a,

Phillip Black: 10:06

Ralph Costner has PO published a little bit. I only, he's literally published a theory of fun, but I feel like a lot of it hasn't been formalized. There's I

Chris: 10:16

Well, there's definitely no models, no mathematical models, but there are a bunch of books that are pretty well agreed upon. It's Like the standards. If.

Eguan: 10:27

I I read a theory of fun. I wasn't super convinced by it. His main thesis there is the fun is in the learning, which I think is true. I think many games are fun from, you encounter a new mechanic, you figure out how it works, you learn how to use it, and you feel like good for learning, have had to use it and overcome some challenge. There's also a lot of games that don't fit that, mold at all. fails to account for them.

Julian: 10:50

Yes. I would say in this class it's actually very applied. I'm not very far into it, but they more start from the design point of view, like almost like maybe design thinking or whatever. But you start by actually very quickly making your own game, and I think that's a nice way of teaching it deal with theory and like advanced stuff all the time anyway this is actually nice for a change to do something hands on a little bit.

Eguan: 11:13

cool. So do they have you like making game prototypes or is there like a course project or something like that?

Julian: 11:17

be honest, I'm currently overdue on my first homework, which would be to make a very simple game, just like drawing it out on a sheet of paper, and I haven't done that yet. Yeah. I'll need to circle back.

Phillip Black: 11:29

Can I pitch you on an experiment for Coursera? Because this is always what happens in Coursera. Like I'm sure if you look at the Coursera retention curves, they're fucking awful. And I also think there's an interesting point that like these MOOCs have just fallen off, but why can't they do the thing where you put down a hundred bucks and refund you after you get like certain grades as a commitment device. Like I feel like people would do that and they could monetize on people's failures.

Eguan: 11:53

Aren't they already monetizing on people's failures?

Phillip Black: 11:55

Let me, yeah think that's true. What would you pay, Julian, if this were the case to insure this? How much would you put up as a deposit?

Julian: 12:02

Me.

Phillip Black: 12:03

Yeah, Much do you, would you really pay for this

Julian: 12:06

for what? To ensure I don't fail on coursera.

Phillip Black: 12:10

pretty much

Julian: 12:11

I,

Phillip Black: 12:11

it's a commitment device to learn game design theory. How much are you willing to pay for that? How much are you willing to like.

Julian: 12:16

Think you have to pay me I guess not to fail wait. Oh, which way around? I'm

Eguan: 12:20

So you give them a hundred dollars and you say, if I complete the course, I get a hundred dollars back, otherwise I lose it.

Julian: 12:28

Yes, that'd be nice. Oh yeah, that's like a piggy bank, but

Phillip Black: 12:31

capital capital means a stronger commitment'cause you've got more at risk.

Julian: 12:35

that's an interesting one

Phillip Black: 12:36

But it also means that you're likelier to have a higher completion rate because you don't wanna lose all that capital that's at risk and you have real and strong autonomy over that too.

Julian: 12:45

actually. Yeah, it's true. There's tons of opportunities for gamification of that sort. I, don't know if anybody's doing that, but.

Phillip Black: 12:51

I think there's a reverse model here, which is you would calculate your hourly wage. I. For per additional unit of studying based on the amount of credit that you've put up. And then you would compare that against your relative opportunity costs and then that's, you'd pick the higher one. I think that's the strategy. So like you have to set it at a high enough so that it crowds out your other opportunity cost and you're You got a lot of shit going on,

Chris: 13:14

You've completely lost Phil. What are you trying to do? Are you what are you trying to do?

Phillip Black: 13:19

economic, what is economic model for like you deciding to put up a deposit for Coursera

Eguan: 13:26

Golden handcuffs basically.

Phillip Black: 13:27

So what do you want to set the wage rate at or the amount of capital that you deposit? And I think you work backwards

Chris: 13:34

is there an. interest rate

Phillip Black: 13:37

We could, are you sure If you want me to put in like an interest rate over

Chris: 13:40

so you you put a deposit and if you don't finish then you lose your money? Is that? Is that your Oh, as a retention mechanic?

Phillip Black: 13:47

right, but it works out to an hourly wage. Once you've committed the capital, you now think of everything as an hourly wage, because you're gaining back money by studying, right? You're earning your way back up to your based on hour of studying you do, because it extends your attention, which means you're more likely to get a payout,

Chris: 14:05

Yeah, but you just, spent the money. You'd be better off to just, not do it in the first place. If there's any positive probability that, you don't complete it, you would not do it. If you're not, if you're rational,

Phillip Black: 14:16

That just tunes down the expected probability, right?

Chris: 14:19

but if the expect, if there's a,

Eguan: 14:20

I I think point is that you know that you're rational, right? You know you want to do this thing, in the moment you'll be lazy. And you're biases.

Phillip Black: 14:30

Hold on. There's a rational, there's a rational expectation for a rational model for this, obviously, which is like you've got the present self and like the past self. Have you heard of this model? So this is how you can maintain rationality your view, Eric, the you have the present self, which wants to commit to that, your future self won't at like time t plus one. And so to your future self version.

Eguan: 14:53

Wait but basically what you got split personality disorder. That's, how you solve for

Phillip Black: 14:58

That's one of the ways we can build like an agent-based around the

Eguan: 15:03

Hold on. just one, one last thing on this is that if you think about it from the Coursera designer's perspective, there's a huge perverse incentive here to make the course artificially hard and like rob customers of their money, right?

Chris: 15:14

Oh if there's this sense, but why it's not like the course instructor gets the excess money, right? That would be super fucked up.

Phillip Black: 15:23

I disagree. First of all I do think they should get the rage rate, but I would argue I would argue that like they probably have more value in retention. Like they want you to get your capital back because that ultimately long run retention on the platform. Feel like you can monetize

Chris: 15:37

Who gets the capital? If you don't get it back, where does it go?

Phillip Black: 15:41

Definitely

Chris: 15:43

Because if it goes to Coursera, then, there's going to be an incentive

Eguan: 15:46

Maybe, maybe it goes to charity and so then, you know, no to,

Chris: 15:49

be cool with that.

Eguan: 15:50

uh. Anyway, there's a whole mechanism design problem here.

Julian: 15:54

paper paying not to go to the gym by is it? De and Mom in

Chris: 15:59

yeah.

Julian: 16:00

AR 2006. think

Chris: 16:04

Yeah, there's probably that that, that. paper about picking your kids up from daycare and paying to, to delay. It's like you can pay 15 to pick them up. I don't remember the paper, but they did an experiment where they wanted to encourage people to pick up their kids on time. So they started charging them 5 for every like 15 minutes or something. They were late.

Phillip Black: 16:22

The moral philosopher Michael Sendell from Harvard.

Chris: 16:26

Wow. Amazing that you were able to stick it. I didn't know the author. No, that's super helpful. I,

Phillip Black: 16:32

a bunch when he talks about the, he published this book called The Morality of Markets, and he references this as like trying to the market mechanic fundamentally the ethics. Sorry, go

Chris: 16:43

yeah, it was probably a list paper, the original one, to be honest. But yeah, it's all of a sudden tardiness skyrockets because now you're paying for the privilege of it's a commodity. It's no longer like a moral transgression. To not pick your kid up on time. You're paying to pick your kid up later. So you're entitled to that because you're paying for it. Anyway, disagree with Phil's Phil's take. I think that was insane.

Julian: 17:08

So anyways, I wanted to one more game that I've been playing. Merge Mayer is a balin based that company was founded by people I worked with closely in the past is

Phillip Black: 17:18

it's

Julian: 17:19

I don't know. I love seeing what other people. Has it been out forever? I played that game and I joined a lot. And I think it's pretty awesome to see what they're up to.

Phillip Black: 17:26

What are they up to?

Julian: 17:28

Just like pretty good. Game elements, so like game design and good meta game and yeah, visuals,

Phillip Black: 17:34

Let me hit you. Let me hit you with, let me hit you with one puzzle question before we get into your experimentation paper.'cause I remember you published this old paper on Jelly Splash. And Jelly Splash. For people who aren't familiar was like this old match game for Moga. It was a linker which is the core mechanic where you drag all the connecting pieces that are matched together. one of the ways the cascading process. And the thing that you had in that game was time-based gates. So like when you would consume X amount of levels, players would put a time gate that could pay to go I could consume levels, then I get to the gate and I could either wait a day or get paid a dollar to consume the next levels. Why do you think that model died out? Why is that not a, why is that not a persistent meta and match?

Julian: 18:18

a, so they, do you mean the model of pay walling, additional game or specifically like mechanic? Yeah.

Phillip Black: 18:27

Pay paywall. And this way, that's a very interesting mechanic that you could apply that meta on single player games too, right? No one in single player does that, which I find curious. You could say, I'm gonna give you X amount of time for free, and then if you get to the levels quickly, I'm gonna time you. I'm very curious about your explanation, but it's just I'm, we haven't seen that meta taken off.

Eguan: 18:47

Can you clarify the mechanic? So if you finish level one really fast, you get to level two,

Julian: 18:52

I think it's inspired actually by what we've seen early in, in the days of news going into freemium. When different news providers started offering content online and they started introducing paywalls, right? So you get 10 articles for free and then you have to pay. So here it's a bit like you get to play 40 levels for free and then you need to pay to unlock the next trench of levels. was definitely in Jelly Splash back in the day. I don't actually know. It might still be in there to be honest. There was when you could unlock, then you could wait. unlock by inviting three friends or by paying 70 coins, which amounted to roughly$1, I think. And did run experiments with what options are an offer there and so on. To be honest, I don't actually fully recall what we found, but why is it that out? I think actually it has died out because the more attractive, more elegant, more seamless way of monetizing is, goes back also this management science paper we discussed. You rather introduce like severe peaks and difficulty and then, make money from consumables that people use to, to, boosters, whatever special pieces and so on. It just feels probably a little more organic than having these paywall thingies. But it'd be super interesting to test this out like in a rigorous way, I think.

Chris: 20:09

I could imagine you lose the tail if you cut everybody off at a specific level and you say you've got to, you've got to turn or pay money. You probably end up with a meta where you have non payers who just cycle from game to game versus maybe if you retain them for team or days, maybe they would have eventually converted or something like that.

Phillip Black: 20:28

multi home almost, right? You can have three games going and

Chris: 20:32

Yeah, you just, yeah,

Phillip Black: 20:33

I think the missing link here is loss aversion and the discovery of win streaks, So most games are about win streak. I don't believe Jelly Splash ever had it. And so now it's all about making sure your win streak doesn't get hampered. And I think that actually combines really well with. Peak difficulty. We've been hinting at this whole episode. We've been talk, I don't know. I completely agree with that theory, I also think you've gotta frame it like in psychology, honestly, that there are s-curves, We've talked about we talk about s-curves all the time in system design, sigmoid curves, like there is difficulty that peaks and then it slows down again, that it peaks up again. That also appears to be where match is also ramped. And I think that's also where win streaks fit in, Because you can increase your win streaks, you can increase momentum. Not only that you can provide skis, you can make everything go faster. So I think there's a lot that'd be very interested in this operations paper, but I think you have to marry it. The social theory that we've had for a while which is flow state, is this optimal period almost of human bliss that I don't think anyone else is taken seriously, I mean to Eric's point, isn't there something that games has something that really does have something to say here about, about flow, like worker challenge, about utility theory. I feel

Julian: 21:46

Yeah, for sure.

Phillip Black: 21:47

Jeremy Bentham and The Utilitarian should be going like bat shit crazy about games. I think we have way more to say about games than any fucking neuroscience bullshit, neuroeconomics bullshit that's been like propagated in the last fucking decade. That's like a research dead end in my opinion.

Julian: 22:02

I am just curious, like when you say like these winning streaks that Jelly Splash didn't have, what do you mean exactly?

Phillip Black: 22:08

So like the whole theory of match right now is that as you have each consecutive win, you build up a booster that is applied on the next level. That booster is cumulative. So by win streak number three, you have win streak number one's benefit win streak. Number two's benefit, and the win streak number, three's benefit. And so if you were to lose a level, that means you're gonna have to play at least three more attempts to build that up again. And so that becomes rather costly. And so now I'm going to spend gold to ensure that I win on the first attempt and I don't lose my wind streak by buying extra

Julian: 22:43

I.

Phillip Black: 22:44

using boosters.

Julian: 22:45

See. I see.

Phillip Black: 22:46

the extension of that. Actually, I don't know. We should talk about this in relation to your experiment paper.'cause one of the things that you say in that paper is about disrupting the player experience, which we should talk about because there's been a lot of like development in Match that I think strikes at the heart of that, that I'd be interested to get your take about.

Julian: 23:05

Sounds good.

Phillip Black: 23:06

Julian, can you tell us little bit about the paper? Gaming and Adoption Guide. It looks like you've been floating it around. We're gonna have a link to it in the show notes, but in your own words, what are you trying to do here?

Julian: 23:19

Ryan Lucked over at APO and I were actually chatting at Coach, so that's the conference on digital experimentation that's at MIT each year, roughly October. Think last year it was October. And among other things we're talking about how gaming companies actually run a ton of experiments and have pretty sophisticated experimentation programs that are really rich and advanced. Many people in other industries have no clue that's actually happening. It's really, per your point, I'd many people outside of don't know what's necessarily going on. And so Ryan invited me to write an overview article about experimentation gaming for. For their learning platform that's called Outperform. It's like an experimentation learning platform. And I think the article on there is called Gaming Companies Run Thousands of Experiments a year. He Is Why and how And when I started writing that I asked Ryan specifically, do you want like a comprehensive piece that gives an overview and allows people to get a lay of the land or do you want with a punchline? That's something clear, like a clear thing that sticks with people and they wanted something comprehensive. So I went more this way and it turned out way longer than I expected it would be. And I still don't cover nearly everything. I think it's a start though. And I can't wait actually to talk with you about it'cause I don't think there's a better crowd to talk about this paper with.

Phillip Black: 24:35

What do you hope after this paper is disseminated that people take away from it? I know it's a guide, but am I supposed to read this and then go out and be like, okay, where I am in the game development process. Lemme go through the guide and now generate some ideas based on framework here.

Julian: 24:51

that's one way. It could be you are running a game studio you have a sense that you're maybe not, and you're like the general manager of it or whatever. You have a sense that you're maybe not using experiment experimentation successfully yet, or that there's like untapped opportunities. I. could dive into this and kind of use this to challenge your analytics team on what are they doing? What are we doing in these different areas where experimentation could be used? I think that's actually the main way I think about it. Or if you, even as an indie, if you're publishing your game and you wanna understand where could I actually start using experiments where I, what would it even be used for, et cetera. Those are the kinds of the audience I had in mind with this.

Eguan: 25:29

I like how you lay things out and you're like I really like the where is experimentation relevant? like, how can we improve game all that. I made a comment about it seems like it's written from a marketer's point of view. And I think the thing that jumped out to me is you had this grid of the four Ps, right? Product, place, price, promotion and then like. The game dev phases, And you had this matrix and like the Bullet points. in the matrix, for example, some of them like game production, that's, build the whole game, But that's one bullet point. And like at that point, 80% of the dev resources, 80% of the man hours or labor costs or whatever are going towards building the game. Like engineers, artists, qa, and then there's a bunch of bullets that are like a small thing that like, a couple people in marketing or a product might do ad monetization or, maybe that's a big one, but or like select tech stack, right? That's a huge thing that the game has to decide what engine are we gonna be on? What kind of integration are we gonna have? Yeah. Anyway that, that was all I meant by that is that it, it seems like,

Phillip Black: 26:24

all it was.

Eguan: 26:25

If this grid is meant to be like, this matrix is not like. evenly weighted. Is that all I'm trying to say?

Julian: 26:29

Yeah, no, definitely not. Most of the so let's start with one thing. So one thing I distinguish in paper that I'm not sure is very commonly done is like quantitative and qualitative experimentation where qualitative experimentation, this idea that you, very early on during prototyping or concept testing, you actually run the game. You have people play the game and you just look for major effect sizes. You don't actually try to quantify any distribution or whatever. You just want to get that wow moment or see where people struggle. You just look for massive effect sizes, not caring about uncertainty, really around these effect And then quantitative you in market, when you actually,

Phillip Black: 27:08

experiment? the experiment? That's an optimization problem. am I experimenting with?

Julian: 27:12

So you could think about like in the core loop I mean I need to now cook up some example, but it would be something around really fundamental to the game to the core of the game design game's where you maybe have two different hypotheses about what would with the market. So then you do a couple of play say with five people for each variants. And you, you just get a read from that. they struggle now? Do they complain? Do they, are they like, oh wow, this is amazing to play. And if you notice

Phillip Black: 27:40

was

Julian: 27:40

yeah. Does that make sense?

Chris: 27:42

Just play testing, right? You have,

Phillip Black: 27:44

to, to Julian's point, this is something I don't see game designers do enough of. So here's what I don't see game designers set out with a play testing theory every single week and say I'm testing this theory, this play test, right? It's about testing a of the a and then getting feedback. And then optimizing the feature based on that aren't sitting there saying, what if I tuned this style to 80% on a Wednesday? It's about new feature incorporation. And I think Julian, like I, I think being more methodological about I'm gonna tune this and tune that for exactly what Julian mentioned, which is massive effect sizes. Like just start hitting fucking buttons until something interesting happens.

Eguan: 28:24

There's a thing by the civ guy. Who? Sid Meyer. Yeah, Who says that if you're gonna crank these knobs in places, like you really crank them, basically double them or cut them in half. If you're doing like a 10% change, you're not gonna get the effects size, not notice in the play test point. You're looking for these big effects on a small sample size. So you need to have big swings. Yeah.

Julian: 28:44

Yeah, exactly. And at that point it doesn't, you don't have a life game in market yet where you might really burn your customers if you do a test like that. So it's the time to do it.

Chris: 28:53

Would you say that the planning phase is mostly qualitative? I know you in the matrix later in the paper, you differentiate between qualitative and quantitative. I'd imagine most of the stuff that's happening in the planning phase. And even the development phase is qualitative research. We're not really able to run a, like a large sample test when we're. Planning without, the game design with six designers.

Julian: 29:16

Definitely for now it's qualitative. One thing I'd note there, and I briefly touch on it in the very end of is like, if we think about gen AI has been done in games for a long time, like procedural content generation and building bots that can play the game, they can also propose and develop new assess how this content would be received by players given their preferences as observed in telemetry. If we think about that kind of technology. As we may be able to infuse it more into the planning and development phases, we might actually get something that's more similar to quantitative experimentation also where you simulate basically like a play a landscape given on personas you believe you'll encounter in the market, et cetera. But this is complete future talk, I don't think with there yet, but I think be that in the future Think is really exciting. But yeah, generally for now planning and development is yeah, and then

Phillip Black: 30:10

And then

Julian: 30:11

experimentation to merr. That is once you actually have decent sizes, which could happen start happening during soft you your game out in countries and you start having, 10,000 users or whatever, that's when you can start doing experimentation, which is really what people think of as AB testing, I'd say. And then you do the typical statistical analysis of that. If you have rich Data. and sufficient sample sizes, of course also start understanding treatment effect heterogeneity and like personalization potential for different experiments. But yeah, this is are leading to far. But the idea with this matrix was simply to say Hey, we take a key concept from marketing that's being used to understand how firms can interface with customers. frankly, like experimentation is about learning from customers, right? We just had it either, both in the qualitative and in the quantitative variant. It's really about learning. From customers about the market. So yeah, we take this marketing mix and marry it with a game development process to then look at what are now in, in each of these key have some relevance for experimentation. And so one thing I want to call in that regard is, for example, if you look in the planning phase in in the price like price row, we, I have pricing and monetization model there. This is not that people would commonly think about, but I think this is why this framework is, can be somewhat useful. So even when you decide if you're gonna go free to play or not, you're already making a decision with major implications for how you'll be able to leverage experimentation in the future. Only with free to play will you be sure to get the sample sizes to iterate on free players and then to also really iterate on conversion and then paying players, et cetera, which you may never get to if you decide for a premium monetization model. And yeah, this is the of this matrix, like what are the different tasks that become in these stages that have something to do with how your experimentation unfold.

Eguan: 32:04

So I guess part of my gripe with that is model, you have it over here on the far left in the planning phase, But that could. totally evolve as the game evolves, Like you might have a game concept like it's gonna be this type of game. You develop it and you find out, your original hypothesis was off and there's a different part of the game that's fun, As the game evolves in the development cycle your pricing and monetization model might totally change. And so it, you probably don't wanna lock yourself in early in the planning cycle.

Julian: 32:30

And this isn't supposed, this isn't saying now you make this decision, this is set in stone. But it's like the point in time when you first wanna think about it. And sure, if you then later decide to change your pricing model because you think it's gonna lead to a much more appealing game then you should just be aware that will impact your experimentation strategy as well. That's really the point of it. It's not like you need to tie this down then. And that's it.

Chris: 32:52

We're thinking this is the experimentation guide to game development. Where does experimentation come in there? That's just idea or ideation, right? It's just oh, what are the different ways we can do this? It's more of a design question than it. is like an experiment question.= So like with the Matrix, we're talking about. Deciding on a monetization model from the very, very beginning. How's there any sort of how's there any sort of like experimentation or even like iteration that could be done there? I understand how we can test our game design at that stage. And there's experimentation there. You have two different builds. You test the two different builds. What, what kind of experimentation happens at section one point. Six,

Julian: 33:35

There's no experimentation happening at that point. It's more about that you should think about it because it will impact your experimentation strategy and how you will be able to leverage. So really this matrix is like for strategic decision making on your experimentation

Chris: 33:49

okay.

Julian: 33:49

And this is super useful questions'cause I'm so much at this from this point of view of a business strategist who's thinking now, like how do we apply Yeah. So this would be hard to think it

Chris: 34:01

There's two different kinds of vectors that I'm that I'm thinking about with respect to games. There's where are you in your development pipeline and then where are you or timeline? And then where are you like on the spectrum of games? Are you like a premium first person? Are you God of war or are you candy crushing? You have this like spectrum where candy crushing God of war on the opposite ends. So with respect to monetization is there any, what kind of experimentation is a premium game able to do, especially when it comes to monetization, right? We can think of a million. We know how King, is running experiments, left, right and center on their platform. But how does that. It's like God of War, is the example I use. What kind of experiment can we run there? We're not doing price optimization for premium games in this industry. We know that it's 60, 70 bucks. Those are your two options. I

Julian: 34:47

yeah, so what comes to mind is ad monetization still gonna be something you wanna probably optimize in a premium title. the other thing is DLC or other content packs that you sell later on where you want to do pricing and promotion, te promotion testing. And then of course the maximizing fun or, revealed fun I guess engagement or playtime through dynamic difficulty adaptation or different matching systems that decide like the structures in the game evolve can be impactful areas of experimentation. Also in premium titles, I think.

Phillip Black-1: 35:23

Can we flip the whole model on its head, right? What if we took a different approach and we went inductive here? And instead we asked when in these stages, planning, development, launch, gaming companies do experimentation right now? that approach, I think what I see here that I don't agree with the paper that you were talking about earlier. Gaming companies do thousands of experiments. Literally that is true. let me ask you this. If you were to compare your time in gaming to, let's say your time at Meta, which I know you were also in traditional tech. Don't you think like we get a lapped in terms of the amount of experi experiments that are run, let's say per ad count because AAA gaming doesn't run, couldn't experiment their way out of a fucking box. You think seeing someone at Blizzard is running experiments? Give me a break, right? That ain't happening. Call of Duty does run experiments, but I think the amount of experiments they would run probably could fit on, maybe both of your hands per year. I look at tech, running hundreds, feels like we're behind in this. Not ahead, not to say there isn't something that we could learn, but it feels like we're actually

Eguan: 36:30

yeah

Phillip Black-1: 36:30

worse at

Eguan: 36:31

I think Julian's the section six. All right. I'll let you talk about it. But he talks about like why they're sometimes hard in games.

Phillip Black-1: 36:37

I I get But I feel to jump to that, so the things that you say is you talk about the here at the end the and when I read a lot of these challenges. Sometimes I understand them as notes about like causal chain in gaming makes of sense, When you run an experiment, there's all these causal events that you could trigger because it's an omnidirectional experience, right? Like you do something, then something else happens, then you do something, and so you're disrupting a chain of events. Totally makes sense. That doesn't happen in other forms of experimentation. Sometimes, but there are other times that I feel like some of these criticisms are ones, I hear a lot about experimentation in gaming and it's one Bill Grosso talked about when he was on too, about the objections people make, for instance, disrupting the player experience. Experiments do not break immersion or alienate players. Use subtle, incremental changes for testing. Like this strikes at the opposite end of everything I'm doing, at least at the experimentation group. Like we're trying to take as big of swings as possible. We don't want there to be subtle changes. We want there to be big in loud changes. Exactly to what you were just saying, right? We were just talking about this,

Julian: 37:46

Yeah, but

Phillip Black-1: 37:46

take big swings to see what the effect is in the supply chain, right? You want it to be visible,

Julian: 37:51

big swings were for qualitative experimentation when we're not yet in market. So I think,

Phillip Black-1: 37:56

so

Julian: 37:56

yeah,

Phillip Black-1: 37:56

isn't it more valuable to run this in market? If when you run an in the more important, and the connection revenue is even more

Chris: 38:03

mean, there's, I

Julian: 38:04

really on how you set up. And I think I mentioned it also in, in this. at some point. But so if you have a portfolio of games, right? Let's say you are running 20 life games and some of them are usually successful, sure you might be to run some pretty drastic experiments on the smaller ones and try to transfer learnings to the bigger ones. But if you really just have one or two big games out there that are driving revenue for your company, you probably want to go incremental. Those are things that you should definitely think about and especially when it comes to gaming communities. Gaming communities are very, can be very tight knit and they're rightfully sensitive when it comes to being treated unfairly. So I think this is just what this point wants to highlight. Be aware of this going on and then think about give them specific of your company, what makes most sense.

Chris: 38:52

mean, there's a retention risk, right? If you like shock the system, think about how pissed off people get when you, there's like a slight rebalance in like Overwatch or something. Everybody freaks out because this weapon is now doing 10 percent more damage than it was. Yesterday, maybe there's maybe that risk isn't as large as I think it would be

Phillip Black-1: 39:10

I don't think it is. I like there was this famous experiment, again, let's think about taking this from the inductive approach. So there was this famous experiment that was published on Kotaku that Zinga ran in one of their racing

Julian: 39:24

SI raising two? Yeah.

Phillip Black-1: 39:25

Yes. C star racing. Two, the, these a fast and furious car, if you

Julian: 39:29

Yes.$5, 15 and 35. I think they did.

Phillip Black-1: 39:33

Yes. Oh my God. Your memory is experimentation gaming pro. If you remember that article got leaked but okay. So what? This is what I don't understand. Like prices change all the time. Like you can look at Amazon Why do we give a shit about the fact that players notice this? This is a normal part of living in a digital This is happening

Eguan: 39:55

I'll.

Phillip Black-1: 39:56

every other platform.

Eguan: 39:57

I'll jump in here first on Amazon. They actually used to price discriminate based on your, like tracking cookies. And they stopped because they got so much flack for it. They do change the prices, but everyone sees the same price, right? And there's other ways they like subtly price discriminate indirectly. But the main issue I see for the experiment validity. When there are social spillover effects, can't just assume oh, treatment effect a, treatment effect B, here's the delta, the impact, right? Because information spillovers between A and B groups affect the results. example, if you price differently, if sees, realizes the are different, then okay, the experiment works. But if the people talk to their oh, you got$10. Oh, you got$35. Oh, and then it changes their evaluation, changes their behavior, and all of a sudden the effect you're measuring is not effect you're trying to isolate. social a game is, the more people have community forums where they talk about it, the more they talk about it with their friends the more multiplayer it is, more likely these social spillover effects are to

Chris: 40:54

such the, social aspect of gaming that makes it difficult because you don't really have that. I'm not like talking to my friends about the price that they're seeing on Amazon. I'm not like,

Julian: 41:02

social and relatedly the emotional aspect. And I think this is also why, you can say it's fine if people upset about that car being priced at 5 15 35, but that's not really what you want. If you want to be known as a customer centric company, This car is like a highly emotional object for somebody playing CS r racing. I, I was a very avid CS r player, both one and two. And if you just take this thing that the whole game is about and change price so drastically by Factor vii, this is gonna upset people and people are gonna churn. They're literally not gonna give you your wallet.

Phillip Black-1: 41:35

Were you the guy who were in the price promotion paper that did exactly what you just suggested though, and radically changed the price of goods? We saw the same thing in the King paper. To be fair, the king paper was about quantity discounting. So is that, what is, so do you think David Nelson gets away with it because they do quantity discounting rather than price discounting? we should take away?

Julian: 41:54

No. Wait. So we discussed that paper and my price promotion paper at length in the previous in the previous recording we did. So the quantity discount thing is super interesting, but you only really reach the people who buy packs that are larger than the small that whole experiment, that treatment doesn't even reach all the players,

Phillip Black-1: 42:12

extensive margin. I think that completely makes sense. I think that was a very fascinating conversation we had about

Julian: 42:17

Yeah. And but just on, on my paper and this goes back to what Eric said, like Amazon only does dynamic pricing. They change the price over and do that substantially, right? Like by factor or whatever. But they don't charge different prices at the same time to in the same geolocation where I think that's usually at the country level. And so importantly in this Price Promotions paper we discussed that I have, we also really only changed prices over time when there was some targeting terms of what discount pack they were targeted with the players. That's fair. But it was more actually looking at what happens if we changed prices over time and really do that in a fully randomized the long time. And then basically we just saw that the intertemporal substitution is much lower than we would've expected based on existing research. Sorry, like I wanted to add that.

Phillip Black-1: 43:07

No fair. But those weren't subtle changes.

Julian: 43:10

true. But no, they weren't. Yeah, that's fair.

Phillip Black-1: 43:14

talk about it.

Julian: 43:15

Yeah. And we did though,

Phillip Black-1: 43:16

did you find

Julian: 43:17

So we monitored the forums and customer very closely. Both here and honestly in all targeting experiments I've run, we tend to do that. there's also certain you can do in terms of how you design a starter pack or bundles. That, that you don't upset people as much necessarily. But yeah, I think that's crucial to monitor that'cause

Phillip Black-1: 43:37

I've actually found Eric the opposite. To be it's not that players are pissed, that they want the offer, Because this is sometimes was get lost, like in all these conversations about price discrimination that I thought I. Will Grosso was really good at bringing up, which is that most of this is about cutting price, right? about increasing like for seniors. this is about decreasing price and getting to elastic And it's usually the case that like when you offer this, players go to customer support and they're like, oh, I heard about this sale. My friend got, I wanted this, I want this offer. They, they, want the, it's not anger

Eguan: 44:18

They're anger that they're angry. That they didn't get, yeah, the jealous. They didn't get the good offer.

Phillip Black-1: 44:22

I, I think very much true. I think like the Zynga offer just stands in really interesting contrast to that.'cause that did get ol, it was Kotaku take it what it's worth. But I guess there, there is some sense like prices are sticky. I think nominal prices are sticky. I think shadow prices sticky, if makes sense. And I think quantity discounting is almost a shadow price, like you are changing price per unit. But I think the nominal prices really matter. I think consumers benchmark on that.

Chris: 44:53

What do you mean shadow prices are less sticky?

Eguan: 44:57

What is a shadow price?

Phillip Black-1: 44:59

If you change the difficulty of a Match three game, or let's say this, let's you decrease the amount of goal or the amount of boosters that are given away in a match three game for free, in some say sense you're increasing the expected price of a booster giving away less for free. So that could be one way that on the supply side, you're effectively changing the price of a booster, or if you change difficulty, you're also changing the price Because in a match level, you can conceive that players are paying to progress by buying extra moves and extra And so if you increase difficulty, you're increasing the price of a level. But that isn't an explicit It's a shadow price,

Chris: 45:36

Huh. I thought a shadow price was a marginal cost. Oh, I thought the shadow price was the it's like the marginal or sorry, the opportunity cost of, cause there's also a shadow price in the context of like classic optimization problems, like when you're solving Lagrangian, like you literally have a shadow price Lambda, but

Phillip Black-1: 45:55

yes.

Chris: 45:55

is, that the, same thing that you're talking about? The shadow price is more elastic. The shadow price is more elastic. I'm, like trying to wrap my head around what, you, what, that.

Phillip Black-1: 46:05

what I mean by that is to changes in explicit than they are into explicit prices. So to give you an example, like we rearranged a grocery store and we move like the farther shadow

Chris: 46:20

Yeah. Okay. I see what you're saying. Okay.

Eguan: 46:23

or less, you mean like change

Chris: 46:25

Opportunity cost.

Eguan: 46:27

without upsetting people like

Phillip Black-1: 46:29

I think that's what the Zynga versus King experiment would show is that, even just do one removed, right? You were you, it was an explicit price, right? The amount of gold bars was explicit. When you bought those

Eguan: 46:43

Yeah.

Phillip Black-1: 46:43

but players, most people price in those nominal prices, the actual real amount of dollars wallet is what consumers, anchor on

Eguan: 46:54

Yeah I totally agree with that. I think actually a lot of these you can design around by using shadow prices. So for example I mentioned this in the show notes. Imagine you wanted to test a progression rate change, right? Plus minus 10%. If you change it so that people get, some people get 90%, 90 XP per level, and some people get 110 XP per level. Players will see it, they'll notice, they'll talk about it. But if you change the odds of a rare loot drop from. Let's say 9% to 11%. They won't be able to tell as much, right? And so you don't have these social spillovers, you don't have these price anchoring and transparency because it's a shadow price change. You can conduct the experiment

Chris: 47:29

Maybe that's an advantage of pricing everything in time. Like a Pokemon TCG is doing. Everything's a shadow price there.

Eguan: 47:35

Julian, I really like in your paper you talk about these problems and propose solutions. I think, as someone reading the paper, I'm like, okay, yeah, there's all these spaces where I could experiment, but what should I be doing? And I really thought this, these, what to do. Sections were pretty strong or common pitfalls, that

Chris: 47:51

one thing. I was thinking when I was. As I was finishing the paper up is okay, who's the audience for this? Because if it's somebody that's at a company that has a million users, they probably already have a data science group or, experimentation arm, that's already doing this stuff. Maybe not, maybe it's a PM and maybe they, they're looking for more juice or they're looking for me, okay, where can we push and prod, but I think a lot of people who are. Who are reading this kind of stuff? And honestly a few of the people that I saw at the mini summit last year, GDC they were more startup be, very small companies. And they were, I remember one of the, one of the kind of complaints that I heard from two individuals actually were at the mini summit. They were like, we're much earlier in our. Development phase, and we have far fewer players. We're not able to run these massive, long term experiments. We're just trying to improve our monetization in a much smaller scale. I'm curious. Does this do you feel like this paper addresses that person? And how does somebody with a smaller player base? You know what? How do they experiment? They clearly do not have the statistical power to be able to actually run, meaningful statistical test. Can they still experiment?

Julian: 49:00

Good question. So I, surely can I don't know if this specific or this framework would be super useful to them other than an overview of what's possible in thinking through the areas where they might be able to do something. So if we look at the matrix and think, yeah, they'd put in the post-release phase and maybe. Realize that price and offer personalization could be pretty impactful and they might not be doing it yet. So that could be starting point then for them to dive adopting experimentation for that use case. For the audience, I think I said something to about that in, in the beginning, but I think it's more for, yeah, like business people who run the strategy of a company to just challenge the data science and analytics team on what's happening in the different areas. Or just if you want to install an experimentation roadmap as a. As a game publisher, you would probably want to consult this or if you Yeah. Do you need a certain level of ambition and scale, I think, to be making use of this. And, but even if you're a small indie, I think still just glancing over it and seeing, Hey, do, what are we actually doing? Did we do marketability testing for our creatives that we want to use in game? And stuff like that could be quite useful. Because it might not have occurred to you because you only did game design in university and didn't think about marketability testing. So I think even there, this might have some value. But overall it's a bit of a work in progress still, and the audience. Yeah, the, I think is in many different areas. Coming back to what Phil said about the maturity of experimentation also in gaming compared to tech. And it's probably true that the large tech companies run more experiments than the large game companies, at least the more traditional game companies. but still the experimentation programs that you have in gaming are super impressive. And just like even coming back to the challenges we discussed, right? Because the systems are so complex and nuanced, you actually have this richness to play with that is just like phenomenal and interesting. And one company I was thinking about specifically is actually Netflix. cause they are known, right? And very open also about their experimentation programs that they use on the streaming side of things. And they are also, as we all know, a pretty big game publisher by now. And, I actually don't know that they are, or like what they're doing on the gaming site specifically. But

Phillip Black-1: 51:19

doing absolutely nothing. Interesting. Don't waste your time.

Julian: 51:23

wow, that's harsh. You mean in experimentation or Generally

Chris: 51:28

Netflix.

Phillip Black-1: 51:31

doing what they

Julian: 51:32

I,

Eguan: 51:32

Yeah. I think Is a big

Chris: 51:34

oh, that was my that was my game for the today. I was going to talk about, I've been playing? Netflix games No, it was fucking horrible. No, they were terrible. No, it's just, it's like a, They're an annoying intermediary, but that's besides the point. Yeah. Yeah. Yeah. I don't know what kind of experimentation they could possibly be doing, like

Eguan: 51:53

probably doing like, placement experiments, like All over the place. The way they do with their shows. I don't think they're doing in-game experiments Yeah, so Netflix does tons of placement, experimentation, the ordering of the rows and which things are at the front and the back. And I'm sure they're doing that with the placement of the Netflix games apps. don't think they're doing in game experimentation. They're actually modifying the games themselves. what Julian was saying earlier, that's where a lot of the interesting stuff with experimentation and games comes from. There's so many more systems and levers and incentives and knobs to turn. But I'm pretty sure Netflix is just buying html five games off, like on the cheap and just like inserting them into

Chris: 52:29

It's really a distribution platform. It feels more like steam for mobile cause I've never seen a Netflix game on my console or on my TV. I've only ever seen a mobile. The only reason I remembered that my Netflix account had games associated with it is because I opened up the mobile app to try and download some shows while I was traveling. And I saw a few games and I had to download those games from the app store via Netflix. And then I had to log into my account, so I couldn't even play him on the plane because I didn't have Wi Fi. So it's just like an absolute fucking shit show but they have some good games on there. Bloons which I guess is premium. My experience was like very, I was like. why do I have Netflix for this? I'm just downloading these off the Internet or off the app store. Anyway, it's very weird.

Phillip Black-1: 53:13

This is the thing that I think I was interested that Julian didn't talk about in the paper is this whole idea of experimentation for optimization or almost as an engineering problem. And then I as almost a Bayesian framework for updating your priors and almost discovering a new space, particularly like in the development phase. So like when I think about a lot of tech experimentation. It's about a very defined problem and you're basically just trying to fill the content at the right time for the consumer, That is Facebook ads. That is also Netflix's algorithm, All these multi-arm bandits that you hear about. Almost no gaming companies doing a multi-arm bandit where you're just throwing thousands of variants into a, a funnel and then seeing which one is the winner. Very few gaming companies are doing that, where this is like the go-to model for almost all. Tech companies to, let's say, pick the right thumbnail for you on Netflix or to recommend you the right amount of content, It's a very defined optimization problem. Let's run millions of these. It's an engineering problem, They've turned it, they've turned experimentation to engineering. Whereas I don't think gaming companies do that a lot because there's so much more uncertainty with games. Like no one is doing experimentation. To figure out how to make a particular creative product inside of Netflix. There's this big famous example of House of Cards coming from their algorithm that people liked David Fincher and people liked Kevin Spacey, but the director for House of Cards, David Fincher. Do you think he was looking at fucking experiments on the Netflix platform? No, there was no role that experiments played in him coming up with a creative medium. And I think that's not the case in gaming because there's this whole uncertainty about what you're building until you build it. And there's this famous quote from this valve psychologist Mike, something or other, and he is on Twitter and he actually just left And one of the things he said is that we see our play tests. As ways to validate our hypothesis about our games. And so to me it's about having a model like this is where I think it's really important is what is your model about like how this game works and then how can we use experiments to challenge or to redefine that model. Like Julian, you were talking, we were talking about match, right? I think to run an effective match experiment to really figure that out, like you need a falsification model, you need to do science. Like you can't talk about experimentation without science. And I don't think you can talk about science without falsification or theory. like even when we go back and we think about like where are games actually doing experiments? I think a lot of it is falsification. Have you ever heard of Geek Lab?

Julian: 55:39

Have.

Phillip Black-1: 55:40

You would fucking love Geek Lab. You should talk to them. So what they do is they create fake store pages to test creative for mobile games. So if you want to build a match game, they'll test like a bunch of different fake absolute, but real app store pages and you can measure impressions and see down the funnel. It's not just people clicking on ads. also clicking fake store pages, right? Like, how much of the funnel can I test and see what CPIs end up like? And that to me starts with a theory about what is the product that's going to resonate with consumers? What is the creative and the only time I mean it's not multi-arm bandit, but hyper casuals does this really well, right? Like they'll take a core and they'll re-skin it a hundred different times and see which is the creative that pairs best with the core gameplay. That stuff is a little more engineering E, but I think a lot of the fun experimentation comes from the model.

Julian: 56:31

Yeah. And this creative stuff also is an area though, where game companies can use bandits also, right? I say like part of. Why bandits aren't

Phillip Black-1: 56:39

Okay.

Julian: 56:40

as common maybe in gaming is actually the complex nature of them and the challenges we discussed right around engaged communities, the complex systems, the long-term effects that matter. If we actually go over to the Game Data Pros block, so game data pros.com/block, we have a case study on there. For Bandit experiment, we ran with a gaming company around creative optimization. And we actually had some interesting findings doing that.

Eguan: 57:03

I wanna touch on something on that note, which, what Julian mentioned earlier, which is about the qualitative, the low sample size, just big impact. You're not really trying to measure anything explicitly. You're just I made this change. Are people wowed? Did they have a strong reaction or not? Do you consider that scientific Phil? If it's like I made head shots. When you shoot someone in the head, there's an explosion of blood and it plays a sound effect, And I have people play it. And if they react strongly to that, then I think it's a good thing. And if they don't, then it's a bad thing. is that an experiment or?

Phillip Black-1: 57:30

Yeah I don't think that science, as we've generally understood it, I don't think you need a randomized control trial to do science. And we know this because there's so many parts of science, that have come with non testable hypothesis, non results, especially in physics too. So I don't think we can boil down science to just that. I would say having a theory and going out and trying to falsify it, to me is the most important part here. And I think the. The gold standard of randomized control tiles is one way to falsify, and I think it gives you a higher level of confidence than what you just suggested. But he, like the designer would have a theory I think this is something that's gonna resonate with players. It's not really well defined. But I think you can say, okay, did this resonate with What's my strategy to figure out whether or not this resonated with players? gonna talk with them afterwards. Don't get me wrong that, that is so rife with the 50 million things that we could all name off the top of our head. But they had a strategy beforehand and they sought to influence the variable and they tried to measure it afterwards. That, to me, is a scientific loop. it's just a sloppy set of tools to get it done.

Eguan: 58:39

do you feel like the people who are being unscientific about it, they're not being explicit enough about initial hypothesis, not being explicit about what would falsify their theory?

Phillip Black-1: 58:49

I don't think they have model, right? They don't have a model of what makes this tick or they haven't made the model explicit. There's this, I just know it when I see it, which is the biggest load absolute cop out crock cock of shit I've ever heard in my life. Like it's avoiding like putting down what thing game. I think is economists are really good at, is making explicitly falsifiable And so again, he doesn't have to be in math. has to be like, I think this is what matters. And okay, how could we test this? And I think development a good way to do that. Play are a good way to do that. Like that vote from that quote is oh, I think this is what makes the game work. So to give you one example, like deadlock from Valve, right? Recently, it's a moa. It's from Ice Frog, who did Dota? And so there's so many different components in that game. But I be disappointed if Valve didn't for existence, do a play test where they took out a major feature of the game, the in-game shop. For, let's say if you played mobs before, you know that there's this in-game shop, you collect gold. Like they should be ripping out major features like that in their core gameplay. And then figuring out or feeling how that affects all these subsequent causal events. Alright. And that tells you a lot about that feature. Like that to me is a really it's not a randomized control tile, but it is an experiment. I don't know, Julian, does this fit in your framework? Is this out?

Julian: 1:00:02

I wanted to say is at the experimentation group, the work you do there, we should do a session or you should do a some it. I'd love to see like how you, exactly what you just said, like how you bring models and. Falsifiable hypotheses to, to the clients and make sure that leads to, okay. Great. that, that leads to, to better experimentation outcomes.

Phillip Black-1: 1:00:23

We could have David back on. He's the real master. He followed a lot of your advice, right? You can do radical changes, but at King they were funneling the experiments from the games that were dead, that no one cared about anymore. Like that to me, is a really solid strategy. You have a new of something. Test on the old one. Test on the ones that you don't care about, but make it more radical. So he did that. So to give you example a company really take your EA and fifa, they have yearly versions of their game, why is FIFA not testing on like FIFA oh nine, radical experiment that could affect 26? Those games have sizable player bases. Two just came out. experiment on Arto one

Julian: 1:01:03

I think there will be people from EA at the mini summit during You should talk to them about it.

Phillip Black-1: 1:01:08

I sent Andrew Wilson an email once about this.

Chris: 1:01:11

Phil, I want to go back to this idea of scientific method during the design phase. It. sounded like you were saying, like, when you're in the design phase, when the designers designing, they're not using the scientific method because they don't have an alternative. They're not, they don't have a hypothesis that they're technically testing. They're really Just searching.

Phillip Black-1: 1:01:28

ones do.

Chris: 1:01:29

Okay. So I guess two questions. First one what does an actual, what does that hypothesis look like? Are they testing mechanics? Should I, add this mechanic or not add this mechanic? Does this does this make the game better? And then, the second thing is, what is, if we were to compare, if we were to look at two different, two designers that were identical, except for the methodology. One is using this scientific method. And one is looking for the North star. They're going to, they're going to know it when they see it. Do you think that the outcomes are really going to be that different? Assuming that the original kind of like concept remains the same for the game.

Phillip Black-1: 1:02:02

Let me take the first one, which is that. I think it depends on the discipline, right? So let's give an example of like when I was on Battlefield, there were level designers. And level designers obviously wanna maximize player engagement and retention and feedback about the maps being good or bad. And so one of the things that you see a lot of going The snare snipers that are being selected higher than otherwise would be on other maps, and a lot of players aren't finding that enjoy, experience enjoyable. picked from a hundred yards away. Okay, we're gonna change the geometry of the map, such that sniping is less favorable. So might in more hills or more cover. And so we're gonna go into the play test and we're gonna see if there's more or less sniping based on the changes we made. You're not necessarily running like a. True experiment. Like you could call an experiment if you did it, on a Tuesday and then on a Thursday play test, it was different. And then you're comparing the data. That's not a true experiment as we think about it. It's certainly not, a randomized control trial, but like you had a theory, which is that, I think snipers are not making this map enjoyable or getting a lot feedback. And that by me making this change to the map, will change the amount of snipers that are being used. That to me is, a scientific loop. Whether experiment with it.

Chris: 1:03:10

Yeah.

Phillip Black-1: 1:03:11

semantics and different I think the science piece is really important.

Chris: 1:03:18

Yeah. It's funny because we're we're actually going to we're working on a free to play version of our game right now. And we're just basically testing it. The cool thing the cool thing about this is we can launch this game on our, on a side net. Basically for free. So we're going to have the full game. It's just going to be a different, completely different version. So we're exploring how would we redesign the whole entire system, all the numbers using a free to play framework versus the premium framework. And we're it's literally a team of three people, me, and two engineers, and it's like, what's the most efficient way. And I'm getting really frustrated because I'm struggling to find the most efficient way to, experiment with new. Okay, we've tested these stats out. We didn't quite like them. There were some issues with this one trying to converge to a semi decent equilibrium as quickly as possible. And it's really just been I would say probably closer to a bandit. Approach than testing one piece at a time, or I don't even know. I don't even know if it would be considered a bandit. But basically, we're not like testing one mechanic at a time. We're throwing basically entire different worlds together. Okay, let's compare world a to world B. We prefer B go down B. Now we want to compare a C to D. Okay, we prefer C and just keep searching for the optimal. It's algorithm. It's an interesting situation that I'm in where it's man, how do we converge to the optimal at least close to an optimal, a local optimum as quickly as possible without having to waste a whole bunch of resources and we've got 15 testers.

Phillip Black-1: 1:04:49

Let me ask you what that you think are not being run in games that should be run right now?

Julian: 1:04:54

One area I'd say is in social matching. Actually. think game companies rely on self selection of players into the kind of teams or groupings they wanna find. I think you, that could be supported by algorithmic intervention more so through smart designs to sure that, to really optimize also for some long-term outcome like a balanced overall global lead leaderboard or I think there's stuff there that could be done Or not necessarily better, but where there's opportunities to really move the needle in a good direction in terms of skill development for that engagement. through that, of course also monetization ultimately. That's one of the key areas I think where I would love to see more.

Phillip Black-1: 1:05:34

What about on the marketing side?

Julian: 1:05:36

I think a lot of good stuff is already happening.

Phillip Black-1: 1:05:40

So hold on gonna gimme this whole matrice and you're not gonna tell me that you don't think we're out along matrice. I haven't seen a single experiment at the planning

Julian: 1:05:47

Okay. So one thing I would find interesting, but it's a bit complex. So there's this idea of satiation that has been proposed. It's also intuitive, but it's also been in some marketing papers recently on the monetization of apps and app-based games. And the idea is as a player plays or a user uses an app, they satiate with that experience, right? It's either about you read articles or you satiate with playing these game levels and so on. How to optimally manage it. It even goes back to the flow ideas, et cetera we discussed earlier, but it's a bit of a different conceptual angle. And so the idea would be you actually don't want a player to satiate, right? You want to maybe even disrupt their play experience before they satiate with the game. So what I've been wondering, and actually I've been discussing with Chris Ryan, who we talked about earlier who has the paper in science on design is this idea, can it be beneficial to disrupt the continued engagement of players to manage satiation and increase the likelihood to come back later? And probably this is actually also gonna be vastly heterogeneous across different players. And I think even the data that we get, like the behavioral telemetry we get from players would be quite helpful in doing effective personalization on, on that. So this is an area where I'd love to run an experiment. it would be maybe,

Phillip Black-1: 1:07:08

The cake eating problem, right? It's the dynamic optimization key eating problem. Do you eat the cake now or do you spread it out?

Julian: 1:07:14

yeah. And then, but here, because this is the awesome thing about game design, right? Or game making, you can cut off the cake. You can keep the player from eating the cake through interventions like timers or a live system where they run out of lives and so on. And see this is also interesting'cause I feel like life systems specifically used to be super popular, even for monetization. But I don't know that they are anymore. You would know that better than me.

Phillip Black-1: 1:07:40

no, it's, I would say the arcade monetization was interesting, right?'cause the arcade is the first form of MPX, I would argue of uncapped he'd go back and you listen to the stories of how much these arcade machines were making. It was an incredible amount of money, right? The problem, I would argue is that the amount of money you make per unit of time is relatively fixed. So if I insert a quarter, it basically buys me, an expected 30 minutes of engagement. And I think we've unlocked how to make that even higher. If you go to four x, I could spend$10,000 in one day just buying peace shields or just buying troops that are consumables and that they'll die and they regenerate. And then I spend to speed up timers that generate more troops. But I think we found that's just like a more interesting lever. Holding back lives and holding back lives. The problem with holding back lives or holding back engagement is that it's a other right? So match three, if I make money on extra moves start to make lives a strong component of rationing, that means I have less attempts and less attempts means less gold spent, and less gold spent means less revenue. And so I think we've just figured out that the amount we can make per hour is much better if we had more hours, right? Like it solves

Julian: 1:08:47

But let me ask you this then. What in your opinion, would be an effective way to manage satiation?

Phillip Black-1: 1:08:53

So I don't necessarily agree with the theory. I would argue that I

Julian: 1:08:56

Oh, okay.

Phillip Black-1: 1:08:57

players satiate themselves. So for I definitely think this is a testable hypothesis, right? What is a live and a match game do holding everything else constant. That to me clear that I think would help resolve this. And I think for your theory to hold true, be the case that if. Intervene in ration, it's more effective. I would argue that you would need to show that the one with the lives have a bigger, has a higher long run retention. We know it'll reduce attempts, right? Per day. That's the whole point. It should do that. But I don't think that would be the case. I think you would still probably win by players just naturally buying down their diminishing returns each day and then naturally coming back. And I think that's the equilibrium. Games have that free to games.

Julian: 1:09:38

I

Phillip Black-1: 1:09:39

that would be what I throw down on the, on

Julian: 1:09:41

yeah, I would love to run that experiment though. So maybe, yeah. also call out to the listeners, if anybody wants to work together on running this experiment, please reach out.

Phillip Black-1: 1:09:49

Think this is also has like another dimension. So there's like the player progression one we were talking about in this rationing problem. But I would say this is also something that AAA studios face all the time, which is that I have a bunch of locker. know if I should dump all that on D one or whether or not I should spread that out over time. And so that I think is very similar to the problem I just mentioned. But I would almost. my own thinking on that. I do think that sometimes you wanna spread out content like that. Like we, I think you need to change your way. You think about games. I think you need to introduce like a dating model for that. It's almost if you have a bunch of like compatible mates that are introduced to you at one time, right? You can't really satiate each one. And sometimes you get into these. We're gonna stop here, but I would I think

Julian: 1:10:34

I.

Phillip Black-1: 1:10:35

models that might like you get into these different have explanatory powers and different

Julian: 1:10:43

whenever Phil gets embarrassed, he just says, model.

Phillip Black-1: 1:10:46

Hey, at the was having you Julian. guys. Our guest has been dr. Julian. Dr. Julian, where can we come? Where can we learn more about your work?

Eguan: 1:10:54

Yeah, give us some outs. Some links,

Phillip Black-1: 1:10:56

I guess she'll.

Julian: 1:10:57

To sign up for the mini go to game data pros.com. There should be an overlay up where you can register about me. Yeah, LinkedIn is good. Or just Google me. I have a personal website where you can find my research also. And then Google Scholar.

Phillip Black-1: 1:11:13

from the show

Julian: 1:11:14

yeah, Google Scholar is pretty good'cause it has all my papers if you're interested.

Eguan: 1:11:18

Cool. Thanks I.

Chris: 1:11:20

Yeah.

Julian: 1:11:21

Thanks for having me.