[Announcer]: Welcome to the Analytics Power Hour.

[Announcer]: Analytics topics covered conversationally and sometimes with explicit language.

[Michael Helbling]: Hey everybody, welcome.

[Michael Helbling]: It's the Analytics Power Hour.

[Michael Helbling]: This is episode 285.

[Michael Helbling]: You know, for some reason, I have always been Bayesian when it comes to statistics.

[Michael Helbling]: I didn't arrive there on purpose.

[Michael Helbling]: It was just sort of intuitive.

[Michael Helbling]: And I also wasn't paying super close attention to my priors along the way, so who knows really how I ended up this way.

[Michael Helbling]: On this episode, we're going to do something incredibly brave and slightly reckless.

[Michael Helbling]: We're going to talk about Bayesian statistics without scaring anyone who doesn't have a PhD.

[Michael Helbling]: And if you do a PhD and keep a stress ball handy, it might get turbulent.

[Michael Helbling]: But before we get into it, let me introduce my co-hosts, Moee Kiss.

[Michael Helbling]: How are you?

[Moe Kiss]: I'm good.

[Moe Kiss]: Thanks for asking.

[Michael Helbling]: Nice.

[Michael Helbling]: I'm hoping you'll be my buddy on this episode.

[Michael Helbling]: And Tim Wilson, how you doing?

[Michael Helbling]: I'm going great.

[Michael Helbling]: I've never come right out and asked you, Tim, but are you or have you ever been a frequentist?

[Tim Wilson]: I beats the hell out of me.

[Tim Wilson]: I live my life for p-values that are less than 0.05.

[Michael Helbling]: That sounds like something a frequentist with.

[Michael Helbling]: Anyways, I don't know.

[Michael Helbling]: Okay, I'm Michael Helbleck.

[Michael Helbling]: We needed a guest, someone who's worked.

[Michael Helbling]: We've been appreciating for years now in the area of media mix modeling.

[Michael Helbling]: Michael Kaminsky is the co-CEO of ReCast.

[Michael Helbling]: He's also one of the organizers of the locally optimistic Slack community.

[Michael Helbling]: And he was previously a guest on episode 232, and now he is back once again.

[Michael Helbling]: Welcome back to the show, Michael.

[Michael Helbling]: So glad to be back.

[Michael Helbling]: Thanks for having me.

[Michael Helbling]: Really excited for this one.

[Michael Helbling]: So you heard Tim say it.

[Michael Helbling]: He is a huge fan of p-values and hacking them to get the answers he wants.

[Michael Helbling]: Why is he wrong?

[Michael Helbling]: No, I'm just kidding.

[Michael Helbling]: We're gonna talk about Bayesian, maybe just start in on sort of a little bit of an explainer into Bayesian statistics.

[Michael Kaminsky]: Yeah, so this is a great question.

[Michael Kaminsky]: So there's a lot of different directions that we can take the conversation.

[Michael Kaminsky]: I'll try to do a little bit of an overview of how I think about Bayesian statistics and how this fits into the sort of universe of different types of analytical strategies that people might take.

[Michael Kaminsky]: maybe we'll start with a little bit of history.

[Michael Kaminsky]: So people, I think today, hear Bayesian statistics.

[Michael Kaminsky]: And if you came up through college and maybe graduate degrees like I did, Bayesian statistics sounds like a new thing, right?

[Michael Kaminsky]: You maybe didn't learn about this in university.

[Michael Kaminsky]: When I took statistics classes and econometrics classes in university, never talked about Bayes, never really talked about Bayes theorem.

[Michael Kaminsky]: And so when I first started hearing about it, I was like, oh, this is some new thing that people invented.

[Michael Kaminsky]: But that's not really true, right?

[Michael Kaminsky]: If we think back about the history of probability and statistics, Bayesian quote-unquote statistics is sort of the original type of statistics, right?

[Michael Kaminsky]: It's a very sort of really simple mathematical approach to thinking about how do we calculate probabilities, right?

[Michael Kaminsky]: How do we estimate the probability of something happening in the future?

[Michael Kaminsky]: And really, it's the frequentist approach.

[Michael Kaminsky]: largely spearheaded by this guy, R.A.

[Michael Kaminsky]: Fisher, around the turn of the 20th century.

[Michael Kaminsky]: That was sort of the new statistics at the time.

[Michael Kaminsky]: And because this frequency approach was very convenient for a lot of important questions that were relevant at the time, it really gained a whole ton of popularity.

[Michael Kaminsky]: But Bayesian statistics has been around for a very, very long time.

[Michael Kaminsky]: It's sort of the original type of statistics, if we want to call it that.

[Michael Kaminsky]: But it has been resurging in popularity in recent years for some very specific technical reasons, which we might get into later.

[Michael Kaminsky]: So what is the idea behind Bayesian statistics?

[Michael Kaminsky]: The way that I like to think about it is that Bayeans tend to think from simulations, where what we want to do is in general as a Bayesian is we want to build a model that describes the world.

[Michael Kaminsky]: Here model, you can generally think about it as simulation.

[Michael Kaminsky]: I want to build a simulation that I think describes the world.

[Michael Kaminsky]: What are the rules of the natural universe of the thing that I am trying to model?

[Michael Kaminsky]: And then I want to compare the implications of that simulated world with actual data that we observe.

[Michael Kaminsky]: and then try to learn about some parameters of interest from comparing the simulated world that I code up, right, in generally in some software programming language with the data.

[Michael Kaminsky]: And so again, a lot of people when they're thinking about statistics or thinking about A-B tests, they're thinking about regressions, but there's all kinds of statistical models that we can imagine that are way far outside of that, right?

[Michael Kaminsky]: If we think back to like the COVID pandemic, a lot of people did really interesting work trying to model the spread of disease.

[Michael Kaminsky]: And so you might have some biological model where you're trying to think about, okay, there's some coefficient of spread and there's some amount of people being treated and not.

[Michael Kaminsky]: And then what we want to do is we want to see, well, how well does this model fit the data?

[Michael Kaminsky]: What would the coefficient of spread have to be in order to explain the pattern of data that we see?

[Michael Kaminsky]: That is a very natural Bayesian approach to statistics.

[Michael Kaminsky]: We're going to start with some model informed by science, informed by our understanding of the world, and we're going to compare the implications of that model with data that we actually observe and then use that to infer things about some parameter of interest.

[Michael Kaminsky]: I'm going to pause there.

[Michael Kaminsky]: Hopefully, that was like a reasonable summary.

[Moe Kiss]: Can I just jump in there?

[Moe Kiss]: Because I am going to talk multiple times probably about the wonderful Cassie Kozyrkov, because I feel like she has this way of explaining things to me like I'm a fifth grader, which is perfect.

[Moe Kiss]: And one of the things she said in explaining a similar, like in an effort to explain it, she was like, it's just a best guess.

[Moe Kiss]: And like, [Moe Kiss]: How does that sit with you?

[Moe Kiss]: She's like, when it comes to Bayesian stats, it's about us making a best guess.

[Moe Kiss]: So it's like, if that is your best possible guess, why would you use any other guess other than your best guess?

[Moe Kiss]: Even if like there's the possibility it could be wrong, it's still the best guess you've got.

[Moe Kiss]: And there was something about that that kind of sat with me well.

[Michael Kaminsky]: So I think I probably get where Cassie is coming from.

[Michael Kaminsky]: I don't think that I would explain it exactly right because I think that that undersells Bayesian statistics a little bit because it's like, oh, it's just guesswork.

[Michael Kaminsky]: And I think a lot of people have this [Michael Kaminsky]: association of Bayesian statistics specifically with the idea of priors.

[Michael Kaminsky]: They're like, oh, you're just guessing, you're just putting your own beliefs out there, and you're not really doing any analysis.

[Michael Kaminsky]: I would shy away from the idea of a best guess.

[Michael Kaminsky]: What I would describe is what we want to do as scientists, I'll just say as scientists in general, is we want to combine all of the information that we have about some phenomenon out in the world, and then combine that information in the most rigorous way possible, [Michael Kaminsky]: and then call that our best guess, right?

[Michael Kaminsky]: So again, there's a scientific process here that I think is really important.

[Michael Kaminsky]: And again, I'm talking about like scientists, right?

[Michael Kaminsky]: And this includes biologists, this includes people who are estimating populations of fish in the ocean, like there's all kinds of different people who might take this approach.

[Michael Kaminsky]: And what we're often doing in the process of doing science is trying to collect evidence from out in the world and then combine it into some theory of being able to explain how the world really works.

[Michael Kaminsky]: And I think the Bayesian process of sort of building a theoretical model and then comparing that with all of the different evidence that we have and bringing that all together to have some best guests or some understanding of the world, that is a good description of the Bayesian approach to science, really.

[Michael Kaminsky]: So I think there is a part where it's a best guess, but it's not just a best guess, right?

[Michael Kaminsky]: It's a strategy and really like a really principled approach to combining information that yields [Michael Kaminsky]: the best guess based on all of that information that we have.

[Tim Wilson]: So when the contrast between frequentist and Bayesian, and I feel like you started to hit a couple of things, and I will throw in a plug because I thought my first run at trying to understand Bayesian was reading a book that was called The Theory That Would Not Die, how Bayes rule [Tim Wilson]: Bayes rule cracked the enigma code, hunted down Russian submarines and emerged tram from two centuries of controversy.

[Tim Wilson]: And what you said was kind of like the one main thing I took away from it was this is not new.

[Tim Wilson]: And it's called Bayes.

[Tim Wilson]: Like it's named after Bayes Theorem from Thomas Bayes from however long ago.

[Tim Wilson]: But you started to say that when what Fisher came up with, [Tim Wilson]: that is kind of, I guess, sort of the frequentist world, is it that it's kind of computationally simpler but relies on, for frequentist statistics to work, the shortcuts or the less computationally intensive aspects of that approach do rely on some underlying [Tim Wilson]: assumptions being true that in the past, if you're going that, when you go that route, you say, well, I don't have a strong enough reason to think they're not true.

[Tim Wilson]: So I'm going to accept them as true and then kind of calculate.

[Tim Wilson]: I mean, I think it is worth talking about what the frequentist world misses.

[Tim Wilson]: And part of what you were leaning towards was some technological or computationally, one is more computationally intensive than the other.

[Tim Wilson]: And I know it didn't end with a question, but that's because I'm kind of confused.

[Michael Kaminsky]: Let me try to restate some of this, and then I'll expand on it a little bit.

[Michael Kaminsky]: So Ari Fisher developed a lot of these really powerful statistical methods to solve some of the problems that were floating around at the time that he was working again at early 1900s.

[Michael Kaminsky]: And a lot of the problems that people were facing statistically were in the field of agriculture.

[Michael Kaminsky]: And so it was either like, hey, we have this field full of, I don't know, watermelons, and we treated each of these different plots differently.

[Michael Kaminsky]: And we want to be able to take a sample of these watermelons and then try to determine if plot A versus plot B versus plot C, which one is yielding the bigger melons.

[Michael Kaminsky]: That is the type of analysis they were doing.

[Michael Kaminsky]: Or, classically, you get a shipment of grain to the Guinness factory.

[Michael Kaminsky]: You want to be able to take a sample of that grain and then use that to guess about the overall quality of the full shipment, obviously, without having to analyze every grain of whatever wheat that is coming in that shipment.

[Michael Kaminsky]: And so they developed a lot of techniques for analyzing from small samples and then generalizing to a population.

[Michael Kaminsky]: And the frequentist sort of body of work often or pretty much always takes that as the philosophical basis and even the mathematical basis of what they're trying to do is they're saying, okay, there's this population that is very big that we can't measure all of.

[Michael Kaminsky]: And so we're going to take samples from that, and then we're going to try to generalize from those samples to the population.

[Michael Kaminsky]: And [Michael Kaminsky]: The frequentist statistics work really, really well when you are in that scenario.

[Michael Kaminsky]: When you are taking random samples from a large population, frequentist statistics work very, very well.

[Michael Kaminsky]: For doing, I mean, and again, I use frequentist statistics a lot.

[Michael Kaminsky]: I've published papers using frequentist statistics.

[Michael Kaminsky]: I am not a zealot for Bayes in any sense.

[Michael Kaminsky]: These methods work especially well in this context, right?

[Michael Kaminsky]: A, B testing.

[Michael Kaminsky]: you're testing feature A versus feature B, you're going to randomly draw from your sample of visitors to your website.

[Michael Kaminsky]: Some are going to get A, some are going to be perfectly aligned with the frequentist approach and the frequentist model.

[Michael Kaminsky]: Those statistics, that works really well.

[Michael Kaminsky]: There's a lot of baked-in assumptions there, right?

[Michael Kaminsky]: You're getting a random sample.

[Michael Kaminsky]: There is this population.

[Michael Kaminsky]: All of those questions tend to line up really nicely.

[Michael Kaminsky]: But there's a lot of other questions that we might ask that don't really fit naturally into that framework.

[Michael Kaminsky]: where we aren't necessarily drawing samples.

[Michael Kaminsky]: Maybe we actually observe all of the data.

[Michael Kaminsky]: There is no sample.

[Michael Kaminsky]: It's just like, we have all of the data.

[Michael Kaminsky]: I work a lot in marketing analytics.

[Michael Kaminsky]: We have all of the data on marketing that we have.

[Michael Kaminsky]: There's no sample.

[Michael Kaminsky]: There's no other universe.

[Michael Kaminsky]: Frequent statistics start to get weird, and you often run into this, people like, okay, help me interpret this p-value.

[Michael Kaminsky]: If you ran this analysis a thousand times, what would the distribution of results be?

[Michael Kaminsky]: You're like, wait, but I have the code for the analysis, the data doesn't change because I have the universe, what do you mean run it a thousand times?

[Michael Kaminsky]: Like, it doesn't really make sense.

[Michael Kaminsky]: And so there's lots of questions that we might ask as a business that don't really naturally fall into this idea of drawing a sample from a population.

[Michael Kaminsky]: Maybe we want to predict how much revenue this business is going to do next quarter.

[Michael Kaminsky]: Like, what's the population and what's the sample there that we would be analyzing?

[Michael Kaminsky]: And all of a sudden, it's like, OK, well, what are we actually trying to do here?

[Tim Wilson]: Can I ask, I remember my brain melting a little bit when that whole sample versus population, specifically when it comes to time, when you can only sample the past to the present, if you're determining your population is all time or the next year, isn't that like, definitionally, you don't get a representative sample, or it's not a, you're getting a sample that's from a [Tim Wilson]: one set of times, you can't go and pull some of your sample data from the future, right?

[Moe Kiss]: So wait, sorry.

[Moe Kiss]: The sample bit is definitely something that is turning ground in my mind, too.

[Moe Kiss]: So have I got it correct that in a frequentist world, you basically would need to [Moe Kiss]: be confident that you have a fully representative sample to then make a decision about the action you're gonna change versus like, because there's a very high likelihood that you have a shitty sample, right?

[Michael Kaminsky]: And so is that part of the- A lot of the Frequentist math is about determining what sample size do you need to believe that it is representative, right?

[Michael Kaminsky]: That's what a lot of like the machinery behind the statistics is actually sort of doing for you.

[Michael Kaminsky]: The p-value has baked into it this idea of, well, how big is the sample?

[Michael Kaminsky]: There's an n minus 1 divisor somewhere in the formula, and that's controlling for, well, how big do we think the population is?

[Michael Kaminsky]: How big is the sample size?

[Michael Kaminsky]: It's trying to do some of that machinery for you.

[Michael Kaminsky]: But you're right that that does matter.

[Michael Kaminsky]: One of often the core, a core assumption of a lot of frequentist methods is that you have this random sample from the true population.

[Michael Kaminsky]: Often, it is the case that we don't have that.

[Michael Kaminsky]: And then again, the sort of frequentist paradigm, again, people who are like arch-frequentists or have PhD statistics are going to quibble with this.

[Michael Kaminsky]: But often, I feel like the frequentist paradigm starts to really break down or get very confusing once you get into this world where it's like, okay, we have a non-random sample, now what do we do with it?

[Michael Kaminsky]: And that's a place where, all of a sudden, it's like, oh, all of these tools that we have no longer really apply, all of the frequency tools we have no longer really apply, but it is a place where you can start doing Bayesian modeling very well.

[Michael Kaminsky]: You need to account for that non-randomness.

[Michael Kaminsky]: You need to have some story that you can program into this simulated world about how your sample came to be non-randomly.

[Michael Kaminsky]: But you can still make forward progress there very easily.

[Michael Kaminsky]: You just simulate what would happen.

[Michael Kaminsky]: Here's what it would look like, here's the implications of that, and you can go on your merry way.

[Michael Kaminsky]: That I think is one of the really powerful tools of the Bayesian approach is that you can start just from what is my model of the world and then work from there.

[Michael Kaminsky]: You don't have to be like, okay, do I have a random sample from a population and then what do I do with that?

[Michael Kaminsky]: It opens up the aperture on the types of analyses you can do and the types of questions that you can answer.

[Tim Wilson]: So can you maybe come up with, I don't know, like a baseball analogy to explain?

[Tim Wilson]: So this is such a good example if you follow baseball and sorry.

[Tim Wilson]: Yeah.

[Michael Kaminsky]: So Tim is taking this up because I sent over some notes via too many notes via email.

[Michael Kaminsky]: And this was one of the examples.

[Michael Kaminsky]: This was one of the examples that resonated with me very early on in my journey.

[Michael Kaminsky]: Actually, I'll give you all some insight into bibliography.

[Michael Kaminsky]: When I first heard about Bayesian statistics, I was incredibly skeptical.

[Michael Kaminsky]: I was like, [Michael Kaminsky]: What is this prior stuff?

[Michael Kaminsky]: This all sounds like BS.

[Michael Kaminsky]: It sounds like you're just putting your thumb on the scale.

[Michael Kaminsky]: I trust my statistical significance.

[Michael Kaminsky]: I don't want to hear about all your priors.

[Michael Kaminsky]: And so it went on a long journey to start to really understand and appreciate the Bayesian approach.

[Michael Kaminsky]: And this example that someone shared with me, I don't remember the original source, really resonated with me.

[Michael Kaminsky]: And so the story is that imagine that you're at a baseball game.

[Michael Kaminsky]: It's called Opening Day.

[Michael Kaminsky]: So again, apologies to everyone who doesn't follow baseball.

[Michael Kaminsky]: Opening Day is like the first game of the season.

[Michael Kaminsky]: And you watch some batter come up to bat.

[Michael Kaminsky]: Batters generally bat about four times in one game.

[Michael Kaminsky]: And you see this batter get a hit four out of four times.

[Michael Kaminsky]: So they are batting on average of 1,000 on this opening game.

[Michael Kaminsky]: And then I say to you, I say, okay, well, what do you think, Tam knows baseball?

[Michael Kaminsky]: So, Tam, what do you think this batter's average is going to be at the end of the season, given that we just saw this batter go four for four, batting a thousand with this at opening day?

[Michael Helbling]: So first I'm sorry, is this MLB or Savannah Bananas?

[Tim Wilson]: Let's call it, let's say MLB for now.

[Michael Helbling]: Yeah, okay, MLB, okay.

[Tim Wilson]: Well, now, so here's, I don't even know whether I'm winding up giving a intuitive or a Bayesian or a frequentist answer, but I would say, well, that sample size is so small, I'm gonna say that I think it's probably 300 or 325.

[Tim Wilson]: Okay, 300, how'd you get to 300?

[Tim Wilson]: Because 300 would be a good solid respectable from this early, [Tim Wilson]: Viewing, it was way better than that, but it's only four at bats, and it's a long season, and there will be some regression to...

Well, yeah, but why 300?

[Tim Wilson]: Where did 300 come from?

[Tim Wilson]: Why does that not jump to your mind?

[Tim Wilson]: To me, not being a super, super close follower of baseball, that's a number that I know is kind of normal.

[Tim Wilson]: Maybe 250 or 300 is standard.

[Tim Wilson]: It's the average.

[Tim Wilson]: Yeah.

[Michael Kaminsky]: So just to talk through this logic, the idea here is we just saw this batter do really well.

[Michael Kaminsky]: Four hits in a game is really, really good.

[Michael Kaminsky]: So we think this batter is probably a good batter.

[Michael Kaminsky]: But we've only seen one game, and we know, based on our other knowledge of baseball, that [Michael Kaminsky]: Batting 300 for a season or for a career is really, really good.

[Michael Kaminsky]: That's Hall of Fame.

[Michael Kaminsky]: So 250 in general, almost every batter is going to end the season between 200 and 350 as an average overall.

[Michael Kaminsky]: This is 0.2 to 0.35.

[Michael Kaminsky]: Again, apologies to those who don't follow baseball.

[Michael Kaminsky]: So what Tim did here is Tim used his Bayesian reasoning to get this idea.

[Michael Kaminsky]: We have a little bit of data, and we want to combine that with our prior knowledge [Michael Kaminsky]: of how baseball players normally perform.

[Michael Kaminsky]: We know that pretty much everybody's gonna land between 200 and 250 in the season.

[Michael Kaminsky]: This player seems pretty good, so maybe at the top end of that range, 250, 300, maybe a little bit better, but that's probably where he's gonna end up.

[Michael Kaminsky]: That reasoning is a Bayesian approach.

[Michael Kaminsky]: We have prior information about in general where people land, we observe a little bit of data, we combine that together in our mind, and we come up with some range that we think is reasonable based on that combination of evidence that we have.

[Michael Kaminsky]: The Bayesian approach to statistics is very intuitive for humans.

[Michael Kaminsky]: Humans do Bayesian reasoning constantly.

[Michael Kaminsky]: You have prior expectations about what's going to happen, and then you observe evidence, and then you update that in your mind.

[Michael Kaminsky]: And your mind is constantly doing this all of the time.

[Michael Kaminsky]: It's not necessarily applying Bayes theorem exactly, but it's doing this process of combining evidence together to come to some conclusion.

[Michael Kaminsky]: And then the apparatus of Bayesian statistics that we have gives us a very formal, very precise way of doing that reasonably efficiently.

[Moe Kiss]: Sorry, go ahead.

[Moe Kiss]: I've always sort of boiled things down to a TLDR, aren't I?

[Moe Kiss]: It's updating your beliefs, right?

[Moe Kiss]: That is the most simple way I try and rationalize this.

[Michael Kaminsky]: That's exactly right.

[Moe Kiss]: coming to the business, which I feel like I'm always trying to do.

[Moe Kiss]: So you're working with your stakeholders.

[Moe Kiss]: You feel like you've done a really good job of being like, this is the way that I have come up, like the assumptions that I have made.

[Moe Kiss]: These are the beliefs that I have.

[Moe Kiss]: And then over time, they adjust.

[Moe Kiss]: What does that process look like with your business stakeholders?

[Moe Kiss]: Because I presume, so we talked about this a little bit earlier about the sample, but the quality of the assumptions you make also are based on your knowledge of that particular area.

[Moe Kiss]: I know shit about baseball.

[Moe Kiss]: So listening to that, I was like, I'd probably make, my assumptions would be pretty shit versus someone who, I don't know, works in data in baseball.

[Moe Kiss]: There's would be much better.

[Moe Kiss]: The same would be true if you're working with a business stakeholder on a problem.

[Moe Kiss]: They might have business expertise that could help make better estimates to start.

[Moe Kiss]: How involved or not involved do you think the business stakeholders should be if you're using a Bayesian approach to solving a problem?

[Michael Kaminsky]: I think ideally very involved.

[Michael Kaminsky]: Again, I talk about how do we do science?

[Michael Kaminsky]: If you're a scientist, you should start from all of the theory.

[Michael Kaminsky]: You're going to go read all of the papers.

[Michael Kaminsky]: You're going to go learn deeply about how other people have studied this particular phenomenon.

[Michael Kaminsky]: And that is going to give you the best starting point for learning something new about the world.

[Michael Kaminsky]: And in business, we want to do the same thing.

[Michael Kaminsky]: As analysts, as scientists, we want to start by talking to the domain experts.

[Michael Kaminsky]: We want to learn about how do you think about how this system works?

[Michael Kaminsky]: What are your expectations?

[Michael Kaminsky]: How do you know that?

[Michael Kaminsky]: And then once they share that, you can use that information to, again, sometimes you can't approve on it.

[Michael Kaminsky]: You're like, you clearly know what you're talking about.

[Michael Kaminsky]: I can't help you.

[Michael Kaminsky]: You're like, good luck.

[Michael Kaminsky]: But sometimes we can take that information and combine it with other data in smart ways to generate new insights.

[Michael Kaminsky]: But I would say, [Michael Kaminsky]: As a scientist, again, we want to be talking to the people who actually know how the system works.

[Michael Kaminsky]: That's what's going to allow us to build better structural models, this idea of a simulation, and it's going to allow us to better incorporate their knowledge explicitly and directly into this model.

[Michael Kaminsky]: If this person says, look, there's no way that this effect that you're talking about is larger than X, [Michael Kaminsky]: Well, great.

[Michael Kaminsky]: I can put a bound on my model and that really helps us to get more precision on the other estimates if we know that it can't be x.

It might save us a bunch of time.

[Michael Kaminsky]: Again, this Bayesian process, even for quote-unquote frequentists, often happens very naturally.

[Michael Kaminsky]: Many of you all might have been in this situation where you've been tasked with doing some analysis, maybe you're running a regression to try to get some answer, and you run your regression and you get a number and you look at the number and you're like, that can't possibly be true.

[Michael Kaminsky]: It doesn't make any sense.

[Michael Kaminsky]: You throw it out.

[Michael Kaminsky]: And then you keep tweaking the model, you tweak the model, you tweak the model, you throw a bunch out.

[Michael Kaminsky]: Finally, you get to a result that's like, okay, yeah, this sort of makes sense.

[Michael Kaminsky]: That is a base in process, right?

[Michael Kaminsky]: Like you have priors about what you expect the results are going to be.

[Michael Kaminsky]: You're throwing out a huge number of potential analyses, but you're just doing it based on your judgment.

[Michael Kaminsky]: You're actually sort of doing this in like a sort of quote unquote bad way, right?

[Michael Kaminsky]: A frequentist looks at that process and is gonna say like, ooh, no, that p-value you have, [Michael Kaminsky]: doesn't mean what you think it means, because there's actually like a ton of other comparisons that you just did.

[Michael Kaminsky]: And so that process, which again, all analysts do, because we look at the results and we think about them, we say like, does this match our expectations about what these results should look like?

[Michael Kaminsky]: You throw out other results that don't, and you're only left with the ones that are.

[Michael Kaminsky]: The Bayesian system is just a more formalized, honestly more rigorous approach to that same process.

[Michael Helbling]: It seems like to be a good poker player, you need to be a Bayesian.

[Michael Kaminsky]: Poker players tend to be good patients.

[Michael Kaminsky]: They tend to think this way in terms of, because again, you think about like a poker hand, right?

[Michael Kaminsky]: Or again, what is the popular type of poker that people play right now?

[Michael Kaminsky]: Texas Hold'em.

[Michael Kaminsky]: Texas Hold'em, right, Texas Hold'em.

[Michael Kaminsky]: You have some information that you have about what cards you have.

[Michael Kaminsky]: You see, you know, cards come out one at a time.

[Michael Kaminsky]: And based on as the cards come out, you have to sort of update your belief about, is my hand a winner or not?

[Michael Kaminsky]: Or is my opponent's hand a winner or not?

[Michael Kaminsky]: It's again, it's this process of updating our beliefs about what is likely to happen in future states on the world as new information comes in.

[Michael Kaminsky]: And that, again, is a very Bayesian approach to just thinking about what's gonna happen in the world.

[Michael Helbling]: But there's a big difference in priors between a professional and a, you know, that's exactly right.

[Michael Kaminsky]: Professionals will have a lot, they'll have a much better model and a lot more confidence about, they'll just know the numbers a lot better.

[Michael Kaminsky]: It's actually like, that's actually a less, maybe not as good of an example on priors because they're sort of like a mathematically correct answer in the case of poker.

[Michael Helbling]: There's a bunch of probabilities they're doing as well.

[Michael Kaminsky]: Yeah.

[Michael Kaminsky]: But like the sports analogy is maybe a good one or like a really good scout or like a recruiter for some sport.

[Michael Kaminsky]: And again, this applies to soccer or football or whatever, Aussie rugby or something.

[Michael Kaminsky]: Yeah.

[Michael Kaminsky]: That like those scouts might have much better.

[Michael Kaminsky]: expectations on how well this player is going to perform because they've seen so many others, they've studied it so much, they're going to have a much better prior on the future career trajectory of some player than I will because I don't know as much about that sport and I'm not a trained scout.

[Michael Kaminsky]: And so experts can have a lot better guesses about future outcomes and a lot stronger beliefs because of all of that expertise that they have.

[Tim Wilson]: So when you say [Tim Wilson]: So I think there's the priors in taking and updating your beliefs, but you also keep saying simulations.

[Tim Wilson]: And I keep thinking, okay, if I'm gonna go back to the baseball example, if I have knowledge that, look, between 200 and 350, that's gonna be where batters wind up being.

[Tim Wilson]: I'm gonna run, I'm gonna build a model where I have batters and take [Tim Wilson]: four at bats and see how many of those, as I range them with their overall batting average between 250 and see how often I come up with four hits in a row.

[Tim Wilson]: Is there literally simulations like you're picking?

[Tim Wilson]: That is literally how I would do this.

[Michael Kaminsky]: What you just described is literally how I would solve this problem.

[Michael Kaminsky]: I am not very good at traditional equations-based math.

[Michael Kaminsky]: There probably is a shortcut to coming up with this answer that someone who is really good at math and probability could write down.

[Michael Kaminsky]: I would just simulate it.

[Michael Kaminsky]: I do exactly what you said.

[Michael Kaminsky]: I have a population of hitters that would have a range of overall batting averages, and then I'd run the simulation and see how often do we observe [Michael Kaminsky]: a person getting four hits in one game and then see how that relates to the range of batting averages they end up with.

[Michael Kaminsky]: It is an analytical way to generate an answer to this question.

[Michael Kaminsky]: That is, in some ways, very computationally intensive.

[Michael Kaminsky]: Running that simulation, it's going to be a pain.

[Michael Kaminsky]: It's going to take a few hours.

[Michael Kaminsky]: It's going to get coded up.

[Michael Kaminsky]: It's going to take a few hours to run the computer.

[Michael Kaminsky]: But it will give you the right answer every single time.

[Michael Kaminsky]: And you don't really need to know any statistics to get the right answer.

[Michael Kaminsky]: You just simulate.

[Michael Kaminsky]: And that is, go ahead, Moee.

[Michael Kaminsky]: No, oh God, I was just gonna say, that is what I like about Bayesian statistics is that you don't need to memorize all of the different rules, right?

[Michael Kaminsky]: You don't need to memorize, oh, what are the assumptions of the linear regression and oh, am I gonna run this check and that check and this check to make sure it's just like, we're just gonna simulate.

[Michael Kaminsky]: We're just gonna simulate from what we believe based on the structure of the problem that we're analyzing.

[Michael Kaminsky]: We're gonna compare it to the data that we observed and we're gonna count.

[Michael Kaminsky]: And so it's a simulation and it's a counting problem.

[Michael Kaminsky]: And again, this goes back to why is this becoming more popular now?

[Michael Kaminsky]: It's becoming more popular now because we have more powerful computers that can do more bigger simulations faster.

[Michael Kaminsky]: If you wanted to answer one of these fairly complex problems 20 years ago, you just couldn't do enough simulations.

[Michael Kaminsky]: It would take way too long.

[Michael Kaminsky]: The computers weren't powerful enough.

[Michael Kaminsky]: The algorithms for doing these simulations efficiently weren't powerful enough.

[Michael Kaminsky]: Now, they're getting a lot more powerful in terms of computers and in terms of the algorithms that we can do to explore different parameter spaces, and that allows us to answer these problems.

[Michael Kaminsky]: Again, where we don't have to learn any rules, we just simulate and then count.

[Michael Kaminsky]: We put way more effort on the computer and way less effort on the theory and the assumptions that go into these shortcuts that can give you really nice answers in the frequentist paradigm, but aren't often as flexible as just the straight simulation approach.

[Moe Kiss]: I want to talk about the Enigma machine because it's something that I'm mildly fascinated about.

[Moe Kiss]: Personally, I'm going to be very open that I'm making more of an effort to learn more about stats.

[Moe Kiss]: This is something that I want to do over the next period of time.

[Moe Kiss]: So I can attempt to explain it, and we'll probably fuck it up.

[Moe Kiss]: Or if someone else feels more confident in trying to explain it, I am also happy with that.

[Moe Kiss]: But this is me live learning.

[Moe Kiss]: So you guys let me know whether we're going to go with option A or option B.

I say take it away, Mao.

[Michael Helbling]: Yeah, go for it, Mao.

[Michael Helbling]: I'm ready.

[Michael Helbling]: Because I'm pretty sure I couldn't, so.

[Moe Kiss]: Enigma Machine used in World War II by the Germans, they thought it was unbreakable because there was basically so many different variations.

[Moe Kiss]: Of course, very famously, Bletchley Park, they broke the code.

[Moe Kiss]: There were some great men involved, also many, many, many amazing women, which is why I know lots about it.

[Moe Kiss]: Like I said, I'm going to do my best to try and explain it.

[Moe Kiss]: My understanding is that the machine A could never equal A in the machine, right?

[Moe Kiss]: So like, you know, A can't equal A, B can't equal B, et cetera.

[Moe Kiss]: So you have like a series of beliefs, rules that like is your starting point.

[Moe Kiss]: And then they had [Moe Kiss]: I don't remember exactly the cribs, but basically they would go in and they would make a guess using what was like a common word from the message.

[Moe Kiss]: So like deer or like kernel or like I'm trying to think or if it was a rain, a weather report, it might be like rain.

[Moe Kiss]: And so they know what the output is they're looking for.

[Moe Kiss]: So they would try and make like a best guess of, I think that if I tried to put be here and we're trying to spell the letter rain, then the next [Moe Kiss]: letter, which is A, should be this.

[Moe Kiss]: And it was basically them testing out a series of these different things.

[Moe Kiss]: And if they got one letter, then they would go down that, I think they called them cribs.

[Moe Kiss]: And that was basically how they ended up breaking the code.

[Moe Kiss]: But they would have to do that again every single day because they change the codes every day.

[Moe Kiss]: This is one of those things that they constantly come back to.

[Moe Kiss]: This is an example of Bayesian [Moe Kiss]: a Bayesian approach.

[Michael Kaminsky]: So it sounds Bayesian to me, right?

[Michael Kaminsky]: Because basically, so let me again, try to just re-describe in my words, what you're describing.

[Michael Kaminsky]: I am not an expert on this, but like the intuition behind this makes sense is we're going to.

[Michael Kaminsky]: So we have some model of how like I'm we're English, right?

[Michael Kaminsky]: We have some model of how German works of the frequency of different words, the frequency of different letter pairings, certain common words, certain common patterns that we're going to look for.

[Michael Kaminsky]: And then what we're going to do is we're going to throw a ton of computation at the problem.

[Michael Kaminsky]: And we're just going to say, look, we're going to run all of the different combinations as fast as we can.

[Michael Kaminsky]: And we're going to see, based on the expected distribution of certain, like again, letter combinations, under a bunch of different assumptions, these are the parameters in the model, right?

[Michael Kaminsky]: How often do these expected combinations come up under these different parameters?

[Michael Kaminsky]: And that lets us back into the idea of, OK, well, [Michael Kaminsky]: the patterns workouts is that this is probably the cipher.

[Michael Kaminsky]: Again, I don't even actually really know exactly how you would say, oh, this is Bayes' theorem applied here, but this idea of we're just going to look for patterns and we're going to run all of the compute and count the things up and that's going to give us some indicator of what's most likely.

[Michael Kaminsky]: That again, it's a very Bayesian feel to the type of analysis that we're talking about.

[Moe Kiss]: Okay, so everyone has homework, but my understanding is they did start with people doing this and then the enigma machine ended up taking over, yeah.

[Tim Wilson]: I mean, that's in the title of the book.

[Tim Wilson]: I don't remember the details of the one that I was that somehow Bayes ruled to help crack the enigma code, but you gave them, I don't remember how they said that that worked.

[Michael Kaminsky]: There's probably something to it where it's like, because like the total number of combinations is too big.

[Michael Kaminsky]: And so you have to smartly guess what is the next combination that you're going to check.

[Michael Kaminsky]: Like what is most likely based on the information on like the things that we have done so far?

[Michael Kaminsky]: And so like my guess is that thinking about what is the next sort of combination of cipher wheels?

[Michael Kaminsky]: Are we going to try that is most likely given what the information that we have.

[Michael Kaminsky]: And then based on the results of that, we'll have some new best guess about what the next best guess to try is.

[Michael Kaminsky]: And then we're going to do that.

[Michael Kaminsky]: And that allows you to not have to run all whatever 7 trillion versions there are because you want to have the smart like you can't run all 7 trillion.

[Michael Kaminsky]: And so you say like, okay, we have to do the ones where we have the best guess first.

[Michael Kaminsky]: And the base theorem would give you a principled way to think about what's the next best guess based on everything that we've observed so far.

[Moe Kiss]: Okay.

[Moe Kiss]: So now to go down another complete rabbit hole because that's what I like to do is like, what are all the questions that I have on this topic?

[Moe Kiss]: Let's talk a little bit more about bias, because I feel like, is that where everyone gets into, like, tips about this, where, like, the Bayesians think frequentists have, like, more bias issues when it comes to selection, and then the frequentists think the bias is to do with the beliefs that the Bayesians are making?

[Moe Kiss]: Like, is that the crux of the bias debate, or is there, I feel like you're going to explain it in a much smarter way than I probably have?

[Michael Kaminsky]: I'll try to jump in with at least what I have seen.

[Michael Kaminsky]: And this is also part of my personal journey.

[Michael Kaminsky]: I talked about I had this similar belief about Bayjens when I was first learning about it, where I was like, oh, you can just use priors to make the model say whatever you want.

[Michael Kaminsky]: And I was like, that feels wrong.

[Michael Kaminsky]: That's not science.

[Michael Kaminsky]: You're just using your priors to make the model say whatever you want.

[Michael Kaminsky]: And what I learned is that Bayjens, the response to that is they say yes, that is true.

[Michael Kaminsky]: I can use my priors to make the model say anything.

[Michael Kaminsky]: just the same way that you can run the regression a thousand times and just selected the version that you want to use.

[Michael Kaminsky]: The problem is that there is no way that we can design an analysis procedure that will prevent an analyst from shaping the results.

[Michael Kaminsky]: The analyst has the power to shape the results always.

[Michael Kaminsky]: There is no procedure that eliminates the possibility of analyst bias.

[Michael Kaminsky]: If we think that we are working with an analyst who is going to try to trick us, [Michael Kaminsky]: There's basically nothing we can do.

[Michael Kaminsky]: The p-value is not going to prevent us from the analyst tricking us.

[Michael Kaminsky]: Bayesian approach, not going to prevent us from the analyst tricking us.

[Michael Kaminsky]: There is no way to get around that.

[Michael Kaminsky]: The analyst can always bias the results.

[Michael Kaminsky]: And again, those of you who are data scientists who have worked in academia or an industry, you probably feel this in your bones.

[Michael Kaminsky]: You have run analysis different ways, and you just know that I can always run an analysis and make the data say anything I want.

[Michael Kaminsky]: It's always possible.

[Michael Kaminsky]: I'm very good at it.

[Michael Kaminsky]: I can do it.

[Michael Kaminsky]: So that, again, there is no procedure that will prevent that.

[Michael Kaminsky]: And so if you're in a world where you're like, we have to figure out the procedure that's going to prevent that, throw it out, never going to happen.

[Michael Kaminsky]: Once we're in a world where we're like, okay, we're interested in finding the truth together, then we want to start thinking about, well, what's the best way to do that?

[Michael Kaminsky]: It's totally correct that, yes, an analyst could bias the results with the priors, but a part of analyzing a Bayesian analysis is also looking at the priors.

[Michael Kaminsky]: The priors shouldn't ever be hidden.

[Michael Kaminsky]: When I am reading the results of a scientific paper, whether it's using Bayesian analysis or frequentist approach, what I'm going to be looking for is, well, what are the assumptions that's being made?

[Michael Kaminsky]: Exactly what is the structure of the model being used here?

[Michael Kaminsky]: How is that combining with the data that they have and generating these results?

[Michael Kaminsky]: The priors should always be a part of that.

[Michael Kaminsky]: If you read an academic paper where people are taking a Bayesian approach, they're always going to share their priors.

[Michael Kaminsky]: That is just part of understanding a Bayesian analysis.

[Michael Kaminsky]: What I tell people is that an analysis is an argument.

[Michael Kaminsky]: It's not truth.

[Michael Kaminsky]: An argument is up to the reader or the person hearing it to make judgments.

[Michael Kaminsky]: Is it a good argument or is it a bad argument?

[Michael Kaminsky]: If I see someone doing an analysis and they're taking a patient approach and I don't agree with their priors, I'm like, those priors don't make any sense.

[Michael Kaminsky]: It's going to be a very not very compelling argument.

[Michael Kaminsky]: I'm going to say that I'm not going to buy your results.

[Michael Kaminsky]: The results aren't going to be convincing to me because I don't think your priors make any sense.

[Michael Kaminsky]: But good analysts or good Bayesians, they're going to try to justify their priors.

[Michael Kaminsky]: They're going to say, look, these are the assumptions that I'm making.

[Michael Kaminsky]: Here's what it's based on.

[Michael Kaminsky]: It's based on these other experiments and these other papers and this theory.

[Michael Kaminsky]: Combine that with the data that we have.

[Michael Kaminsky]: It's going to be limited in some way.

[Michael Kaminsky]: And this is the implication on the other side once you go through that process.

[Michael Kaminsky]: And maybe that's very compelling.

[Michael Kaminsky]: Maybe it's not, depending on exactly what assumptions are made.

[Michael Kaminsky]: But it's the totality taken together that makes it a compelling argument or not.

[Michael Kaminsky]: Not does it have two or three asterisks in table number two.

[Moe Kiss]: What I don't get is can't frequentists just solve this by also being more open about potential misinterpretations or reasons that I don't know, they might have run the same test 20 times, but only taken one result.

[Moe Kiss]: The fact that they took the results early.

[Moe Kiss]: Any of the ways that you're typically p-hacking, who did they just have the same thing where they're like, these are the explicit [Moe Kiss]: ways we could have made an error in this analysis, and then wouldn't that just be like, great, now both methods are both great, and we use one depending on which one's better for the problem we're trying to solve.

[Michael Kaminsky]: I think the really good scientists do that, and that's what we see.

[Michael Kaminsky]: The good scientists do exactly that.

[Michael Kaminsky]: And it's great.

[Michael Kaminsky]: I love that.

[Michael Kaminsky]: Again, I am not a zealot on one side or the other.

[Michael Kaminsky]: I think the good scientists do that.

[Michael Kaminsky]: They admit where there is risk, where certain assumptions are potentially biasing the results.

[Michael Kaminsky]: Really good scientists, again, both frequentist and Bayesian analyses, they'll run sensitivities.

[Michael Kaminsky]: And they'll say, if this assumption that I made was different, if the priors were different, how much does that affect the results?

[Michael Kaminsky]: If this assumption is different, how much does that affect the results?

[Michael Kaminsky]: That's the sort of stuff that you really want to see to understand.

[Michael Kaminsky]: And that's what makes for a very compelling argument.

[Michael Kaminsky]: The problem comes when people think that a p-value of less than 0.05 means that something is truth.

[Michael Kaminsky]: And that's, again, I think we all, there's growing awareness that that's not true.

[Michael Kaminsky]: People listening to this podcast probably know that that's not true.

[Michael Kaminsky]: So there's growing awareness of that.

[Michael Kaminsky]: But if we're taking sort of like an honest approach to science, we need to recognize that all of these different assumptions that we make are impacting the final results.

[Michael Kaminsky]: We need to be transparent about what those are.

[Michael Kaminsky]: We need to, in the interest of, again, getting towards truth, the scientific process, explore the implications of those different assumptions.

[Michael Kaminsky]: And if we do that, we're going to end up closer to the truth.

[Michael Kaminsky]: And again, now all of a sudden, there's not such a difference between this Bayesian approach and the Frequentist approach, right?

[Michael Kaminsky]: If we're doing this all very transparently, we're going to end up probably in a very similar world.

[Tim Wilson]: I think that's, as you were saying that, I remember Matt Gershoff saying that at some point.

[Tim Wilson]: I want to say Chelsea Parlatel-Riedi said it as well.

[Tim Wilson]: The people who are genuinely knowledgeable, if you have [Tim Wilson]: enough data and you put a similar amount of rigor on it and you have that you'll, you're going to wind up coming up with the same answer.

[Tim Wilson]: You don't hear people saying, well, the Bayesian says that the truth is X and the rigorous Franckletist says that the truth is Y.

One, neither one of them is declaring a truth.

[Tim Wilson]: And the other is they should probably be getting at results that even if what they're saying is a little bit different, they, [Tim Wilson]: Neither one would contradict the other if they're both done rigorously and well, and maybe I'll add on to that.

[Tim Wilson]: Frequencies have p-values and confidence intervals, but Bayesian have credible intervals.

[Tim Wilson]: Does that tie in that if you've only got a credible interval, if you've done this Bayesian approach, you've got a confidence interval, you've done this other approach, [Tim Wilson]: They're not the same thing, but they're both kind of pointing to the level of uncertainty.

[Tim Wilson]: And if you're doing a rigorous job, you're going to be indicating an appropriate breadth of uncertainty in whatever conclusion you've arrived at.

[Tim Wilson]: Yeah, so I think that that's great.

[Michael Kaminsky]: It's mostly fair.

[Michael Kaminsky]: The things that I would add on to it is that almost always you can get to the same frequentist results with a Bayesian approach.

[Michael Kaminsky]: There's a few exceptions, but almost always you can sort of back in.

[Michael Kaminsky]: And if you give me some frequentist analysis that you've done, I can sort of write those same assumptions down using priors and model structures and get effectively the same results.

[Michael Kaminsky]: The wrinkle is that you can't always go the other way.

[Michael Kaminsky]: There are sort of questions that you can answer with a Bayesian approach that are just really not possible with a frequentist approach because of the limitations of the specific assumptions that are being made and the specific mechanics there.

[Michael Kaminsky]: Again, because the simulation based approach is effectively infinitely flexible, right?

[Michael Kaminsky]: If you can simulate it, you can generate a statistical analysis with it.

[Michael Kaminsky]: And that means that there are more questions that we can answer with a Bayesian approach that can't be answered with a frequentist approach.

[Moe Kiss]: I really want to be a jerk and ask an example.

[Michael Kaminsky]: Great question.

[Michael Kaminsky]: As an example of a type of analysis that I think is very difficult to do in a frequentist analysis, this comes from Richard McElrath.

[Michael Kaminsky]: He's an amazing statistics educator.

[Michael Kaminsky]: A lot of my knowledge and even some of the explanations that I'm sharing on this podcast come from him.

[Michael Kaminsky]: Everybody should check out his books, Statistical Rethinking, and then his online YouTube course.

[Michael Kaminsky]: So he gave this example, which I think is really interesting.

[Michael Kaminsky]: It's technical and sort of in the weeds, so apologies for that.

[Michael Kaminsky]: But it's sort of the best example that I have coming to my mind.

[Michael Kaminsky]: And this is an example of trying to analyze multi-level models.

[Michael Kaminsky]: And so we could imagine, you see this a lot in like, [Michael Kaminsky]: education econometrics, where you have a model, where you have like the school level effects, and then you have classroom level effects, and then you have individual student level effects.

[Michael Kaminsky]: And you're trying to estimate, okay, if we teach phonics versus if we don't teach phonics, like, how does that happen?

[Michael Kaminsky]: And there's often we wanna estimate like all of those different effects, student level, classroom level, school level.

[Michael Kaminsky]: And in frequentist approaches, it can be often very difficult to estimate all of those simultaneously.

[Michael Kaminsky]: Frequentists invented this machinery to estimate these multi-level models called Munlock machines.

[Michael Kaminsky]: They're weird, and again, it requires a bunch of assumptions under the hood, and it gives okay results.

[Michael Kaminsky]: It works.

[Michael Kaminsky]: But the Bayesian approach, we just simulate the whole thing.

[Michael Kaminsky]: We simulate the students, we simulate the classrooms, we simulate the schools, and we can estimate all of those parameters jointly no problem.

[Michael Kaminsky]: zero issue at all.

[Michael Kaminsky]: This is in chapter 12 of statistical rethinking on multi-level models.

[Michael Kaminsky]: Richard McElweth goes into the details of exactly how this works, and we can just show there's no real straightforward way to estimate all of these different parameters simultaneously with a quote unquote, frequentist approach.

[Michael Kaminsky]: We simulate the whole thing, use a lot more computing power to get there.

[Michael Kaminsky]: It takes 30 minutes to run as opposed to 30 seconds, but we get the answer that we want on the other side.

[Michael Kaminsky]: That's one example, and there are a bunch of others of just questions that aren't even really well formulated in a quote-unquote frequentist analysis.

[Michael Kaminsky]: A frequentist would be like, that question doesn't make sense to me, but in a Bayesian approach, we can just write it down.

[Michael Kaminsky]: We can say, okay, look, we're just going to have this model and we're going to simulate and we're going to get a result on the other side.

[Michael Kaminsky]: Another good example that Moee flagged while we were on break is very small data.

[Michael Kaminsky]: So another thing that Professor Richard McKellar likes to say is that the minimum sample size of a Bayesian analysis is zero samples.

[Michael Kaminsky]: You can do a totally reasonable Bayesian analysis with zero samples with no data at all.

[Michael Kaminsky]: We're just going to simulate and we're going to see what happens.

[Michael Kaminsky]: If I have these beliefs, this is what the implications of those beliefs are.

[Michael Kaminsky]: That is a totally reasonable Bayesian analysis.

[Michael Kaminsky]: It's no data at all.

[Michael Kaminsky]: We're just going to simulate from our beliefs from the papers that we've read.

[Michael Kaminsky]: If this is true, if this is true, if this is true, if this is the structural model of how the world works, this is what the implications are.

[Michael Kaminsky]: That's a very reasonable thing to do as a Bayesian.

[Michael Kaminsky]: If you have one data point, you can see how well that aligns with that.

[Michael Kaminsky]: If you have four data points, you can see how well those four data points align with that analysis, and you can move from there.

[Michael Kaminsky]: But the Bayesian approach allows you to really, or actually encourages, which is the thing that I really like.

[Michael Kaminsky]: It encourages you.

[Michael Kaminsky]: to think from the physical system or the biological system or what a Bayesian would call the data generating process.

[Michael Kaminsky]: How does the world work such that it would generate these data and then work from there, as opposed to starting from a dataset and being like, how can I run a regression that will get a statistically significant p-value, which I think is the wrong way to do science.

[Michael Kaminsky]: You want to start from the science, what do we know about the world, compare the theory to the data, as opposed to being like, oh, I found this dataset, how can I get a statistically significant p-value out of it?

[Tim Wilson]: But that does seem like that is a heavy lift when working with the business counterparts.

[Tim Wilson]: I mean, it's the battle I feel like we've been fighting for 20 years and that we have taught the world of business that the data is this [Tim Wilson]: objective quantitative thing and you just have to pick the right model as opposed to saying, no, no, no, you should be thinking about your beliefs, your expectations, your knowledge, what you've seen, what you've theorized.

[Tim Wilson]: You should do all of that first and then see how well the data matches up with that and then [Tim Wilson]: adjusted, right?

[Tim Wilson]: I mean, Moe, it kind of gets back, I think, to your question earlier when you were bringing up, like, working with a business user, all of a sudden, you're going to them saying, I need to tap into your knowledge.

[Tim Wilson]: And they are saying, why?

[Tim Wilson]: Like, my knowledge, I'm just a marketer.

[Tim Wilson]: Why can't you just have the day to give me the answer?

[Tim Wilson]: Which I think, Michael, is where you were just [Tim Wilson]: landing up because that's absolute lunacy if you actually step back and think about it because you're leaving information on the cutting room floor.

[Michael Kaminsky]: I hear both sides.

[Michael Kaminsky]: I think I talk to a lot of business people who complain about their data science team and they're like, why does the data science team never ask me about this section of the business that I know everything about?

[Michael Kaminsky]: They come in with some presentation that has a bunch of results that [Michael Kaminsky]: clearly don't make any sense.

[Michael Kaminsky]: They should have come and talked to me.

[Michael Kaminsky]: I think you get both sides.

[Michael Kaminsky]: I do think that there have been some executives that have been trained into the cult of statistical significance, unfortunately, and they want to turn off their brain and just allow the data to speak to them.

[Michael Kaminsky]: But I actually think that if you look at especially really successful executives in practice, they don't do that.

[Michael Kaminsky]: They actually are really carefully weighing the different pieces of evidence that they have.

[Michael Kaminsky]: If they see a fourth-year analyst come in with some presentation and some analysis, they're like, that doesn't match my expectations at all.

[Michael Kaminsky]: They're going to poke a bunch of holes in it very quickly.

[Michael Kaminsky]: And that again is like the sort of Bayesian process at work here, being like, this analysis doesn't match all of my beliefs.

[Michael Kaminsky]: Like what could be going wrong with it?

[Michael Kaminsky]: And sometimes analysts get frustrated by that and they're like, oh, you know, this executive doesn't listen to the data.

[Michael Kaminsky]: But actually in my experience is that like nine times out of 10, the executives are right.

[Michael Kaminsky]: And it's the analysts that like didn't understand something or did the analysis wrong or missed some crucial assumptions.

[Michael Kaminsky]: And I think that that's actually like a reasonable process.

[Moe Kiss]: Can I walk through a real situation that is bubbling in my mind?

[Moe Kiss]: Okay.

[Moe Kiss]: So you have a new, a product team.

[Moe Kiss]: They want to build a new feature.

[Moe Kiss]: We have all been like auto trained.

[Tim Wilson]: Feel free to use specific names if you want.

[Moe Kiss]: We're not recording this, Moe.

[Moe Kiss]: Go ahead.

[Moe Kiss]: The best way to do that is always through an AB test because that's the gold standard and we want to have as much confidence as possible in the new feature yada yada yada.

[Moe Kiss]: What I'm picking up from this conversation is that you could potentially, even if you haven't built or yet rolled out the feature, you could use a Bayesian approach to have a better understanding to how your users might respond to it.

[Moe Kiss]: That might help you then decide whether you should do feature A or feature B or feature C.

[Moe Kiss]: I feel like working with product folk, they still come back to the like, we need an experiment, it has to be done through an experiment.

[Moe Kiss]: And the thing that I'm kind of reconciling or like rolling around in my head right now is you could get to a decision a lot faster with a different approach.

[Moe Kiss]: That might be good enough, but then is the like, let's say we don't have any data because the feature hasn't been built.

[Moe Kiss]: All your expectations about that feature are going to be based on previous features and how they performed.

[Moe Kiss]: Are you not just going to potentially then end up with a very average assumption of how it might perform, which is potentially not reflective of a new feature?

[Michael Kaminsky]: Okay, so I think this is a really interesting topic and there's a bunch of different directions that we can go.

[Michael Kaminsky]: And I would say that like everything you're describing sounds very reasonable.

[Michael Kaminsky]: And what I would say is that you should design your decision-making approach based on what you're trying to achieve as a business.

[Michael Kaminsky]: And this again is another place where I think Bayjens do really well.

[Michael Kaminsky]: Bayjens care a lot about decision-making, right?

[Michael Kaminsky]: How do we make good decisions and how do we weigh the trade-offs of different decisions?

[Michael Kaminsky]: And this again is one of their obsessed with like uncertainty.

[Michael Kaminsky]: And this is one of the reasons is because you can take the full distribution of your estimated parameters and then plug that through some decision-making process and then evaluate on the other side.

[Michael Kaminsky]: Like how much would we regret making a wrong decision, right?

[Michael Kaminsky]: How much [Michael Kaminsky]: profit is going to generate on average, and the framework allows you to do that in a really nice and convenient way.

[Michael Kaminsky]: That's one nice thing.

[Michael Kaminsky]: If I was giving advice to you or to this product team, I would say, set up your decision-making process to optimize for whatever it is the thing that you care about.

[Michael Kaminsky]: You could imagine a product team saying, look, all that we care about is not making a huge mistake.

[Michael Kaminsky]: We're going to design tests [Michael Kaminsky]: that are only going to rule out huge mistakes.

[Michael Kaminsky]: We're probably going to release a bunch of stuff that isn't actually better than the old stuff.

[Michael Kaminsky]: That's fine.

[Michael Kaminsky]: We're fine with that.

[Michael Kaminsky]: We're going to design our decision-making procedure.

[Michael Kaminsky]: And again, we can set up our Bayesian framework to do this so that we run a lot of very small tests.

[Michael Kaminsky]: And we're going to power these tests just to be able to rule out the very bad errors.

[Michael Kaminsky]: And everything else is just going to go through.

[Michael Kaminsky]: you could have a different decision-making procedure that goes the opposite directions.

[Michael Kaminsky]: We only care about winners.

[Michael Kaminsky]: Everything that isn't a huge winner, we're going to throw out.

[Michael Kaminsky]: You can flexibly design your statistical analysis of these different experiments that you're going to run to optimize for whatever it is the thing that you care about.

[Michael Kaminsky]: And that's all, again, I think totally reasonable.

[Michael Kaminsky]: We don't have to have like the perfect A-B test 50-50 split statistically significant.

[Michael Kaminsky]: We can do a bunch of other stuff that isn't bad and still have a very reasonable business outcome.

[Michael Kaminsky]: But we need to use our like business judgment to decide what that is.

[Michael Kaminsky]: And then we can make the statistics sort of do whatever we want.

[Michael Kaminsky]: Another good example of where Bayesian statistics can be a nice add-on is we know in AB tests, there's this problem of the winner's curse, which is to say that businesses that run lots of AB tests will get a bunch of winners that actually aren't very good.

[Michael Kaminsky]: There's a bunch of reasons why this happens, but it has to do with [Michael Kaminsky]: Eventually, if you run enough tests and you only select the statistically significant ones, a bunch of those aren't going to be actual true winners.

[Michael Kaminsky]: Again, because of false positive rates, this should be fairly intuitive.

[Michael Kaminsky]: You can apply a Bayesian prior to your test results to account for that.

[Michael Kaminsky]: We're going to actually make sure that tests get even further.

[Michael Kaminsky]: There's even more signal than what would be implied by a normal traditional one-off AB test to try to account for the winner's curse.

[Michael Kaminsky]: That's, again, very reasonable thing to do.

[Michael Kaminsky]: You can make that as strong or as weak as you want, but it allows you to take your information from previous tests, from your own business judgment, from the cost of releasing something bad, and then pull that together into one single decision.

[Michael Kaminsky]: The Bayesian approach gives you a nice framework for being able to do that.

[Michael Helbling]: All right.

[Michael Helbling]: We do have to start to wrap up.

[Michael Helbling]: Who could have predicted that that would happen?

[Michael Helbling]: Uh, this is good.

[Michael Helbling]: Really good.

[Michael Helbling]: I'm, I'm, my mind is going a lot faster than it was when we started.

[Michael Helbling]: And so I'm thinking about a lot of different applications.

[Michael Helbling]: So this is awesome.

[Michael Helbling]: Thank you, Michael.

[Michael Helbling]: And now it's time for a quick break with our friend, Michael Kominsky.

[Michael Helbling]: Think of that.

[Michael Helbling]: From ReCast, they are the media mix modeling and GLIF platform, helping teams forecast accurately and make better decisions.

[Michael Helbling]: I can't wait to see what you have to share with us.

[Michael Helbling]: This time was, which is a marketing science bite-sized lesson to help us all measure smarter.

[Michael Helbling]: Take it away, Michael.

[Michael Helbling]: And this is pre-recorded, Michael.

[Michael Kaminsky]: Causality is a complex topic with deep philosophical roots.

[Michael Kaminsky]: Determining or proving causality is notoriously difficult, and in the context of analytics, there's lots of confusion about what we mean by causality at all and how to determine if we found it.

[Michael Kaminsky]: Because of the correlation versus causation problem, simply being able to predict what will happen in some system is not enough to demonstrate causality.

[Michael Kaminsky]: That's because if the system is stable, you can use correlations to make very good predictions.

[Michael Kaminsky]: So predictions alone are not good indications of a good causal model.

[Michael Kaminsky]: To demonstrate that our model is causal, we have to be able to forecast accurately when there's an intervention.

[Michael Kaminsky]: This is the core of the philosophy of causality in the sciences.

[Michael Kaminsky]: In the chemistry lab, if we can consistently predict what will happen when mixing different compounds together, we likely have a good understanding of the underlying causal mechanics.

[Michael Kaminsky]: And in business, we should think the same way.

[Michael Kaminsky]: We can demonstrate that we understand causality when we can intervene on some system and still predict what will happen.

[Michael Kaminsky]: By combining research and analysis with taking action, we can validate our understanding of causality and actually make our business better at the same time.

[Michael Kaminsky]: Remember, you can't demonstrate causality from the armchair.

[Michael Kaminsky]: You have to make a falsifiable prediction and then actually go out in the world and take action.

[Michael Helbling]: All right, thanks, Michael.

[Michael Helbling]: And for those who haven't heard, our friends at ReCast just launched their new incrementality testing platform, GeoLift, by ReCast.

[Michael Helbling]: It's a simple, powerful way for marketing and data teams to measure the true impact of their advertising spend.

[Michael Helbling]: And even better, you can use it completely free for six months.

[Michael Helbling]: Just visit getrecast.com slash geolift.

[Michael Helbling]: That's getrecast.com slash geolift to start your trial today.

[Michael Helbling]: All right.

[Michael Helbling]: Well, one of the things we love to do on the show is do a last call, something that we shared might be of interest to our audience.

[Michael Helbling]: Michael, you're our guest.

[Michael Helbling]: Do you have a last call you'd like to share?

[Michael Helbling]: Sure.

[Michael Kaminsky]: I'm reading this book currently called The Maniac by Benjamin Labotute.

[Michael Kaminsky]: It's very bizarre, but maybe of interest to people who listen to this podcast.

[Michael Kaminsky]: It's sort of a biography slash [Michael Kaminsky]: you know, fictionalized biography of Johnny von Neumann, who was one of the original inventors of the computer, spent a lot of time not necessarily working on cracking the enema code, but working with the scientists that were doing that, developed a lot of amazing science and mathematics, and then also sort of a meditation on AI and sort of the trajectory of the world as computing power explodes.

[Michael Kaminsky]: So very sort of bizarre literary fiction biography thing, but if that sounds good to you, I highly recommend checking it out.

[Michael Kaminsky]: out.

[Michael Helbling]: Very cool.

[Michael Helbling]: All right, Moe, what about you?

[Michael Helbling]: What's your last call?

[Moe Kiss]: I have two, but they are both very show related, so that feels appropriate.

[Moe Kiss]: The first one is Cassie Kozyrkov has an article, Statistics for People in a Hari.

[Moe Kiss]: It leans very heavily on the frequent aside.

[Moe Kiss]: But the favorite thing in my life at the moment is that she also does an audio version where she reads out the article she's written, which I adore because she is a very witty [Moe Kiss]: uh human and so getting to hear her read it you can definitely appreciate the wit a bit more and but also just on the enigma code if you want to read something for fun it is not a science book it is i think they call it historical fiction so it's based largely on a lot of historical events but it's [Moe Kiss]: totally fiction, with maybe some characters inspired by people in history.

[Moe Kiss]: One that I really enjoyed was called The Rose Code, which was about the enigma and the women who worked at Bletchley Park, the author is Kate Quinn.

[Moe Kiss]: And she does these amazing historical fiction books that are just actually enjoyable to read, but you get to know, I don't know, a little bit about a topic or a time in history that you wouldn't have known about otherwise.

[Moe Kiss]: And so those are my two for today.

[Tim Wilson]: Over to you, Tim.

[Tim Wilson]: I'm going to go with two as well.

[Tim Wilson]: These are both a couple of posts from people who have been on this show in the past now that I realize that both the little bit cranky, both kind of spitting into the wind or screaming into the void.

[Tim Wilson]: Yeah, which resonated with me.

[Tim Wilson]: So one, Juliana Jackson, her sub-stack, she had a piece called how Silicon Valley's hype machine continuously creates solutions nobody is asking for and has a lot of good history of past places and I think just makes some really spot-on points.

[Tim Wilson]: And then even in Juliana, I would say kind of [Tim Wilson]: raging against the machine is kind of on brand.

[Tim Wilson]: The other is a medium post by Eric Sandisham, who I think of as being a little bit kind of calmer and less likely to scream into the void.

[Tim Wilson]: But he wrote a piece called Another Bullshit Job, the AI Transformation Officer exclamation point, which is also a pretty nice little takedown on that front.

[Tim Wilson]: Couple of good reasons.

[Michael Helbling]: I would go as far as say, if you have transformation in your title, I'm nervous.

[Michael Helbling]: I'm nervous for you.

[Tim Wilson]: Well, Michael is the Chief Podcast Transformation Officer.

[Tim Wilson]: What's your last call?

[Tim Wilson]: Thanks so much for asking, Tim.

[Michael Helbling]: Wait, should I be nervous?

[Michael Helbling]: Actually, it's a good question.

[Michael Helbling]: Should I be nervous?

[Michael Helbling]: A lot of people are nervous about AI and whether it's going to take our jobs or whatever.

[Michael Helbling]: And so to that end, Cedric Chin over at Common Cog wrote something which I really like, and I'm passing out to everybody I run into who expresses those concerns.

[Michael Helbling]: It's called a letter to a young person worrying about AI.

[Michael Helbling]: And it's an excellent read as well as being Bayesian in its approach.

[Michael Helbling]: I realize now after the facts, which is cool tie into the show.

[Michael Helbling]: Anyways, I highly recommend it because I think we're all having that conversation about where's AI going?

[Michael Helbling]: What's it going to be like in five years?

[Michael Helbling]: Are we all going to work on robot repair and stuff like that in three, four months from now?

[Michael Helbling]: Whatever the case may be.

[Michael Helbling]: So that's [Michael Helbling]: That's a really good one.

[Michael Helbling]: All right.

[Michael Helbling]: As you've been listening, you've probably been thinking, oh, wow, this is a topic that I hadn't ever thought about before or I want to learn more about.

[Michael Helbling]: We'd love to hear from you.

[Michael Helbling]: And the best way to do that is to reach out to us and you can do that on the measure slack chat group or on LinkedIn or by email at contact at analyticshour.io and [Michael Helbling]: Obviously, we'd love it if you left us a review or a rating on your favorite platform where you listen to podcasts.

[Michael Helbling]: That's something we always appreciate hearing from you as well.

[Michael Helbling]: And if you're a huge fan of the show and you want to show it to the rest of the world, we've got stickers.

[Michael Helbling]: And so you can request that on analyticshour.io.

[Michael Helbling]: Just go to our website and we will send you some in the mail.

[Michael Helbling]: And we ship internationally.

[Michael Helbling]: for no apparent reason, but we do.

[Michael Helbling]: It will find Tim, Tim will get on a plane and he'll bring them to you through rain, snow, sleet and hail.

[Michael Helbling]: No, but anyways.

[Michael Helbling]: And yeah, so feel free to reach out and ask for stickers.

[Michael Helbling]: Okay.

[Michael Helbling]: Michael Kaminski, thank you so much.

[Michael Helbling]: This has been outstanding.

[Michael Helbling]: Appreciate it.

[Michael Helbling]: Thanks for having me.

[Michael Helbling]: This is a blast.

[Michael Helbling]: It's been such a pleasure.

[Michael Helbling]: So thanks again for coming back on the show.

[Michael Helbling]: We really appreciate it.

[Michael Helbling]: Keep up the great work.

[Michael Helbling]: We read all the stuff you put out and then we steal it and we bring it into the show as last calls and things like that.

[Michael Helbling]: So big help to us as well out there.

[Michael Helbling]: And I think I speak for both of my co-hosts because I have a prior about that.

[Michael Helbling]: Moe and Tim, when I say, [Michael Helbling]: No matter what your process, frequentist or Bayesian, keep analyzing.

[Announcer]: Thanks for listening.

[Announcer]: Let's keep the conversation going with your comments, suggestions and questions on Twitter at @analyticshour, on the web at analyticshour.io, our LinkedIn group, and the Measure Chat Slack group.

[Announcer]: Music for the podcast by Josh Grohurst.

[Announcer]: Those smart guys wanted to fit in, so they made up a term called analytics.

[Charles Barkley]: Analytics don't work.

[Charles Barkley]: Do the analytics say go for it, no matter who's going for it?

[Charles Barkley]: So if you and I were on the field, the analytics say go for it.

[Charles Barkley]: It's the stupidest, laziest, lamest thing I've ever heard for reasoning in competition.

[Moe Kiss]: I walk to work.

[Moe Kiss]: I walk to kids.

[Moe Kiss]: I mean, I can't listen when I have kids.

[Moe Kiss]: Or I'm like prepping dinner.

[Tim Wilson]: Interesting.

[Tim Wilson]: You have a Bluetooth speaker in the shower?

[Tim Wilson]: Or are you not?

[Moe Kiss]: I have one in the bathroom.

[Moe Kiss]: I was listening to ABC News Radio in the morning and I'm like, I should use this time more productively.

[Moe Kiss]: I don't get that much value from the news.

[Moe Kiss]: Yeah, that is incredibly optimized.

[Michael Helbling]: Yeah.

[Michael Helbling]: Why she's a co-host on the analyst power hour.

[Michael Helbling]: That's right.

[Michael Helbling]: I see how you recruited her.

[Moe Kiss]: I mean, there are many ways I let everybody down constantly.

[Michael Helbling]: So, you know.

[Michael Helbling]: Meanwhile, I'm in the kitchen.

[Michael Helbling]: Do you believe in life after love?

[Moe Kiss]: Well, last night I did make dinner to the dulcet tones of a song called Chicken Banana.

[Moe Kiss]: I highly encourage you to go listen because it is possibly the first song.

[Moe Kiss]: I don't think I'm going to have time for that.

[Michael Helbling]: Could you maybe just do it quick?

[Michael Helbling]: I mean, go ahead and knock it out.

[Michael Helbling]: Go ahead.

[Michael Helbling]: You can make it your last call if you want.

[Moe Kiss]: I mean, I should get the video of Harry, but it basically is like, uh, dance music.

[Moe Kiss]: I don't know why the kids like it, but it goes, chicken, banana, chicken, banana, to like high tech music.

[Michael Helbling]: That's why right there.

[Moe Kiss]: Yeah.

[Moe Kiss]: It's, it's a thing.

[Moe Kiss]: Yeah.

[Michael Helbling]: Something about the word chicken delights children.

[Michael Kaminsky]: Really?

[Michael Kaminsky]: Wait, so actually, are we doing, is this video, do we do video or is this audio?

[Tim Wilson]: Yeah, so we do the recording.

[Tim Wilson]: You're getting ahead of yourself.

[Michael Helbling]: Yeah.

[Michael Helbling]: Oh, he, you know, we might have changed the checklist to the last one.

[Moe Kiss]: Well, he's very sweaty.

[Michael Helbling]: Yeah, I just actually want to confirm.

[Michael Helbling]: We won't release the entire episode.

[Moe Kiss]: Do I have a massive lag?

[Michael Helbling]: But we typically will.

[Michael Helbling]: do shorts, so there'll be snippets that we'll take out and those we'll have video, but the whole show we do not do video.

[Michael Kaminsky]: All right, I'm gonna keep the shirt on for now, but I might beg off.

[Michael Helbling]: Yeah, that's fine.

[Michael Helbling]: It's fine.

[Michael Helbling]: Whatever you have to, and we won't...

I won't be fully shirtless.

[Michael Helbling]: I have an under shirt on, but I get it.

[Michael Helbling]: That's fine.

[Michael Helbling]: All right, so there's that.

[Michael Kaminsky]: I try to use the collared shirt for public reasons.

[Moe Kiss]: Oh, that is not where I thought this was going.

[Tim Wilson]: Rock Flag and Bayesian Enigmas!