[Announcer]: Welcome to the Analytics Power Hour.

[Announcer]: Analytics topics covered conversationally and sometimes with explicit language.

[Michael Helbling]: Hey everyone, welcome.

[Michael Helbling]: It's the Analytics Power Hour.

[Michael Helbling]: This is episode 277.

[Michael Helbling]: Given the chance, we'll frequently take any excuse to do just whatever we feel like doing.

[Michael Helbling]: But ostensibly, this episode is part of an unofficial series from our listener survey.

[Michael Helbling]: It's another deep dive into a statistical concept.

[Michael Helbling]: That's right.

[Michael Helbling]: We've got two keys in ANOVA, and we're going to drive straight into a post-hoc analysis of all of our life choices that led to this moment.

[Michael Helbling]: All right, we're doing Analysis of Variance, or ANOVA.

[Michael Helbling]: I really don't know much about it.

[Michael Helbling]: It was something that the SAS developers and the other department would talk about sometimes.

[Michael Helbling]: But I've, this one's been out of my league for a long time.

[Michael Helbling]: So I'm pretty excited to learn a little more.

[Michael Helbling]: Julie Hoyer, are you stoked about another stats-focused episode?

[Julie Hoyer]: Of course I am.

[Julie Hoyer]: I can't wait to get into it.

[Michael Helbling]: Awesome.

[Michael Helbling]: And Tim Wilson, I know you're probably a rarer end to go.

[Tim Wilson]: I'm looking forward to being more confused coming out of this than I am going in.

[Michael Helbling]: Well, I don't know if that's the result we're looking for, but we'll see if there's any significant difference in your knowledge after the fact.

[Michael Helbling]: All right, I'm Michael Hobling.

[Michael Helbling]: And for our guest, well, we had to do it.

[Michael Helbling]: We had to bring back our favorite statistician from just 10 episodes ago.

[Michael Helbling]: And it's her third time on the show, Chelsea Parlett Pellerini.

[Michael Helbling]: She's a consulting statistician at recast.

[Michael Helbling]: And she uses her PhD in computational data science.

[Michael Helbling]: She teaches statistics and math at Chapman University.

[Michael Helbling]: And once again, she is our guest.

[Michael Helbling]: Welcome back again, Chelsea.

[Chelsea Parlett-Pelleriti]: Thank you so much.

[Chelsea Parlett-Pelleriti]: I'll need to do something notable before the next time I come on, so you have a good bio for me that's different than this one.

[Michael Helbling]: Well, you know, we'll do a, we need to go deeper into like what your interests are.

[Michael Helbling]: So like, you know, Corgis, Stardew Valley, those kinds of things.

[Michael Helbling]: And we can start to put a bio together around that stuff.

[Chelsea Parlett-Pelleriti]: That'll come up today, actually.

[Michael Helbling]: Perfect.

[Michael Helbling]: It's not coming from me, but I think a good way to start this process is maybe start at the very beginning of analysis of variants, and even maybe start with how you feel about it, because I know it's not your favorite thing.

[Tim Wilson]: It wasn't a topic that Chelsea pitched us.

[Michael Helbling]: Yeah, yeah.

[Michael Helbling]: We were like, hey, please, can you come talk about this?

[Chelsea Parlett-Pelleriti]: Yes.

[Chelsea Parlett-Pelleriti]: Well, I'd actually like to start with a poem if you don't mind.

[Michael Helbling]: I love it.

[Michael Helbling]: Yeah.

[Michael Helbling]: That's right up my...

Absolutely.

[Michael Helbling]: Exactly the kind of start I think we need here.

[Chelsea Parlett-Pelleriti]: Perfect.

[Chelsea Parlett-Pelleriti]: Okay.

[Chelsea Parlett-Pelleriti]: So this kind of encapsulates how I feel and I want to clarify.

[Chelsea Parlett-Pelleriti]: I don't have a problem with the ANOVA itself.

[Chelsea Parlett-Pelleriti]: It is more [Chelsea Parlett-Pelleriti]: the way we communicate about it that sort of distracts people from things that are good about the ANOVA.

[Chelsea Parlett-Pelleriti]: So I'm not wholly against, but if I may, from the archives of my Twitter account, May of 2020.

[Chelsea Parlett-Pelleriti]: There once was a model ANOVA who, along with their cousin, ANCOVA, made a great big confession, were the same as regression, but we've established a separate persona.

[Michael Helbling]: Wow.

[Michael Helbling]: That's a memory.

[Michael Helbling]: Yeah.

[Michael Helbling]: Yeah, GPT could never.

[Chelsea Parlett-Pelleriti]: Yeah.

[Chelsea Parlett-Pelleriti]: They could never come up with that.

[Chelsea Parlett-Pelleriti]: So that's the basis of how I feel about ANOVA's, which is that they're linear models.

[Chelsea Parlett-Pelleriti]: And when we teach them as separate concepts, people sort of lose that connection.

[Chelsea Parlett-Pelleriti]: And so that's my biggest [Chelsea Parlett-Pelleriti]: gripe with ANOVA.

[Chelsea Parlett-Pelleriti]: It's not the actual math or anything behind it.

[Chelsea Parlett-Pelleriti]: It's that often, especially in my, you know, original field of psychology, people teach these ANOVA models, ANCOVA's, MANOVA's, you know, all the different letters that you can cram in there.

[Chelsea Parlett-Pelleriti]: And they teach them as something that's distinct from a regression model.

[Chelsea Parlett-Pelleriti]: And when you do that, people really lose two things.

[Chelsea Parlett-Pelleriti]: One is [Chelsea Parlett-Pelleriti]: a connection to any of the really great linear regression knowledge and content that they have.

[Chelsea Parlett-Pelleriti]: And two is the generalizability of the concepts that you learn in an ANOVA.

[Chelsea Parlett-Pelleriti]: One of the things that I would run into a lot, especially in my psychology days, is people thought an ANOVA is one thing and ANCOVA is another thing.

[Chelsea Parlett-Pelleriti]: A Minova is a third thing, a repeated measure.

[Chelsea Parlett-Pelleriti]: A Minova is a fourth thing.

[Chelsea Parlett-Pelleriti]: And they didn't see how they were related because they weren't taught in the linear model context.

[Chelsea Parlett-Pelleriti]: So they didn't see like, oh, an ANCOVA is just like, you basically add a covariate to your regression model.

[Chelsea Parlett-Pelleriti]: And so that's my main issue is that when people talk about using ANOVA's, they're typically talking about it in this framework of like, this is a separate thing from a regression model when really what you're doing when you fit in ANOVA, when you use ANOVA is [Chelsea Parlett-Pelleriti]: you're fitting a linear model and then you're looking at the outputs of it slightly differently than you might if you ran a traditional LM in R where you're running a linear model.

[Tim Wilson]: Putting aside all of the linkages there, defining where an ANOVA is or could or should be, what's it doing?

[Tim Wilson]: What's the purpose of that class of methodologies?

[Chelsea Parlett-Pelleriti]: Yeah, it's in the name.

[Chelsea Parlett-Pelleriti]: So an analysis of variants or an ANOVA is...

Well, that just turned off the business users right there.

[Tim Wilson]: You're like, come on, what for do you need to know?

[Chelsea Parlett-Pelleriti]: Yeah, of course, obviously.

[Chelsea Parlett-Pelleriti]: But it's analyzing the variants and the data.

[Chelsea Parlett-Pelleriti]: So if you think about, let's say, a data set that we could have, say, you're trying to see there's three different ad campaigns that you trialed, and you're trying to figure out [Chelsea Parlett-Pelleriti]: Are they different?

[Chelsea Parlett-Pelleriti]: Are they all giving you the same click rate or are they not?

[Chelsea Parlett-Pelleriti]: Are they all giving you the same average order or are they not?

[Chelsea Parlett-Pelleriti]: When you look at an ANOVA, essentially what it does is it says, we'll look.

[Chelsea Parlett-Pelleriti]: Let's take the average value of the order.

[Chelsea Parlett-Pelleriti]: So like if you have all of the order values during your experiment, you have these three different marketing campaigns say that you sent out, one of the things that you might wanna do is say, okay, there's a lot of variance here, right?

[Chelsea Parlett-Pelleriti]: Some of my orders are for $70, some of them are for $20, some of them are for $400.

[Chelsea Parlett-Pelleriti]: What can explain that difference?

[Chelsea Parlett-Pelleriti]: in the orders that we see.

[Chelsea Parlett-Pelleriti]: So we're observing that orders are not all the same.

[Chelsea Parlett-Pelleriti]: Why are they different?

[Chelsea Parlett-Pelleriti]: And we basically take all of the variation that we see.

[Chelsea Parlett-Pelleriti]: Not everyone has the same order value.

[Chelsea Parlett-Pelleriti]: You can picture the mean order value and everyone's order value is hovering around that mean value.

[Chelsea Parlett-Pelleriti]: Some are really high, some are really low.

[Chelsea Parlett-Pelleriti]: And so what we're doing is we're partitioning that variance [Chelsea Parlett-Pelleriti]: into sources that we care about.

[Chelsea Parlett-Pelleriti]: And the simplest case, like a one-way ANOVA, you'll often hear people talk about.

[Chelsea Parlett-Pelleriti]: The simplest case is that we have two categories, a variation that we really care about.

[Chelsea Parlett-Pelleriti]: One is variation due to the group, or in this case, the marketing campaign, and then variation due to what we would call, quote unquote, randomness, right?

[Chelsea Parlett-Pelleriti]: So it's variation within a group.

[Chelsea Parlett-Pelleriti]: And so at its simplest level, the ANOVA is basically taking that variance [Chelsea Parlett-Pelleriti]: and it's partitioning it into those groups.

[Chelsea Parlett-Pelleriti]: So what variants can we attribute to marketing campaign?

[Chelsea Parlett-Pelleriti]: What variants can we not contribute to marketing campaign?

[Chelsea Parlett-Pelleriti]: And then it compares those things.

[Chelsea Parlett-Pelleriti]: And essentially what you're doing when you're running what we typically think of as an ANOVA is you're seeing if there's statistical significance or you could use a Bayesian framework, but usually you're using a Frequentist framework.

[Chelsea Parlett-Pelleriti]: You're seeing like, is this statistically significant?

[Chelsea Parlett-Pelleriti]: Is the amount of variance that this explains [Chelsea Parlett-Pelleriti]: something that is notable or unexpected under the null.

[Chelsea Parlett-Pelleriti]: That's what we're doing.

[Chelsea Parlett-Pelleriti]: We're just partitioning the variants, and variants like the ANCOVA are just adding another category.

[Chelsea Parlett-Pelleriti]: If we have a covariate, say age or...

Wait, hold on.

[Tim Wilson]: Can we stop before we go one level?

[Tim Wilson]: So, just to say took the dumb dumb analyst or the marketer who's just looking and says, I'm just looking at average order value and I break it down by campaign and one average order value is $75.

[Tim Wilson]: One is $80 and one is $90.

[Tim Wilson]: Just by looking at the, which is a mean, the average order value, that there's a tendency to say, well, these are different and [Tim Wilson]: It's easy to say, well, that's the difference between these.

[Tim Wilson]: But everything you just described was saying, well, if order values are all over the place and it just happened to be that you dropped in and partitioned them, sliced them by campaign, yeah, you just happened to get bigger ones in one and not in the other.

[Tim Wilson]: So it's giving you a way to say, [Tim Wilson]: given these observe different means, how am I confident that that means the way that I partition them actually is contributing to that.

[Tim Wilson]: It's not just that I'm just kind of arbitrarily seeing that it's a noisy, wide spread.

[Tim Wilson]: Am I playing that back accurately?

[Chelsea Parlett-Pelleriti]: Exactly.

[Chelsea Parlett-Pelleriti]: So you can imagine a scenario where, let's say you have this magical campaign where everyone who gets variant A, their order is right around $80.

[Chelsea Parlett-Pelleriti]: Sometimes it's $81.

[Chelsea Parlett-Pelleriti]: Sometimes it's $79, but it's right around $80.

[Chelsea Parlett-Pelleriti]: And variant B, it's right around $60.

[Chelsea Parlett-Pelleriti]: Sometimes it's $61.

[Chelsea Parlett-Pelleriti]: Sometimes it's $59, but always around the same.

[Chelsea Parlett-Pelleriti]: In that case, it would be super clear, even without a statistical test, that you could visually plot that data out and you would see that the amount that orders vary within your campaign variance is so small compared to the amount that the two differ from each other.

[Chelsea Parlett-Pelleriti]: I think I said $20 difference between them.

[Chelsea Parlett-Pelleriti]: and that's what you're quantifying mathematically for basically cases where you can't immediately see on a graph like in the example I just described.

[Chelsea Parlett-Pelleriti]: So technically, in an ANOVA, the null hypothesis that you're testing is that whatever groups you have, so it's usually two or more because if you had only two, you could use a t-test.

[Chelsea Parlett-Pelleriti]: But basically, you're saying, [Chelsea Parlett-Pelleriti]: All of the means of these groups, however many there are, are equal.

[Chelsea Parlett-Pelleriti]: That's the null hypothesis.

[Chelsea Parlett-Pelleriti]: And the alternative hypothesis is that at least two of these means are different.

[Chelsea Parlett-Pelleriti]: And so this gets into something that maybe is too deep.

[Chelsea Parlett-Pelleriti]: You can stop me again and we'll go back.

[Chelsea Parlett-Pelleriti]: But this is an ANOVA is like the F test in an ANOVA that would say, OK, here I have three campaigns or I have 10 campaigns.

[Chelsea Parlett-Pelleriti]: Is there a statistically significant amount of variance explained by campaign?

[Chelsea Parlett-Pelleriti]: You're essentially doing something called an omnibus test where you're testing.

[Chelsea Parlett-Pelleriti]: Is there a difference somewhere in this mess?

[Chelsea Parlett-Pelleriti]: but it won't tell you by itself the F test, will not tell you by itself where is that difference.

[Chelsea Parlett-Pelleriti]: The omnibus test is looking overall at the variance explained when you know what campaign someone has been exposed to, whereas typically we'll often have questions that are a little more targeted than that.

[Chelsea Parlett-Pelleriti]: We want to know, okay.

[Chelsea Parlett-Pelleriti]: This is our business as usual campaign.

[Chelsea Parlett-Pelleriti]: Here's an experimental campaign and here's an amped up version of that experimental campaign.

[Chelsea Parlett-Pelleriti]: In that case, what we really probably care about is business as usual different from the other two and is our amped up experimental campaign better than the [Chelsea Parlett-Pelleriti]: like regular experimental one.

[Chelsea Parlett-Pelleriti]: And so in ANOVA by itself won't tell you that you'd have to follow it up with post hoc tests, which you mentioned in the intro.

[Chelsea Parlett-Pelleriti]: And yeah, so that's essentially what you're doing at kind of the simplest level of ANOVA.

[Julie Hoyer]: So in that example, actually, would you be able to just run an ANOVA on the data for the business as usual campaign and the experimental one and just do that pairing?

[Julie Hoyer]: So you could choose the pairs of those to look at.

[Julie Hoyer]: So then you could have the clear answers you're talking about.

[Julie Hoyer]: But traditionally, an ANOVA, somebody might be like, no, we're going to throw all three in.

[Julie Hoyer]: And to your point, the result that would come out of that ANOVA would just say if there is [Julie Hoyer]: actual variance between those three categories.

[Julie Hoyer]: It won't tell you between which two.

[Julie Hoyer]: But with three, it's easy to be like, I'll just split it out.

[Julie Hoyer]: But to your point, like a lot of times if we have a ton of categories, it becomes very cumbersome and not realistic.

[Michael Helbling]: All right, let's talk data.

[Michael Helbling]: We all love insights, but let's face it, setting up integrations, that's not exactly a party.

[Michael Helbling]: That is why there's 5Tran, the smartest, easiest way to automate your data pipelines.

[Michael Helbling]: Think of it like the ultimate, set it and forget it gadget for the discerning data professional.

[Michael Helbling]: Connect, relax, and let 5Tran handle the heavy lifting.

[Michael Helbling]: Your data lands safely and swiftly into your warehouse ready for action and analysis.

[Michael Helbling]: Curious?

[Michael Helbling]: I think you are.

[Michael Helbling]: Head over to 5Tran.com slash APH right now to stay updated on the latest news and events and see how 5Tran can make your data dreams come true.

[Michael Helbling]: That's F-I-V-E-T-R-A-N dot com slash APH.

[Michael Helbling]: Trust me.

[Michael Helbling]: If you're in the litics, well, thank you.

[Chelsea Parlett-Pelleriti]: Totally.

[Chelsea Parlett-Pelleriti]: Well, not only come...

I have so many things to say to that because that was such a good point.

[Tim Wilson]: Well, so I think you want to hit the, what if there were just two and then you want to hit the, what if there are a whole bunch?

[Chelsea Parlett-Pelleriti]: Well, if there's just two, it kind of doesn't matter what you do because you're essentially by running in a nova, running a t-test between them.

[Chelsea Parlett-Pelleriti]: Something I loved.

[Chelsea Parlett-Pelleriti]: I don't know why this fact was so fun to me back in the day, but when I first learned this, [Chelsea Parlett-Pelleriti]: I learned that the F statistic you get under very specific conditions, including like there's only two groups, is just the T statistic squared that you would have gotten if you had run the same type of T test instead of an ANOVA.

[Chelsea Parlett-Pelleriti]: So there's a really like one-to-one relationship there.

[Chelsea Parlett-Pelleriti]: But you said two things that I thought were really important.

[Chelsea Parlett-Pelleriti]: One is that it's cumbersome to run a bunch of these different comparisons, which is true, but in a sense unavoidable if you're interested in all of pairwise comparisons.

[Chelsea Parlett-Pelleriti]: But I think the point that you're implying but not saying out loud is there's also a problem if you're using the frequentist framework in multiple testing.

[Chelsea Parlett-Pelleriti]: Let's say I have 10 groups and I want to compare every pair of two.

[Chelsea Parlett-Pelleriti]: I can't do that in my head, but it's what [Chelsea Parlett-Pelleriti]: 10 choose 2.

[Chelsea Parlett-Pelleriti]: I don't know what that number is.

[Chelsea Parlett-Pelleriti]: Lots of comparisons that are happening.

[Chelsea Parlett-Pelleriti]: And usually in a frequentist framework, we are choosing at an alpha level.

[Chelsea Parlett-Pelleriti]: So usually we use 0.05.

[Chelsea Parlett-Pelleriti]: So 5% as our expected error rate under the null.

[Chelsea Parlett-Pelleriti]: So it's like type 1 error rate if there is no effect.

[Chelsea Parlett-Pelleriti]: This is how often we'll be misled by the conclusions we make of the test.

[Chelsea Parlett-Pelleriti]: But if you're running [Chelsea Parlett-Pelleriti]: 30 of those comparisons.

[Chelsea Parlett-Pelleriti]: Suddenly, your family-wise error rate, which is the error rate of making a mistake in that family of comparisons, is huge.

[Chelsea Parlett-Pelleriti]: That's a problem.

[Chelsea Parlett-Pelleriti]: Another thing that is important is, yeah, we could just go filter the data, only include baseline, business as usual, and the experimental.

[Chelsea Parlett-Pelleriti]: But one of the things that can be really helpful within ANOVA is that your [Chelsea Parlett-Pelleriti]: actually increasing your power, statistical power that is not, you know.

[Chelsea Parlett-Pelleriti]: I don't know what other kind of power you're increasing.

[Chelsea Parlett-Pelleriti]: You're actually increasing your power because the estimate of your error is going to be more precise with more groups.

[Chelsea Parlett-Pelleriti]: Because one of the assumptions of an ANOVA is that you have, I think this is just heteroscedasticity, but basically you're assuming that the variance is the same.

[Chelsea Parlett-Pelleriti]: across your different groups.

[Chelsea Parlett-Pelleriti]: And one of the things that that gives you, if that's true, is that you get a better estimate of what that error is if you look at all 10 groups that you have than if you truncated your data and only looked at the two groups that you, for instance, in this case, are interested in.

[Chelsea Parlett-Pelleriti]: And so you're actually increasing your power of estimation a little bit if assumptions hold, et cetera, et cetera.

[Chelsea Parlett-Pelleriti]: And so there's [Chelsea Parlett-Pelleriti]: actually a benefit to running the ANOVA itself power-wise.

[Chelsea Parlett-Pelleriti]: But also, like you're pointing out, you could partition everything, but you'd have to be more thoughtful about what comparisons you want to run and control your family-wise error rate.

[Chelsea Parlett-Pelleriti]: Now, I'll take it one step back to my critique of the ANOVA.

[Chelsea Parlett-Pelleriti]: I actually think it's better to be thoughtful about the contrast.

[Chelsea Parlett-Pelleriti]: So in Innova, we usually call them contrast, right?

[Chelsea Parlett-Pelleriti]: Like which comparisons do you want to run?

[Chelsea Parlett-Pelleriti]: And I actually think it's better to be thoughtful about that, correct for any family-wise error rate inflation that you're causing, and just look at those.

[Chelsea Parlett-Pelleriti]: rather than rely on the omnibus test, unless you're actually just trying to answer the question the omnibus test answers, which is, is there variation somewhere?

[Chelsea Parlett-Pelleriti]: Are all the means equal?

[Chelsea Parlett-Pelleriti]: It's a bit of a weird way to say that.

[Chelsea Parlett-Pelleriti]: I actually think it's better to be thoughtful.

[Chelsea Parlett-Pelleriti]: And one of the things that is good about how people teach the ANOVA is usually you teach, okay, you run the omnibus test, but then you follow it up with post hoc comparisons, right?

[Chelsea Parlett-Pelleriti]: Different pair-wise comparisons you might want to know about.

[Chelsea Parlett-Pelleriti]: And one of the things I really love [Chelsea Parlett-Pelleriti]: when I was learning this back in the day, is that there are some really thoughtful frameworks.

[Chelsea Parlett-Pelleriti]: You can define contrast however you want.

[Chelsea Parlett-Pelleriti]: If you've ever worked with a novice in R, you know you can define your own contrast matrix.

[Chelsea Parlett-Pelleriti]: Whatever contrast you want to run, you just put them in there and it'll run in.

[Chelsea Parlett-Pelleriti]: But there's some established ones that I think are really thoughtful.

[Chelsea Parlett-Pelleriti]: And some of them have to do with the example we talked about of, okay, here's a business as usual.

[Chelsea Parlett-Pelleriti]: So in a sense, like a control group.

[Chelsea Parlett-Pelleriti]: And then here's a moderate experimental and an extreme experimental condition.

[Chelsea Parlett-Pelleriti]: There's different types of contrast where they kind of predefined for you what you're interested in.

[Chelsea Parlett-Pelleriti]: So I'm interested in control versus the average of the experimental.

[Chelsea Parlett-Pelleriti]: That sort of answer is, is my experimental condition working?

[Chelsea Parlett-Pelleriti]: and then I might be interested secondarily in the contrast between moderate experimental and extreme experimental, because then that tells me, hey, when I really take this campaign to the nth degree, force someone to click on my ad, essentially, is that actually helping compared to my more moderate, hey, click on my banner?

[Chelsea Parlett-Pelleriti]: Those contrasts are very thoughtful.

[Chelsea Parlett-Pelleriti]: It's very specific to the situation you're in.

[Chelsea Parlett-Pelleriti]: My overall critique of statistics as a whole is that sometimes we encourage people to not be thoughtful, and I'm always in favor of something that encourages someone to be thoughtful.

[Chelsea Parlett-Pelleriti]: It's not the ANOVA's fault per se, but it can encourage people to just look at the omnibus F statistic, F test, when that's not really what their question is.

[Chelsea Parlett-Pelleriti]: And because they haven't thought about it, it's just this like, I learned in ANOVA five years ago, I'm gonna throw an ANOVA at it.

[Chelsea Parlett-Pelleriti]: You really lose a lot, both of statistical power as well as clear answers to your questions.

[Julie Hoyer]: And to be a little specific, when you were saying you get more power by having more categories put into your ANOVA, is that because to calculate the F statistic, it's comparing the variance within the groups [Julie Hoyer]: to the variance between the groups.

[Julie Hoyer]: And so if you have more groups, you get more inputs for both of those measures.

[Julie Hoyer]: So inherently, you're getting an F statistic that's more representative of, or something you could generalize more across the [Julie Hoyer]: Like the categories, am I getting close?

[Julie Hoyer]: But I'm thinking of sample size.

[Julie Hoyer]: So sample size of these variance measures, you're getting more of them with the more categories that you give to the ANOVA.

[Julie Hoyer]: So that's kind of where my brain was going, but I don't know if that's actually how that works.

[Chelsea Parlett-Pelleriti]: It's even simpler than that.

[Chelsea Parlett-Pelleriti]: I think you're correct.

[Chelsea Parlett-Pelleriti]: But also, even if I am just interested in category A versus category B, if I'm assuming that all of my groups have the same variance, [Chelsea Parlett-Pelleriti]: That's something I need to estimate with my model.

[Chelsea Parlett-Pelleriti]: I don't know what the population variance is there.

[Chelsea Parlett-Pelleriti]: If I have seven groups and you're saying bigger sample size to estimate what that variance is, even if it doesn't help me with the between group thing, it helps me with the within group estimation, which is exactly what can happen here.

[Chelsea Parlett-Pelleriti]: But I will say that relies on the assumption that they're all the same and that the pooled variance is a good estimate.

[Tim Wilson]: I have two questions and you can choose to ignore the first one if it's like that is a whole other episode but just some fundamental intuition about what a t-test is and does and maybe it is a companion because as you're talking about [Tim Wilson]: You've got a control group and an experimental group, and that doesn't necessarily have to be run in a controlled experiment.

[Tim Wilson]: You just got different groups.

[Tim Wilson]: But when you run a controlled experiment where you do have multiple groups in an experimental fashion, [Tim Wilson]: you wouldn't really use an ANOVA or would you?

[Chelsea Parlett-Pelleriti]: In the example you gave, it sounds like there's only two, like an experimental and a control group.

[Chelsea Parlett-Pelleriti]: And in that case, the T test should give you roughly equivalent, if not exactly equivalent results to an ANOVA on the same value.

[Chelsea Parlett-Pelleriti]: So like you, I've actually run this before, like when I was teaching or when I back when I was working in psychology where like you can [Chelsea Parlett-Pelleriti]: set up data like that and then run it through the ANOVA function in R, run it through a t-test in R.

You'll basically see what I said earlier is that you're going to get the same p-value with rounding and computational error, and then you're going to get the t-statistic is, or the f-statistic is t-squared.

[Chelsea Parlett-Pelleriti]: You're really answering the same question there.

[Chelsea Parlett-Pelleriti]: In that case, it does not matter.

[Tim Wilson]: What's the approach of the t-test?

[Tim Wilson]: I get that you wind up in the same spot, but presumably if you're teaching a t-test, you talk about it in a completely different way.

[Chelsea Parlett-Pelleriti]: It will say it's a different framework.

[Chelsea Parlett-Pelleriti]: And the one thing I do love about how we teach ANOVA's is that in a t-test, what you're testing is the difference, like the delta between the two means.

[Chelsea Parlett-Pelleriti]: And you're comparing that to a distribution under the null and blah, blah, blah.

[Chelsea Parlett-Pelleriti]: In an ANOVA, [Chelsea Parlett-Pelleriti]: you're really thinking of things not as like, okay, here's a difference in means that I'm testing, but here's the variance that's explained by knowing what category someone's in compared to variance that's not explained by that.

[Chelsea Parlett-Pelleriti]: And then again, you stopped me before, but if you have an Encova, I'm going to squeeze it in now, you can partition into a third category, which is variance due to a covariate like [Chelsea Parlett-Pelleriti]: age or location or something like that.

[Chelsea Parlett-Pelleriti]: So ANOVA is really focused on this partition of variance, how the data points vary about the mean.

[Chelsea Parlett-Pelleriti]: Can we explain part of that variance with your category and part with randomness?

[Chelsea Parlett-Pelleriti]: Whereas a t-test is mathematically, like you're pointing out exactly the same, you're going to get under certain circumstances, basically the exact same output.

[Chelsea Parlett-Pelleriti]: But you're thinking about it in a different way.

[Chelsea Parlett-Pelleriti]: A t-test is looking at [Chelsea Parlett-Pelleriti]: What is the difference in these group means?

[Chelsea Parlett-Pelleriti]: Say one group mean is 10 and the other one's five, that difference would be five.

[Chelsea Parlett-Pelleriti]: How likely are we to get a difference of five if there's truly no population difference between these groups?

[Chelsea Parlett-Pelleriti]: Whereas in ANOVA is answering, what is essentially the same question but from a slightly different perspective, which is, okay, if I know what group you're in, how much of the variance of the scores I'm getting, can I explain?

[Chelsea Parlett-Pelleriti]: And the benefit here is that Ananova technically generalizes to more groups, whereas a t-test, you would run pairwise t-tests between them.

[Tim Wilson]: So for the ANCOVA, can you introduce multiple covariates?

[Tim Wilson]: Yeah.

[Chelsea Parlett-Pelleriti]: You can do whatever you want.

[Chelsea Parlett-Pelleriti]: It's just a linear model.

[Tim Wilson]: Or regression.

[Tim Wilson]: So that starts to, OK.

[Chelsea Parlett-Pelleriti]: Exactly.

[Chelsea Parlett-Pelleriti]: Right.

[Chelsea Parlett-Pelleriti]: Okay.

[Chelsea Parlett-Pelleriti]: And so to get into that complaint, I think it's Danielle Whitten who had that series on Twitter months or maybe years ago, where she would just retweet things and say, it is just a linear model.

[Chelsea Parlett-Pelleriti]: Well, OK, let's be clear.

[Chelsea Parlett-Pelleriti]: Technically, what you're doing when you fit in ANOVA is you're fitting a linear model, and then you're using this framework of variance, the ANOVA, the analysis of variance, to analyze the results.

[Chelsea Parlett-Pelleriti]: But at its core, what you're analyzing around is just a linear model.

[Chelsea Parlett-Pelleriti]: And you can add more covariates.

[Chelsea Parlett-Pelleriti]: You can add tons of different things.

[Chelsea Parlett-Pelleriti]: And that's my problem with the framing of how we teach ANOVA's is it doesn't make it clear that that's the case.

[Chelsea Parlett-Pelleriti]: Whereas when we teach linear regression, we're a little bit better about [Chelsea Parlett-Pelleriti]: Yeah, throw in whatever covariates you want, throw in random effects, do GAMs, so do some smoothing and transform with polynomials, your predictors, and then put them in.

[Chelsea Parlett-Pelleriti]: And that flexibility is not inherent to the way that people have been communicating about ANOVA's.

[Julie Hoyer]: And I'm having this maybe light bulb moment unless I'm really not following it off the rails here.

[Julie Hoyer]: But Tim, remember when we talked about blocking in tests, RCTs and all that?

[Julie Hoyer]: And we're like, you just use a linear regression to analyze the result of your test.

[Julie Hoyer]: You can put all these covariates in and blocking something you represent in there.

[Julie Hoyer]: So if you're doing an ANOVA on a pair, like two simple values of a category, [Julie Hoyer]: It's similar to a T test.

[Julie Hoyer]: They're all linear regressions.

[Julie Hoyer]: And if I was doing this on an AB test, I could run a linear regression.

[Julie Hoyer]: Like I'm having this kind of like moment where Chelsea said like, it all goes back to regression.

[Chelsea Parlett-Pelleriti]: It's regression all the way down.

[Chelsea Parlett-Pelleriti]: Well, I don't know if this is too soon to bring this in, but I think what you're saying reminds me of something that I said when you reached out about this episode, which is, [Chelsea Parlett-Pelleriti]: sort of jokingly, but definitely not jokingly, that Cupid, which people use for A-B testing, I'm pretty sure is just an ANCOVA, right?

[Chelsea Parlett-Pelleriti]: So the whole idea is that you take this like, I believe I'm a little rusty on my Cupid, but you take kind of like pre-test metrics that you have about the customers that you're testing on, and you use that to reduce the variance in the data because you're accounting for it, you're partitioning variance, [Chelsea Parlett-Pelleriti]: It's blocking.

[Chelsea Parlett-Pelleriti]: I know.

[Chelsea Parlett-Pelleriti]: And you're just getting a better estimate, a more precise estimate, because of all the variants that's out there, you're accounting for some of it that would have previously been attributed to random variants.

[Chelsea Parlett-Pelleriti]: You're now accounting for it with your category.

[Chelsea Parlett-Pelleriti]: And that's what an ANCOVA really is trying to teach you, is you can add these additional non-experimental [Chelsea Parlett-Pelleriti]: groupings or continuous variables, and it'll reduce the amount of variance in the error estimation, giving you a more precise, more statistically powered result.

[Julie Hoyer]: And so this goes, okay, it is all coming together.

[Julie Hoyer]: I am having like a mind-blowing moment.

[Julie Hoyer]: Because that makes sense, then, where you want to use covariates that you know explain the outcome variable.

[Julie Hoyer]: So if you know age is a factor that would affect the outcome that you're trying to understand the variance for another category like your campaign, you're like, well, this age and the next age group, we know they spend really differently.

[Julie Hoyer]: By adding that in as a covariate, [Julie Hoyer]: like you're saying, you are narrowing in on then being able to detect variants from your campaign because you've isolated and muted the noise of variants from age that you know is a factor that affects it.

[Chelsea Parlett-Pelleriti]: Exactly.

[Chelsea Parlett-Pelleriti]: Whenever you add a covariate in a regression model, you're essentially saying, what can the other factors tell me after I have accounted for?

[Chelsea Parlett-Pelleriti]: this variable.

[Chelsea Parlett-Pelleriti]: And so if age is really important in explaining how people are behaving, then you're basically saying, okay, like if I know what campaign you got after accounting for all of the noise that happens because of your age.

[Chelsea Parlett-Pelleriti]: What does it tell me?

[Chelsea Parlett-Pelleriti]: And you're going to hopefully get a more accurate and precise measurement by including that.

[Chelsea Parlett-Pelleriti]: Now, finding things to include that are actually useful can sometimes be a challenge, but if you can find them, they really help the precision of your estimate.

[Tim Wilson]: My impression is that Cupid has become in the over the last couple of years, at least in the CRO world, like, oh, it's the latest kind of shiny bobble.

[Tim Wilson]: You can reduce your runtime.

[Tim Wilson]: This is great, I guess, and there might be at least one person who's probably already been triggered, because I know every time he sees Cupid, he winds up [Tim Wilson]: I get text messages.

[Tim Wilson]: And he's essentially saying, but it's not magic.

[Tim Wilson]: And I think it is probably because it's what you just got to that you can't just assume that you're going to have covariates that you can identify that actually have an effect on the independent variable or the dependent variable.

[Tim Wilson]: So you can't just assume that you're not going to have age in some cases.

[Tim Wilson]: Even if it is, you're not going to have that data.

[Chelsea Parlett-Pelleriti]: And even more than that, I've seen some examples, again, not my area of expertise, but I've seen a lot of examples where you have the same cold start problem you have in recommendation models where you may not have that data for a really important [Chelsea Parlett-Pelleriti]: sector of the people that you're experimenting on, especially this probably would come up most with new customers.

[Chelsea Parlett-Pelleriti]: People are right.

[Chelsea Parlett-Pelleriti]: Cupid would be so helpful.

[Chelsea Parlett-Pelleriti]: It means you can run shorter tests.

[Chelsea Parlett-Pelleriti]: It means you can run smaller tests.

[Chelsea Parlett-Pelleriti]: It means you can have more precise estimates.

[Chelsea Parlett-Pelleriti]: But there's no free lunch, right?

[Chelsea Parlett-Pelleriti]: You have to have like this quality data that's going to behave in the way you think it will.

[Chelsea Parlett-Pelleriti]: And it's the same idea as in ANOVA or in ANCOVA, right?

[Chelsea Parlett-Pelleriti]: You're just, can we account for or partition out some variants that we sort of know is there, is not the category of interest?

[Chelsea Parlett-Pelleriti]: Can we like section that off?

[Chelsea Parlett-Pelleriti]: And if you can, then I imagine Cupid is incredibly powerful.

[Chelsea Parlett-Pelleriti]: If you can't, [Chelsea Parlett-Pelleriti]: maybe less so, but agree with whoever you're vaguely referring to.

[Chelsea Parlett-Pelleriti]: It's not magic and we shouldn't act like it is.

[Tim Wilson]: You might be the same person that's had, we think, as many appearances on this show as you have.

[Tim Wilson]: That's true.

[Tim Wilson]: If I name him by name, I'll definitely hear from him.

[Tim Wilson]: But let me ask another question on those, because I can think of in a simple website experience digital, that there are things like, what was the most recent traffic source?

[Tim Wilson]: There are things like, what device type are you on that both, if you're looking to a conversion, seem like they would be legit covariates.

[Tim Wilson]: When you're talking about [Tim Wilson]: whether you're doing ANCOVA or whether it's Cupid, is that inherently a you that's part of the input to make your actual question of interest more useful as opposed to the flip side?

[Tim Wilson]: Oh, [Tim Wilson]: We looked at the overall test results, and now we're going to slice them by this other thing and see if significant pops up.

[Tim Wilson]: Is that a fundamentally different thing where you're continuing to slice?

[Tim Wilson]: This is saying, no, I'm identifying this as a covariate so that my question I can get [Tim Wilson]: a tighter, better answer to my actual question.

[Tim Wilson]: I'm trying to use that to remove variability.

[Chelsea Parlett-Pelleriti]: Yes.

[Chelsea Parlett-Pelleriti]: You're asking slightly a different question.

[Chelsea Parlett-Pelleriti]: If we're going to go in the linear model framework, you're asking a slightly different question when you say, is this relationship consistent between Android, iPhone, computer, whatever type users?

[Chelsea Parlett-Pelleriti]: That would actually be an interaction effect in your model where you're saying, does the relationship between [Chelsea Parlett-Pelleriti]: my campaign and order value change for web-based, phone-based, whatever.

[Chelsea Parlett-Pelleriti]: That would be an interaction term, which is something you can just add to a linear model, by the way, because it's so generalizable, which sometimes we don't realize with an ANOVA.

[Chelsea Parlett-Pelleriti]: But that's a slightly different question than I'm just having campaign in here, and I'm soaking up variance by telling you what platform someone was using.

[Chelsea Parlett-Pelleriti]: Because in that case, you're just saying, if I know your platform, can I [Chelsea Parlett-Pelleriti]: what additional information do I get from knowing about your campaign, whereas the interaction specifically would model that relationship differently for each platform and would allow you to answer that question of like, [Chelsea Parlett-Pelleriti]: Is it different?

[Chelsea Parlett-Pelleriti]: We'd probably look at the interaction terms there and see if they're significant, or you could even use a mixed effect model for this type of thing where you say, oh, all of the effects are similar, but they might deviate a little bit.

[Chelsea Parlett-Pelleriti]: How much do they deviate?

[Chelsea Parlett-Pelleriti]: You could answer the question that way as well.

[Julie Hoyer]: Because with an ANOVA and covariates, you're not actually interested in the difference between the covariates, like you're saying.

[Julie Hoyer]: You're just giving it extra information.

[Julie Hoyer]: But Tim was kind of posing it as more of a question of finding out the differences across that extra covariate dimension of device type, right?

[Chelsea Parlett-Pelleriti]: Yeah, so yeah, I mean, it's just two different questions.

[Chelsea Parlett-Pelleriti]: I will say, now I'm having to rely on really years old information that I haven't thought of.

[Chelsea Parlett-Pelleriti]: But I'm pretty sure for an ANCOVA, one of the first things that you're supposed to do, I don't think people do it.

[Chelsea Parlett-Pelleriti]: And I might make you cut this if I'm incorrect.

[Chelsea Parlett-Pelleriti]: But I'm pretty sure one of the assumption checks for an ANCOVA is that there's no significant interaction effect in terms of the covariate having.

[Chelsea Parlett-Pelleriti]: different relationships, like the interaction effects being significant.

[Chelsea Parlett-Pelleriti]: And I'm fairly certain that you're supposed to check that.

[Chelsea Parlett-Pelleriti]: And so in that case, it would be like, if you thought that was happening, you wouldn't want to just include the covariate.

[Chelsea Parlett-Pelleriti]: You would want to include interaction effects because clearly they're meaningful, but that's like a slightly different [Chelsea Parlett-Pelleriti]: question that you're answering.

[Chelsea Parlett-Pelleriti]: So yeah, and I think that's a really good point about like, you should be thoughtful about like, do you want to know that?

[Chelsea Parlett-Pelleriti]: Like, if you do, include the interaction effects.

[Chelsea Parlett-Pelleriti]: It's not a traditional ANCOVA, but because we're all brilliant and we know that this, you know, they're not discrete different models, it's just [Chelsea Parlett-Pelleriti]: different forms of a linear model, we can so easily just add an interaction effect and be like, okay, cool.

[Chelsea Parlett-Pelleriti]: We want to answer this question.

[Chelsea Parlett-Pelleriti]: We'll add those interaction effects.

[Chelsea Parlett-Pelleriti]: That's what I love about the linear model framework compared to the way that some people teach ANOVA's and COVA's as separate tools that you can use.

[Julie Hoyer]: I think that's the hardest part is [Julie Hoyer]: A lot of that thoughtfulness and the levers you can pull on these different statistical tools is how people think about them is really lost unless you deeply understand some of the math and the basics behind it.

[Julie Hoyer]: But as we know, a lot of times it's just a simple [Julie Hoyer]: you know command in your code to run this thing and if you aren't really good at checking all the assumptions and really thinking through the exact question you're answering it's so easy to use a slightly wrong tool and get a number on the screen.

[Julie Hoyer]: And thank you you're answering the right question and you're not and I think that's what is.

[Julie Hoyer]: scary in two ways.

[Julie Hoyer]: You have to be really knowledgeable to answer that question of like, is this the right number to answer the business question I'm asking?

[Julie Hoyer]: And two, it's really easy for people to give you a number and they haven't asked themselves that question or been thoughtful about it.

[Julie Hoyer]: Both of those equally scare me.

[Chelsea Parlett-Pelleriti]: And how are you supposed to be an expert both in like, you need the business expertise to know what question is actually important.

[Chelsea Parlett-Pelleriti]: And like you said, you need the statistical expertise to know if the number you're getting is targeting that question and what the caveats there are.

[Chelsea Parlett-Pelleriti]: And that's one of the things that scares me the most is like, [Chelsea Parlett-Pelleriti]: How are you supposed to be an expert in both?

[Chelsea Parlett-Pelleriti]: I think the answer is you're not, and you have to collaborate.

[Michael Helbling]: Yeah, thanks, Julie.

[Michael Helbling]: My anxiety had been going down as I was understanding this better, and now I just want to break back up again.

[Tim Wilson]: But I think there's the flip side.

[Tim Wilson]: Maybe this is...

[Tim Wilson]: This is part of the reasons that I wanted to talk about ANOVA because I have a very, very clear memory and this was when I was sort of still learning R and I'd kind of gone down the, okay, there's the benefit of just programmatically being able to do stuff.

[Tim Wilson]: that's not clicking around on an interface.

[Tim Wilson]: I kept being told, well, to learn R, you're just going to inherently learn statistics.

[Tim Wilson]: What I don't really think is true, but at some point, I mean, it was just sort of said that if you're going to learn R, you're going to have to learn the statistics.

[Tim Wilson]: They'll come hand in hand, and that didn't really happen.

[Tim Wilson]: when it comes to illustrating and ANNOVA, and I don't know if I've seen it since.

[Tim Wilson]: I don't know what came first.

[Tim Wilson]: I wound up arriving at a spot where I said, showing somebody who says, I want to have a deeper understanding.

[Tim Wilson]: I don't know that this is full on a marketer, but it certainly could be an analyst and just showing distributions and saying, if you're showing [Tim Wilson]: normal distributions with different variances and different means, or the same variance in different means, are showing two different examples of here's a case where your example of $59 to $61 and $79 to $80, that's a really tight distribution.

[Tim Wilson]: It does seem like you can visually help [Tim Wilson]: someone at least understand the nature of the variability so that when they go and interact with the statistician or the data scientist, there's a more productive conversation.

[Tim Wilson]: And it probably also injects in the same [Tim Wilson]: the anxiety I've been living with now for seven years.

[Tim Wilson]: I don't know anything.

[Tim Wilson]: I can punch in and run the linear regression, but I am absolutely convinced that there's something totally wrong with it.

[Tim Wilson]: I think there's a case where knowing [Tim Wilson]: Developing some of the intuition without getting all the way to I'm picking the right method and the interpretation of that correctly Still has value where I get terrified is people just looking at a Chart and not even having any any under any any intuition about why if they see $90 $80 and $70 they can't just make a declarative statement about the difference in those [Tim Wilson]: groups.

[Tim Wilson]: Sorry, I don't know why now is the time for me to square that circle.

[Chelsea Parlett-Pelleriti]: Thank you for sharing.

[Chelsea Parlett-Pelleriti]: I think that's a valid fear.

[Chelsea Parlett-Pelleriti]: That fear hasn't gone away for me yet, that I'm doing something wrong.

[Chelsea Parlett-Pelleriti]: There's something that I'm not thinking about that makes this not ideal.

[Tim Wilson]: Well, now I'm really, I give up.

[Tim Wilson]: It's time I'm going to go get a greener and a rose now.

[Chelsea Parlett-Pelleriti]: Maybe you should.

[Chelsea Parlett-Pelleriti]: There's probably lots of more qualified people that are like, oh, I'm past that.

[Chelsea Parlett-Pelleriti]: Chelsea's just not at that stage yet.

[Chelsea Parlett-Pelleriti]: But I do think it's healthy to spear that.

[Tim Wilson]: You have freaking BHD in statistics.

[Chelsea Parlett-Pelleriti]: Yeah.

[Chelsea Parlett-Pelleriti]: I mean, didn't help too much.

[Chelsea Parlett-Pelleriti]: It actually made it worse in some ways.

[Chelsea Parlett-Pelleriti]: I thought I knew so much about statistics back when I was learning the ANOVA.

[Chelsea Parlett-Pelleriti]: And now I go, uh-oh, I really know only a very little bit of statistics.

[Chelsea Parlett-Pelleriti]: But I will say that fear [Chelsea Parlett-Pelleriti]: I think is a really good motivator to have the conversations we're having about the assumptions of an ANOVA, what you're actually getting out of an ANOVA.

[Chelsea Parlett-Pelleriti]: And I think that's really important.

[Chelsea Parlett-Pelleriti]: And I will say, I had this thought when you were talking of we as statisticians or whoever it is who's putting out all this material on ANOVA are not always good about [Chelsea Parlett-Pelleriti]: talking about the real-world applications of these tools, for instance, you may often hear with linear regression with t-test with ANOVA's, oh, it's robust to violations of this assumption.

[Chelsea Parlett-Pelleriti]: That's true, but we don't really talk about that well.

[Chelsea Parlett-Pelleriti]: I think it can lead to this thing where it's like, okay, technically, there's an assumption of normality for t-test and for ANOVA, but we don't really talk about [Chelsea Parlett-Pelleriti]: Okay, what happens when you violate it?

[Chelsea Parlett-Pelleriti]: And that usually ends up going one of two ways, which is like people care way too much about that assumption.

[Chelsea Parlett-Pelleriti]: And they're like, oh no, my, you know, Kolmogorov, Smirnov test is insignificant or whatever.

[Chelsea Parlett-Pelleriti]: And they care way too much about it when it is robust, the inferences you make are robust, or they go the opposite and people go, oh, it's robust.

[Chelsea Parlett-Pelleriti]: I don't care about it.

[Chelsea Parlett-Pelleriti]: You go, no, no, no, no, no.

[Chelsea Parlett-Pelleriti]: It's robust to minor violations of this.

[Chelsea Parlett-Pelleriti]: And so I think it does make the waters really muddy.

[Chelsea Parlett-Pelleriti]: If you were trying to decide, am I going to use a t-test?

[Chelsea Parlett-Pelleriti]: Am I going to use an ANOVA?

[Chelsea Parlett-Pelleriti]: Am I going to use a nonparametric method to analyze my form?

[Chelsea Parlett-Pelleriti]: core group experiment model that I did.

[Chelsea Parlett-Pelleriti]: It makes it really hard to figure out what should you actually do, because it's not always clearly communicated what the pitfalls are.

[Chelsea Parlett-Pelleriti]: So to validate your fears, you should be fearful, but also people aren't really doing what they could do to help make it easy.

[Michael Helbling]: And with that, we probably need to start to wrap up.

[Michael Helbling]: No.

[Michael Helbling]: Wait, what were you going to say, Joy?

[Chelsea Parlett-Pelleriti]: I have so many questions.

[Chelsea Parlett-Pelleriti]: You should ask them.

[Chelsea Parlett-Pelleriti]: I do have something to share that I should have shared at the top, which is, as you know, as I've talked about a million times, I got my start in psychology.

[Chelsea Parlett-Pelleriti]: So while I don't use ANOVA that much in my day-to-day life, I have a soft spot because it was in the intro stats classes that made me fall in love with statistics.

[Chelsea Parlett-Pelleriti]: I love it so much.

[Chelsea Parlett-Pelleriti]: My dog is named after.

[Chelsea Parlett-Pelleriti]: and a Nova.

[Chelsea Parlett-Pelleriti]: Her name is Nova, so she's a analysis of variants, I guess.

[Tim Wilson]: Which I think we got after we stopped recording last time.

[Tim Wilson]: So that's Michael's fault that we didn't manage to insert that for you.

[Chelsea Parlett-Pelleriti]: So it was very apt for me to be the guest here because I love it so much.

[Julie Hoyer]: Are you actually going to let me ask the last question, Michael?

[Michael Helbling]: Well, I've got a lot of noise happening on my end, so yeah, go ahead.

[Julie Hoyer]: I just wanted to, and this is probably a little bit of a can of worms to be ending on.

[Tim Wilson]: You know what, Moee is not here, so you are just slipping right in.

[Julie Hoyer]: I have to take this honor and carry the baton for Moee.

[Julie Hoyer]: We talked a lot about covariates, which means it would be an encova.

[Julie Hoyer]: But then you talked about using an ennova, understanding the question that it's actually answering is that variation is explainable by the category you chose somewhere across these categories or across the category.

[Julie Hoyer]: And then you said you can follow it up with responsible post-hoc analysis.

[Julie Hoyer]: And we never really talked about a little bit of covariates or post-hoc.

[Julie Hoyer]: Which way do you go?

[Julie Hoyer]: And just the way you were talking about using ANOVA's in practice, do you tend to lean towards one of those options instead of just a pure ANOVA?

[Chelsea Parlett-Pelleriti]: Yeah.

[Chelsea Parlett-Pelleriti]: In complete transparency, I do not use these a ton in my daily life, but when I have mostly back in my psych research days, it's not an either or.

[Chelsea Parlett-Pelleriti]: It's a, what question am I asking?

[Chelsea Parlett-Pelleriti]: Because when you do a post-hoc test, what you're usually doing is something like, okay, I had four campaigns, I want to know which [Chelsea Parlett-Pelleriti]: two are different or like which ones are different.

[Chelsea Parlett-Pelleriti]: And so post hoc tests can help you answer that, but you still could have like covariates in that that are soaking up that variance.

[Chelsea Parlett-Pelleriti]: So it's sort of a separate question of like, do I want to include covariates to partition that variance as a novice or want to do or not or and do I care about these pairwise comparisons?

[Chelsea Parlett-Pelleriti]: Like if I have more than two groups, [Chelsea Parlett-Pelleriti]: Do I care which ones are different?

[Chelsea Parlett-Pelleriti]: And honestly, I'm sure there are some out there, but I really struggle to think of a question where you...

[Chelsea Parlett-Pelleriti]: be better to use the omnibus F test that there is some difference somewhere in here versus most people have questions and most people are going to action on those post-hoc.

[Chelsea Parlett-Pelleriti]: I can't imagine many scenarios where you'd want to do some type of ANOVA or in the ANOVA family and not want to follow that up with post-hoc tests.

[Chelsea Parlett-Pelleriti]: Some might argue, [Chelsea Parlett-Pelleriti]: you should just start with those postdoc tests and control your family-wise error rate.

[Chelsea Parlett-Pelleriti]: But in any case, I think it's a very important part of like actually gaining actionable insight from the Innova.

[Tim Wilson]: Gotcha.

[Tim Wilson]: That does seem like that's kind of the weird.

[Tim Wilson]: Also back when I was trying to get some intuition around it and I found myself going down the, and then you'll need to do a postdoc and the 2K postdoc is the most common.

[Tim Wilson]: And I feel like I wound up in the [Tim Wilson]: the same spot like if you're always gonna post hoc just feels like you're like, ah, I did this thing.

[Tim Wilson]: And then I'll kind of do this other thing.

[Tim Wilson]: You're like, well, if you're always going to do that other thing, it's somehow it has this Latin phrase on it as though it's like this kind of incidental tack on, but you're almost always going to use it does feel kind of, [Tim Wilson]: weird.

[Julie Hoyer]: I have to do the ANOVA before you do the push post.

[Julie Hoyer]: Now this is the can of worms.

[Chelsea Parlett-Pelleriti]: This is like what I was taught.

[Chelsea Parlett-Pelleriti]: This is what I was taught is sort of like you do the omnibus test.

[Chelsea Parlett-Pelleriti]: And if the omnibus test is significant, it tells you something's going on in there.

[Chelsea Parlett-Pelleriti]: So you throw it out.

[Chelsea Parlett-Pelleriti]: I don't know that that's widely agreed upon as the appropriate way to control your error rate.

[Chelsea Parlett-Pelleriti]: In fact, [Chelsea Parlett-Pelleriti]: I might be wrong, but I feel like I've heard that might be overly conservative, especially if you're also correcting for your family-wise error rate in your post-talk test.

[Chelsea Parlett-Pelleriti]: So I would say my current recommendation, ask me next time we talk about an OVAZ.

[Chelsea Parlett-Pelleriti]: My current recommendation is be thoughtful about the post-talk comparisons you're doing.

[Chelsea Parlett-Pelleriti]: So if you don't need to do all 10 groups compared to all of the other groups, [Chelsea Parlett-Pelleriti]: don't, and then use some type of like a Bonferroni correction, a Psyduck correction, the Tukey HSD that Tim was talking about, and just correct for your family-wise error, right?

[Chelsea Parlett-Pelleriti]: There's lots of arguments about what counts as a family and what you should correct for, but we'll save that for the next episode I'm on.

[Michael Helbling]: That's right.

[Michael Helbling]: That's for analytics power hour after hours.

[Tim Wilson]: For the analytics hour plus listeners, they can get access to there.

[Michael Helbling]: All right.

[Michael Helbling]: Well, before we start to fully wrap up, we do want to go around and share a last call, something that might be of interest.

[Michael Helbling]: Chelsea, do you have something you'd like to share as a last call?

[Chelsea Parlett-Pelleriti]: I do.

[Chelsea Parlett-Pelleriti]: It's a little out there, but it does relate to statistics and machine learning.

[Chelsea Parlett-Pelleriti]: You may have seen the movie Project Hail Mary is coming out soon based on one of the books that I thought was one of the best books I read years ago when I read it.

[Chelsea Parlett-Pelleriti]: It's by the Andy Ware, I think is how you say that from The Martian.

[Chelsea Parlett-Pelleriti]: And it is not only an excellent book, and apparently might be an excellent movie with Ryan Gosling if you're into that.

[Chelsea Parlett-Pelleriti]: But the reason I'm recommending it here that is sort of related to statistics [Chelsea Parlett-Pelleriti]: is I actually read a section of this in my stats classes or my machine learning classes because they have this really beautiful scene.

[Chelsea Parlett-Pelleriti]: I don't want to give any spoilers because it is quite a bit into the book where they're doing something.

[Chelsea Parlett-Pelleriti]: Science won't go into it.

[Chelsea Parlett-Pelleriti]: And they have this beautiful explanation of someone goes, did you use artificial intelligence to do this?

[Chelsea Parlett-Pelleriti]: And the person says, no, we have to be able to test it in thousands of ways and know exactly how it responds and why we can't do that with a neural network.

[Chelsea Parlett-Pelleriti]: And I thought that was just such a great explanation in the context of the book.

[Chelsea Parlett-Pelleriti]: You'll have to read the book of why machine learning and some of the black box methods can be a little tough to swallow for some people.

[Chelsea Parlett-Pelleriti]: So for both statistical and literary reasons, highly recommend both the book and the upcoming movie, Project Hail Mary.

[Tim Wilson]: Nice.

[Tim Wilson]: I loved that book.

[Tim Wilson]: My sister gave me that book for Christmas a couple of years ago.

[Tim Wilson]: And I didn't realize it was that he wrote the Martian until like after I was like, I gotta read something else by this guy.

[Tim Wilson]: I was like, oh, he also wrote the Martian.

[Tim Wilson]: So.

[Michael Helbling]: Nice.

[Michael Helbling]: All right, Tim, what about you?

[Michael Helbling]: What's your last call?

[Tim Wilson]: So mine is a post.

[Tim Wilson]: There are times where I feel like I'm going back to the same wells, but usually when Jason Packer writes something, it is entertaining and really thoughtful.

[Tim Wilson]: And he, along with Yuliana Jackson, wrote a post called The Duality of ChatGPT.

[Tim Wilson]: And the premise is kind of we [Tim Wilson]: We make these, there's like two sides on multiple dimensions around discussing AI, like AI will write our code and do analyses for us and or AI produces slap and won't make our jobs easier.

[Tim Wilson]: And he just gets kind of thoughtful and has like, [Tim Wilson]: hilarious references slips in like a John Lennon reference that is actually just making a joke of a list.

[Tim Wilson]: But it's a good read where he walks around the duality interest to square the circle in each case.

[Tim Wilson]: I'm hooked on people who are not completely in the bag for AI and are also not completely [Tim Wilson]: anti-AI and his was, I did actually, I physically grinned.

[Tim Wilson]: I don't know that I laughed out loud, but I was definitely smirking while reading it.

[Michael Helbling]: All right, Julie, what about you?

[Michael Helbling]: What do you got?

[Julie Hoyer]: Mine is very off topic and just something I enjoy, not related to the industry literally at all, but I hope one of you listeners maybe are looking for this type of app and I hope you enjoy it as much as I have.

[Julie Hoyer]: It's called The Short Ears and I had such a fear when I was having my daughter a few years ago.

[Julie Hoyer]: I was like, how the heck do people work a full-time job, have a child and keep up with a baby book?

[Julie Hoyer]: But I was also like, I want to remember these things.

[Julie Hoyer]: I want to have pictures.

[Julie Hoyer]: I want to do the baby book thing.

[Julie Hoyer]: So I was on the hunt for an answer to that problem.

[Julie Hoyer]: And the short ears has been amazing.

[Julie Hoyer]: It's just an app on your phone and it can give you daily questions.

[Julie Hoyer]: And so as I would lay in bed at night, I could just [Julie Hoyer]: go through and be like, oh, here are three questions for you lately about your kid.

[Julie Hoyer]: And you can upload photos, upload videos.

[Julie Hoyer]: And then as you finish chapters, they just mail them to you.

[Julie Hoyer]: You've bought and then you buy the book.

[Julie Hoyer]: But you don't even have to buy the book or pay for anything before you start entering photos and information.

[Julie Hoyer]: So you could just go along and be like, [Julie Hoyer]: Okay, I've really stuck with this.

[Julie Hoyer]: I'm six months in.

[Julie Hoyer]: I'm going to order the book now.

[Julie Hoyer]: They send you the chapters.

[Julie Hoyer]: You put them in the book and then you can even extend it to the toddler years, which I think I'm going to do.

[Julie Hoyer]: But again, I've just been able to like keep up in the app and then I can decide to purchase or not.

[Julie Hoyer]: And I also feel like then any subsequent kids, you can keep up with this so you don't have like the first child got it all and the next kids got nothing.

[Julie Hoyer]: Like I feel like this could help.

[Julie Hoyer]: So if you have any anxiety about baby books, I really love the short years.

[Michael Helbling]: That's just funny because I was literally going to be like, yeah, that second kid.

[Michael Helbling]: Yeah, not so many details.

[Tim Wilson]: I mean, I had to go through as soon as you said like the short years, like the former D1 volleyball player, I was like, and what were the short years for you, Julie?

[Tim Wilson]: Was that like zero to 18 months at which point?

[Tim Wilson]: So.

[Michael Helbling]: Well, my last call is, I was going to ask.

[Michael Helbling]: OK, please.

[Michael Helbling]: Michael, what's your last call?

[Michael Helbling]: So glad you asked.

[Michael Helbling]: Mine is also AI related, because it seems like it's dominating everything we do.

[Michael Helbling]: But Anthropic ran a little experiment recently with an AI agent that they put in charge of a little shop in their office.

[Michael Helbling]: And they basically gave it instructions to try to buy things and sell things to the people that worked in the office that would help it make money.

[Michael Helbling]: And then they wrote up the results.

[Michael Helbling]: It did a terrible job.

[Michael Helbling]: And it's kind of cute what it was trying to do and kind of funny.

[Michael Helbling]: But it just goes to show you that the level of complexity we can achieve with AI agents is not quite ready to replace us all yet.

[Michael Helbling]: But it's kind of a fun read.

[Michael Helbling]: It kind of dives into a little bit of the details of what the AI was trying to accomplish and things like that and where it went wrong and what it did right.

[Michael Helbling]: And they're going to keep running that experiment.

[Michael Helbling]: I think they're working with an AI safety company as well all on that project.

[Michael Helbling]: So kind of interesting.

[Michael Helbling]: OK, Chelsea, who knew that a TikTok about Moente Carlo simulations would lead to all this?

[Michael Helbling]: The one thing that we can say TikTok was good for way back in the day, but thank you so much.

[Michael Helbling]: It's incredible.

[Michael Helbling]: I don't know why this is, but statistics, statistical concepts, [Michael Helbling]: Well, feel hard to grab onto you for us, Mayor Moertals.

[Michael Helbling]: And you're a very unique and special person.

[Michael Helbling]: And I hope people recognize that all the time in the way that you're able to like bring those concepts to life.

[Michael Helbling]: So I just really want to say thank you very much.

[Chelsea Parlett-Pelleriti]: Thank you.

[Chelsea Parlett-Pelleriti]: You're thanking me by continuing to have me back on over and over and over at your podcast.

[Michael Helbling]: Yeah, we're going to see.

[Michael Helbling]: Secretly, we're just going to shift the whole show like over just be like, oh yeah, now here's the statistical power hour.

[Chelsea Parlett-Pelleriti]: Statistical significance and why you shouldn't ignore non statistically significant lift tests.

[Julie Hoyer]: Ooh, that sounds so good.

[Michael Helbling]: That sounds great.

[Michael Helbling]: Give us 10 episodes.

[Tim Wilson]: Look, I've been on three times.

[Tim Wilson]: I'm gonna tell you fuckers what you really need to talk about.

[Chelsea Parlett-Pelleriti]: Exactly.

[Tim Wilson]: Listen.

[Chelsea Parlett-Pelleriti]: I just don't want to talk about it anymore.

[Michael Helbling]: No, that's totally fair.

[Michael Helbling]: Well, and this whole show came about because our listeners wanted more topics like that.

[Michael Helbling]: And as you're listening, maybe there are other things you're like, please bring back Chelsea to talk about this.

[Michael Helbling]: We'd love to hear from you.

[Michael Helbling]: So reach out, let us know.

[Michael Helbling]: And you can do that on our LinkedIn or on the Measures Slack chat group or on via email at contact at analyticshour.io.

[Michael Helbling]: So we'd love to hear from you.

[Michael Helbling]: Yeah, I mean, this is awesome, really great, and I guess we're going to ramp this up.

[Michael Helbling]: And I think I speak for both of my co-hosts, whether it's Sinova, a t-test, or an ANCOVA.

[Michael Helbling]: Keep analyzing.

[Announcer]: Thanks for listening.

[Announcer]: Let's keep the conversation going with your comments, suggestions, and questions on Twitter at @analyticshour on the web at analyticshour.io, our LinkedIn group, and the Measure Chat Slack group.

[Announcer]: Music for the podcast by Josh Grohurst.

[Charles Barkley]: Those smart guys wanted to fit in, so they made up a term called analytics.

[Charles Barkley]: Analytics don't work.

[Charles Barkley]: Do the analytics say go for it, no matter who's going for it?

[Charles Barkley]: So if you and I were on the field, the analytics say go for it.

[Charles Barkley]: It's the stupidest, laziest, lamest thing I've ever heard for reasoning in competition.

[Chelsea Parlett-Pelleriti]: I don't think you ever said the title, ANOVA.

[Michael Helbling]: ANOVA?

[Chelsea Parlett-Pelleriti]: I'm hardly knowing it.

[Michael Helbling]: Yeah, the title is sort of like a thing we don't actually say.

[Michael Helbling]: And Ian, sometimes guests have come on and then they just talk about something completely off the wall.

[Michael Helbling]: And then we change the title again.

[Michael Helbling]: Not off the wall, but like completely different action.

[Tim Wilson]: That's an option?

[Chelsea Parlett-Pelleriti]: What the hell?

[Chelsea Parlett-Pelleriti]: I didn't know I could go without it.

[Chelsea Parlett-Pelleriti]: Wait, I can go rogue and you'll just completely reconfigure?

[Chelsea Parlett-Pelleriti]: So it looks intentional?

[Chelsea Parlett-Pelleriti]: Good to know.

[Michael Helbling]: Yes, we will.

[Michael Helbling]: Absolutely.

[Michael Helbling]: Basically, the process of the show, Chelsea, just so you understand it, is Tim comes up with the titles and then I just say whatever I want in the intro.

[Julie Hoyer]: And then Tim reconfigures the titles if they don't match up.

[Michael Helbling]: Yeah, and if they're totally different, like, I just go off into left field, Tim will be like, let's make the title.

[Michael Helbling]: First off, there's a committee for this, Julie, so now I'm just kidding.

[Tim Wilson]: It's like a really poor implementation, because it can't complete a sentence and kind of goes all over the place and doesn't make sense.

[Tim Wilson]: Or it's like, no, that's like the best that's ever made.

[Julie Hoyer]: Representation.

[Michael Helbling]: That's right, that's like presentation.

[Michael Helbling]: Got Tim to a team.

[Tim Wilson]: Rock flag and it's linear models all the way down.

[Michael Helbling]: That's right.

[Michael Helbling]: I saved that one for you, Tim, because I had an inkling you might do that.

[Chelsea Parlett-Pelleriti]: That's all I had.

[Chelsea Parlett-Pelleriti]: I love that choice.

[Chelsea Parlett-Pelleriti]: It's so good.