[Announcer]: Welcome to the Analytics Power Hour.

[Announcer]: Analytics topics covered conversationally and sometimes with explicit language.

[Tim Wilson]: Hi, everyone.

[Tim Wilson]: Welcome to the Analytics Power Hour.

[Tim Wilson]: This is episode number 283.

[Tim Wilson]: And it's the show where finally, finally, we're going to answer the question.

[Tim Wilson]: Does size matter?

[Tim Wilson]: I mean, data set size, that is.

[Tim Wilson]: I felt like the explicit rating we have for this show made it safe for me to make that joke, but Prudence suggests that maybe I shouldn't take it any farther than that.

[Tim Wilson]: So for the past 15 years or so, the business and analytics worlds have been obsessed with big data, collecting it, storing it, and deploying increasingly sophisticated models and techniques to glean value from it.

[Tim Wilson]: Some people have even gone back to school to learn more about it.

[Tim Wilson]: And we'll get to that in a little bit.

[Tim Wilson]: Yet arguably, many businesses are awash in small data, small and mid-sized businesses and many nonprofits, for instance.

[Tim Wilson]: As podcast listener Barrett Smith put it way back in 2024, when he proposed this very topic, quote, what are the right analytic tools for small organizations to use on the data they have to make decisions?

[Tim Wilson]: How do we, as analysts, help these organizations be as data-focused as the big orgs?" [Tim Wilson]: So what can we do with data measured more in kilobytes rather than terabytes, or petabytes, or exabytes, or even yodabytes, something I learned as I was prepping for this very show?

[Tim Wilson]: It's fun to say, though, a yodabyte.

[Tim Wilson]: Simply deriding it for its laughable, teeny weeniness seems like a missed opportunity.

[Tim Wilson]: So let's talk about it.

[Tim Wilson]: So I'm joined for this little discussion about small data by my co-hosts Moee Kiss and Julie Hoyer.

[Tim Wilson]: Welcome to both of you.

[Moe Kiss]: Hey, hey, howdy.

[Tim Wilson]: That's coming in, coming in strong with the enthusiasm, trying out the throwing it to two people at the same time and seeing who will try to out polite the other one.

[Moe Kiss]: Yeah, see if we fight for the mic.

[Tim Wilson]: Yeah, neither one of you is used to talking over people like I am right now, as I am talking over you right now.

[Tim Wilson]: Oh, good lord, we have a problem.

[Tim Wilson]: But of course, we do love to bring on a guest who's put some thought into whatever topic we're covering.

[Tim Wilson]: And that actually proved to be a little challenging given the nature of this topic.

[Tim Wilson]: So when Barrett proposed this topic, we're like, that's a great topic.

[Tim Wilson]: Who can we have on to talk about it?

[Tim Wilson]: and outside of just us trying to riff on it.

[Tim Wilson]: And so it just sat there for over a year with us pondering it until we saw that Joe Domoreski had written a medium post titled, How to be Data Driven in Marketing, even if your small business doesn't have a lot of data.

[Tim Wilson]: So we pretty much picked up the phone and reached right out to him.

[Tim Wilson]: Joe is the owner of Country Fried Creative, which is a full-service creative digital marketing agency serving the Metro Atlanta area.

[Tim Wilson]: He regularly publishes pretty great content on Medium in the marketing data science with Joe Domoleschi publication.

[Tim Wilson]: I've actually used this content as a last call at least once on the show.

[Tim Wilson]: Welcome to the show, Joe.

[Tim Wilson]: Well, thanks, Tim.

[Tim Wilson]: Howdy, y'all.

[Tim Wilson]: Hello from Atlanta.

[Tim Wilson]: You are a Southern native as polite by your first words spoken on this podcast.

[Tim Wilson]: I take it.

[Joe Domaleski]: Absolutely.

[Joe Domaleski]: I like to tell people I'm a Southern pollock.

[Tim Wilson]: Okay.

[Tim Wilson]: It feels like the Southern has kind of taken over the whatever the Polish lineage there is.

[Tim Wilson]: I think so.

[Tim Wilson]: A little bit.

[Tim Wilson]: We're excited to have you.

[Tim Wilson]: We're excited to have this discussion.

[Tim Wilson]: Maybe we can sort of kick things off by Joe having you tell us kind of what prompted you to write like an article digging pretty deeply into the challenges for businesses that don't have like so-called big data to work with.

[Joe Domaleski]: Absolutely.

[Joe Domaleski]: And before I jump into that, just want to say it's really great to be on this podcast.

[Joe Domaleski]: first indirectly met Tim, reading his book, and Tim needs to tell the listening audience what my claim to fame is with your book.

[Joe Domaleski]: I was the first to find a mistake.

[Joe Domaleski]: That's how much I read it.

[Joe Domaleski]: And we won't say what the mistake is, but I'm also friends with his co-author, Joe Sutherland.

[Tim Wilson]: I would suggest that anybody who would love to see if they could find the mistake, if you go to analyticstrw.com or amazon.com or target.com or walmart.com and search for analytics the right way, you can get your own copy of the book and see if you too can find the error that Joe made.

[Joe Domaleski]: It's a great book, though.

[Joe Domaleski]: All kidding aside, enjoy this podcast.

[Joe Domaleski]: I'm normally listening to this podcast, Walking My Dog at 5.30 in the morning, so getting educated and enjoying the [Joe Domaleski]: All the great guests you had so it's an honor to be here.

[Joe Domaleski]: What led me to writing about small data?

[Joe Domaleski]: I've been a small business owner for 22 years.

[Joe Domaleski]: As we'll talk about in the show, everything skews toward big data and big algorithms.

[Joe Domaleski]: What about the little guy, the little person who just doesn't have a lot of data?

[Joe Domaleski]: Quite honestly, in my blog, a lot of the things that I like to write about are topics that are either not covered a lot or they're not covered in a way that [Joe Domaleski]: is kind of relevant to a small business.

[Joe Domaleski]: As a small business myself, I also wrestle with the problem of small data.

[Tim Wilson]: I have strong thoughts on this.

[Tim Wilson]: Do you have clients who are coming to you saying, can you run machine learning algorithms?

[Tim Wilson]: They're thinking that they need you and your team to [Tim Wilson]: dive into their data and you're like, guys, your email list is like 150 people.

[Tim Wilson]: That's not...

[Tim Wilson]: not going to be workable?

[Tim Wilson]: Are you running into that in a day-to-day basis, or is it more controlled by the agency saying you're using what you can to help serve the clients with whatever they already have?

[Joe Domaleski]: We are so far down on the spectrum.

[Joe Domaleski]: Let me reframe that question, Tim and Julie and Moe.

[Joe Domaleski]: In some cases, I am literally doing battle [Joe Domaleski]: on some very fundamental issues of I don't believe in marketing or marketing is the same thing as sales or the only thing that matters is revenue or what's a dashboard which I know is a favorite topic of all of yours so in many cases [Joe Domaleski]: Even getting somebody to understand what data is, is kind of fundamental.

[Joe Domaleski]: And I'm kind of late to the game.

[Joe Domaleski]: When I started the business, we started as a web design company.

[Joe Domaleski]: And of course, with 22 years of history, we've seen things like social media, search engine optimization, email marketing, and all the things we think about with digital marketing come on their own.

[Joe Domaleski]: And it used to be that we could sell on sex appeal, honestly.

[Joe Domaleski]: Oh, I need a new website.

[Joe Domaleski]: Make it pretty.

[Joe Domaleski]: They didn't care about analytics.

[Joe Domaleski]: I've seen in the last five years in the small business space, and there's a lot of different definitions of small businesses.

[Joe Domaleski]: Let's just say 10 million in lower in sales, just so people can kind of get a picture.

[Joe Domaleski]: they're not thinking about these things.

[Joe Domaleski]: In some cases, I'm actually having to educate them on the fundamentals.

[Joe Domaleski]: Moest of them don't even know what machine learning is, Tim.

[Moe Kiss]: Then do you see that the problem that they're facing is fundamentally, there might be the first step, which is like, [Moe Kiss]: they're not using data at all, or there's a fundamental misunderstanding or trust.

[Moe Kiss]: Do you also see the next step, which is, are they making the best decisions they can with what they've got?

[Moe Kiss]: Is that the evolution that you have to go through?

[Joe Domaleski]: Moe, when you're talking to very small businesses, [Joe Domaleski]: Many of them don't have a marketing department, or they have a one-person marketing department, or they're outsourcing it, which, of course, I'm happy with that.

[Joe Domaleski]: We'd love to be their marketing department.

[Joe Domaleski]: I think the first step is really, and this is to analytics pros like all of you who are way on the other side of the analytics knowledge spectrum, [Joe Domaleski]: you know, cliches like you can't manage what you can't measure.

[Joe Domaleski]: You know, we've heard that zillions of time, but it's fundamental to, you know, even the smallest of organizations.

[Joe Domaleski]: And so, you know, I think for many small businesses, it's that education part that, yes, marketing can be measured, which sounds fundamental and self-evident, and yet so many small businesses don't even think about that.

[Tim Wilson]: But this is maybe going to take a little bit of a turn because now I'm fascinated.

[Tim Wilson]: Take email marketing because small businesses are running on, they have limited, they don't have a marketing department.

[Tim Wilson]: They don't have often dollars sloshing around that they can say, directionally, it has to be good enough.

[Tim Wilson]: My experience with consultants who do analytics support or digital support for small businesses and a little bit of my own experience is they're being really, really, they're like, we only have, we're spending $4,000 a month.

[Tim Wilson]: And should we put that in Google, in Facebook, or [Tim Wilson]: in an email marketing?

[Tim Wilson]: Like, are they not coming saying, we want to spend as little as possible with you and when you do something, if you do an email marketing campaign for me, tell me whether it worked or not?

[Tim Wilson]: Like, are they asking that question or are they just saying, you're a line item?

[Joe Domaleski]: Yeah, some are.

[Joe Domaleski]: And then others, we've got this concept of a minimally viable product, right?

[Joe Domaleski]: We've all heard of that.

[Joe Domaleski]: A couple of weeks, a couple of months ago, I don't know, they all ran together.

[Joe Domaleski]: I wrote an article, I applied that and I called it minimally viable marketing.

[Joe Domaleski]: And so, you know, what I tell people is there is a minimum level [Joe Domaleski]: It's kind of your basal metabolic rate, if you will, of just, I need to have some type of presence.

[Joe Domaleski]: That'll be a website and social media and some other things like that.

[Joe Domaleski]: Layered on top of that, of course, are all the different things one might do with marketing.

[Joe Domaleski]: What I've seen for a lot of [Joe Domaleski]: Executives, which may just be one person, the business owner or a small management team, is there really isn't awareness of the numbers they already have.

[Joe Domaleski]: You guys already know this, but a lot of small business.

[Joe Domaleski]: You mean you can tell how many people went to my website?

[Joe Domaleski]: I mean, you would be shocked, maybe not, that many people don't even fund it.

[Tim Wilson]: I give you a note, Rick.

[Tim Wilson]: It's not Rick.

[Joe Domaleski]: They got rid of my link, Chowner.

[Joe Domaleski]: Why did they get rid of link counters?

[Joe Domaleski]: And I don't know how many people go to my site.

[Joe Domaleski]: You know, that sort of thing.

[Joe Domaleski]: And so a lot of times it's just uncovering what they already have.

[Joe Domaleski]: You know, when you send that email, you're actually collecting data without even knowing.

[Joe Domaleski]: Oh, really?

[Joe Domaleski]: So it's just getting our arms around that.

[Moe Kiss]: So talk to us.

[Moe Kiss]: Once these companies kind of start down the path of the minimum...

Oh, geez.

[Moe Kiss]: minimal viable marketing.

[Moe Kiss]: There you go.

[Moe Kiss]: They're starting to have this small data set.

[Moe Kiss]: What are some of the techniques that you work with them on so that they can use the data they've got to inform their decisions?

[Joe Domaleski]: Yeah.

[Joe Domaleski]: I think, Moe, once I can get a client past that initial barrier that marketing is different than sales, [Joe Domaleski]: Yes, I need to track certain things.

[Joe Domaleski]: Then we start to look at things related to the quality of the data.

[Joe Domaleski]: the volume of the data, and when I'm talking to many of...

And I'm still very much involved.

[Joe Domaleski]: I have 12 employees, which makes my company bigger than two people, but I'm not a 50 or 100-person agency.

[Joe Domaleski]: We've got a nice little niche.

[Joe Domaleski]: I still do a lot of the selling involved in our services.

[Joe Domaleski]: Rarely do I see data or analytics be the lead.

[Joe Domaleski]: Normally, we're going to start a campaign and try to do some more awareness of that.

[Joe Domaleski]: Normally, it is in response to a specific pain point where that is involved, but the client's not aware of it.

[Joe Domaleski]: But once we engage a client, [Joe Domaleski]: I think the first step for us is to educate the client on the data they already have, and they don't even know it.

[Joe Domaleski]: Then we can start to look at the volume, the quality, and some of the things that I talk about in the article.

[Joe Domaleski]: I normally like to pick one thing to focus on.

[Joe Domaleski]: to try to prove the point.

[Joe Domaleski]: It could be the email open rate, if it's a nonprofit, you know, a conversion rate, if it is a social media manager, you know, maybe it's the reach, you know, very fundamental things like that.

[Joe Domaleski]: And I think once there's recognition on the part of a client, [Joe Domaleski]: Hey, there is this thing called data and we actually have some.

[Joe Domaleski]: Then we can start to explore different issues related to that data.

[Julie Hoyer]: Can we actually go back because you started to touch on it and I know you talk about it in your article more in depth.

[Julie Hoyer]: I wanted to chat about this idea of not enough data that you had covered and you rattled off just now.

[Julie Hoyer]: different categories of not having enough data.

[Julie Hoyer]: Because it's interesting, I hadn't even thought of when we were talking obviously like small data sets, I didn't even think of just this broader idea of not enough data.

[Julie Hoyer]: So when you are working with clients and they realize, oh, I have some data, even if they had a large volume of data entry, it sounds like though you run into a lot where they still don't have quote unquote enough data.

[Julie Hoyer]: to what we're used to, like, at larger organizations.

[Joe Domaleski]: Exactly.

[Joe Domaleski]: You know, case in point, a large email list in terms of subscribers, some of the people that we're working with, might be 10,000 people.

[Joe Domaleski]: Moest of them, most of our client base, 2,000.

[Joe Domaleski]: like our local Chamber of Commerce.

[Joe Domaleski]: I'm a former board member of the Chamber, and they're very engaged, and so what we see is perhaps a higher open rate, but we have a lower overall, I think I called it an article, an N, but just the number, the shared number, and when you're dealing with very small, [Joe Domaleski]: But we're working with a nonprofit that literally started up a year ago.

[Joe Domaleski]: They have 100 people on their email list.

[Tim Wilson]: Now, what do you do with that, right?

[Tim Wilson]: But is that the sort of thing that if you say you have 100 and you can talk through the math of saying, imagine if you had 200 or 500 or 1,000.

[Tim Wilson]: So does it go?

[Tim Wilson]: Do you wind up sometimes having a discussion of like, [Tim Wilson]: If marketing matters, then it's who you can reach matters.

[Tim Wilson]: We know how many you can reach with an email.

[Tim Wilson]: Maybe we should consider trying to grow your email.

[Tim Wilson]: list.

[Tim Wilson]: And if we're going to grow the email list, then we're going to need to measure whether we're growing it and what the cost is, or is that not really how the...

Tim, you actually nailed it.

[Joe Domaleski]: And I had this very conversation last week with that nonprofit that just started up.

[Joe Domaleski]: They were focused on getting donations and other marketing goals, which are important.

[Joe Domaleski]: And I told the executive director, I said, [Joe Domaleski]: probably the most important thing you need to do right now is grow that email list because you know the donors are going to come and go [Joe Domaleski]: Who cares if you have five likes on your Instagram posts, but you need to build that marketing database.

[Joe Domaleski]: So even with small data, and even before we try to figure out how to work with it, that is a finding in of itself.

[Joe Domaleski]: We need more people in that database.

[Joe Domaleski]: We can do better marketing.

[Joe Domaleski]: And so there is value there, even with that low volume data set.

[Julie Hoyer]: And then another angle, though, is to say, like, when you're the other example that you brought up, you know, if their list is more like 2,000 people or 10,000 people, like, that's not necessarily tiny sample size, depending on what you're trying to look at, but that's to get better.

[Joe Domaleski]: No, now we're starting to have some decent numbers, yeah.

[Julie Hoyer]: So then, though, you know, do you ever run into they simply have, this is like their email address, I sent them an email, and then they have nothing else about them, because I know [Julie Hoyer]: What can you do with just a few data points, even if it's on 10,000 people?

[Tim Wilson]: Which was one of the things, when you said small data, I always think of the number of rows and Julie's getting to another point that I think- The number of attributes.

[Moe Kiss]: Yeah.

[Tim Wilson]: Yeah, like you talked about in the article, I was like, oh, it could be small data because it's-

[Joe Domaleski]: Yeah.

[Joe Domaleski]: I mean, if we're looking at a classic data frame to use a Python term there, when we say small, yeah, it can either be in terms of features.

[Joe Domaleski]: That brings up a good point too.

[Joe Domaleski]: I was talking to another client of ours, and we were trying to determine the optimal layout [Joe Domaleski]: for a contact form.

[Joe Domaleski]: Now, if you're needing lots of features, then you want to ask everything, right?

[Joe Domaleski]: Fill this thing out.

[Joe Domaleski]: If any of you have ever adopted a dog, the Humane Society is one of our clients here in our neck of the woods.

[Joe Domaleski]: I'm a dog lover, love dogs.

[Joe Domaleski]: Have have low key he's a therapy dog all over our social media and they look at the form to adopt the dog and because we had to put this on the website i mean it.

[Joe Domaleski]: I think it was almost as much as a FAFSA form.

[Joe Domaleski]: So, you know, my kids are grown and gone.

[Joe Domaleski]: Half your audience is like, oh yeah, that FAFSA, that's awful.

[Joe Domaleski]: The rest are like, I don't know what a FAFSA is.

[Joe Domaleski]: Oh, you will.

[Tim Wilson]: That's financial.

[Tim Wilson]: He's pulling the American financial aid form for college.

[Joe Domaleski]: I had to stick one on your mode, the financial aid form.

[Joe Domaleski]: I don't know what the essay stands for.

[Joe Domaleski]: Basically, when your kids go to college, you have to fill out a form to document all your revenue as a parent.

[Joe Domaleski]: And it is, it is, and it's a small business owner.

[Joe Domaleski]: it's like an audit.

[Joe Domaleski]: I mean, it's worse than tax form.

[Joe Domaleski]: Okay, so you fill this thing out, and okay, now we've got 50 things we know about our target market, right?

[Joe Domaleski]: That's too many.

[Joe Domaleski]: But hey, if I just have a name and an email, is that really enough?

[Joe Domaleski]: And so, yeah, sparse, we might not have enough rows, but we may have too much or too few columns.

[Joe Domaleski]: And in most cases, it's too few.

[Moe Kiss]: But then do you also find that the businesses are...

Do they overemphasize [Moe Kiss]: decisions with not enough data.

[Moe Kiss]: I'm just thinking of that form example.

[Moe Kiss]: We either have two fields or we have 200, but we think it's working because people are filling it out.

[Moe Kiss]: Tell me about how they critically evaluate that.

[Moe Kiss]: when it is such a small sample like that's the thing that I mean I feel like I have the inverse problem where people assume that everything is significant because we have so much data.

[Joe Domaleski]: Here's what's interesting in working with small businesses and you know for those listeners that are consultants with a handful of clients you know this this episode is going to resonate with you because you encounter these problems a lot it's not talked about a lot it's not taught about [Joe Domaleski]: in schools and so in many cases Moee we're dealing with and I don't want to make small business owners or small enterprises sound like you know they fell off the back of the truck and they don't know what they're doing uh although you know I've been doing this for 22 years I still don't know what I'm doing I've been making it up as I go along somehow somehow figuring it out but you know in many cases they don't know what they don't know [Joe Domaleski]: So they're literally, you know, go back to the, you know, okay, you're tracking this data, you actually have, you know, we need to look at this.

[Joe Domaleski]: Many of them don't know that, you know, three, fours of the people are abandoning the form.

[Joe Domaleski]: So they're just looking at the forms that come in.

[Joe Domaleski]: Oh, this is great.

[Joe Domaleski]: We had 20 people fill out the form last month.

[Joe Domaleski]: They have no idea that maybe there was 200 that gave up on it.

[Joe Domaleski]: And so part of engaging a client where they're at with the small data is to create awareness that, you know, here's the big picture of what's going on.

[Joe Domaleski]: And I would submit that even the absence of data is kind of a finding in and of itself, right?

[Moe Kiss]: What sounds tricky there is like they're making some like [Moe Kiss]: pretty simple mistakes.

[Moe Kiss]: That's like a really hard conversation to have around whether you're overreacting to small changes, whether you're not actually looking, say, at the data that you're not collecting because folks are dropping off or they're using the wrong metrics.

[Moe Kiss]: How do you start to navigate that conversation without [Moe Kiss]: kind of sounding like a jerk.

[Joe Domaleski]: You know, it's not easy.

[Joe Domaleski]: And I, you know, would say based on some of the interactions you guys have had with some of your other guests and just what you all do for a living, right?

[Joe Domaleski]: In analytics, whether you're talking, whether I'm talking to a peer, a fellow small business owner who I can appeal to owner to owner and say, you know, look, [Joe Domaleski]: I understand where you're at.

[Joe Domaleski]: Let me help you.

[Joe Domaleski]: Here's what we do.

[Joe Domaleski]: We take our own medicine, or maybe you're presenting [Joe Domaleski]: It's kind of like emperor's new clothes, right?

[Joe Domaleski]: You're talking to the big boss and they don't have any clothes on.

[Joe Domaleski]: And you're just trying to get them to have a fundamental grasp of something, you know, very basic.

[Joe Domaleski]: And so you do have to approach it delicately.

[Joe Domaleski]: One thing that I've tried to do, and I guess this is the best place to insert this than anywhere else is, you know, I decided [Joe Domaleski]: almost two years ago at the, you know, tender age of 57, to go back to graduate school.

[Joe Domaleski]: And I am currently a master of science and analytics student at Georgia Tech.

[Joe Domaleski]: Part of the reason I am doing that, number one, is so that I get educated, not on the small data, but, you know, on all the cool machine learning things.

[Joe Domaleski]: But the other aspect of it is to have that brand [Joe Domaleski]: where when I'm speaking about analytics, I actually have some academic credibility.

[Joe Domaleski]: It's not just [Joe Domaleski]: Hey, I've got gray hair.

[Joe Domaleski]: I've been doing this for a while, but I can actually, you know, legitimately look somebody in the eyes and say, look, I'm literally studying, you know, state of the art stuff here.

[Joe Domaleski]: You don't have to do all these things, but at a minimum, you need to be doing X, Y, and Z.

And so that really kind of served as the basis for writing that article, because even in grad school, [Joe Domaleski]: everything is large data set.

[Joe Domaleski]: It's like, hey, what about me?

[Joe Domaleski]: What about the little guy?

[Tim Wilson]: I've got a little bit of an axe to grind.

[Tim Wilson]: You're articulating this as a more extreme kind of [Tim Wilson]: lockout than I was thinking that starting, and maybe some of this is your experience, and I'm trying to, as I'm thinking through the anecdotal interactions I've had, I feel like it's more small businesses or small analytics consultancies that have gotten kind of enamored by [Tim Wilson]: Either this is what my platform, my CRM, or my digital analytics platform, or whatever I have, my media platform, it's just telling me stuff.

[Tim Wilson]: I don't really understand it, but it's giving me numbers that are telling me I should pump more money into it.

[Tim Wilson]: kind of in a mode of where we have to have more data.

[Tim Wilson]: We can't do anything because we don't have enough data, because there's so much talk out there of, oh, you got to feed the model, you got to feed the beast, you've got to build the big data warehouse.

[Tim Wilson]: Whereas, and you make this point in your article, and I mean, I would, Matt Gershoff has sort of made this point, like if you have no data, [Tim Wilson]: then you're making a decision with no data.

[Tim Wilson]: If you have small data and you do the simplest dumbest little line chart and draw a conclusion, you may make a big mistake.

[Tim Wilson]: You may say, I'm not thinking about confounding or seasonality or something, but it seems like overall you're gonna be in better shape using small data [Tim Wilson]: that's noisy and messy.

[Tim Wilson]: And sure, the more you know, the better, the more you know and think about something like seasonality, that's better.

[Tim Wilson]: But shouldn't there be an encouragement to say, start with what you have?

[Tim Wilson]: Have somebody who's holding your hand a little bit that's keeping you from [Tim Wilson]: over interpreting something.

[Tim Wilson]: But it sounds like your experience has been more, they're not even asking that question, which I'm struggling with saying, but they're paying somebody $500 or $5,000 a month to do stuff.

[Tim Wilson]: And how are they feeling like there's any accountability that that's a worthwhile investment if they're not looking at data?

[Joe Domaleski]: You know, don't, right?

[Joe Domaleski]: They don't see the value in it.

[Joe Domaleski]: And I think by, you know, I often tell people, you know, bad breath is better than no breath, right?

[Joe Domaleski]: Never heard that before.

[Joe Domaleski]: Never heard that before.

[Joe Domaleski]: All kinds of Southernisms.

[Joe Domaleski]: But let me add a corollary to that.

[Joe Domaleski]: But it can be so bad that it actually knocks everybody out, right?

[Joe Domaleski]: So there's this fine line, right?

[Joe Domaleski]: And so when it comes to data, yeah, start where you're at.

[Joe Domaleski]: And normally, I never encounter [Joe Domaleski]: a business leader who doesn't have some sense of financial numbers.

[Joe Domaleski]: And I think that this is a place to kind of differentiate.

[Joe Domaleski]: And so we literally had an accounting client once.

[Joe Domaleski]: And the phone call started, you know, the marketing manager brought us in and said, I need some help.

[Joe Domaleski]: And, you know, we need marketing.

[Joe Domaleski]: We're a services company.

[Joe Domaleski]: And my great, we're a services company too.

[Joe Domaleski]: And, you know, we'll set up a call with the CEO.

[Joe Domaleski]: And the CEO starts off the call.

[Joe Domaleski]: Joe, I think you're wasting your time.

[Joe Domaleski]: I don't believe in marketing.

[Joe Domaleski]: Literally, we started the phone call and I said, okay, we have nothing to talk about.

[Joe Domaleski]: And then the marketing manager's like, wait, wait, wait.

[Joe Domaleski]: And, you know, I think they were posturing a little bit.

[Tim Wilson]: Isn't that for an inspiring employee?

[Tim Wilson]: Yeah.

[Joe Domaleski]: And, you know, we started to have the conversation.

[Joe Domaleski]: And once the foot was in the door, the truth kind of came out.

[Joe Domaleski]: Well, we got burned and, you know, blah, blah, blah, blah, blah, blah.

[Joe Domaleski]: And I started appealing to this person who was a CPA on a financial basis.

[Joe Domaleski]: I said, you know, I actually have an MBA in finance.

[Joe Domaleski]: I know financial statements.

[Joe Domaleski]: You want to talk about ratios or whatever?

[Joe Domaleski]: we can do similar sorts of things with your marketing it was like a light bulb went off she was like really.

[Joe Domaleski]: I didn't know what I was paying all this money for.

[Joe Domaleski]: I wasn't getting any results.

[Joe Domaleski]: I said, well, you're actually tracking the numbers.

[Joe Domaleski]: And so I think from a data literacy standpoint that, you know, money talks, right?

[Joe Domaleski]: And so even if you can get it on those terms, you know, and then come over to marketing and it could be any other data, right?

[Joe Domaleski]: I mean, that's my background is marketing, but it could be operations.

[Joe Domaleski]: What I find though is that marketing [Joe Domaleski]: of all the areas of a business, I think many times that's the last to get the analytics treatment.

[Joe Domaleski]: Other areas of a business typically inventory and you know operations and logistics and finance and those sorts of things and even sales But but marketing sometimes in a small business is late to the analytics party.

[Moe Kiss]: That's so interesting and And I can see how I mean I can see the fact that I mean particularly like operations and finance to some degree can't function without [Moe Kiss]: data, but yeah, I don't know.

[Moe Kiss]: I've experienced maybe the inverse where those areas are like, I would say less mature, definitely established first, but less mature.

[Moe Kiss]: That's a really interesting perspective, and I can see how for small companies that would be the case.

[Joe Domaleski]: With a small business, a lot of times, Moe, their sphere of influence might be a town.

[Joe Domaleski]: or a county or a suburban area or metropolitan area and [Joe Domaleski]: Frankly, they can get by with projecting.

[Joe Domaleski]: We won't make this into a marketing class, but we'll toss out there.

[Joe Domaleski]: When I went to marketing school, we had five P's of marketing.

[Joe Domaleski]: I think they're currently teaching the kids four P's of marketing.

[Joe Domaleski]: I read an article somewhere, there's seven P's of marketing.

[Joe Domaleski]: I don't care how many P's there are, but let's just go with one of them.

[Tim Wilson]: I only know four.

[Tim Wilson]: Yeah.

[Joe Domaleski]: Okay.

[Joe Domaleski]: Yeah.

[Joe Domaleski]: Well, yeah.

[Joe Domaleski]: And, you know, promotion, let's just say it's promotion.

[Joe Domaleski]: A lot of times their ad spend is kind of low.

[Joe Domaleski]: They don't have to spend a lot to maintain an online presence in a, in a, in a small town.

[Joe Domaleski]: And so, you know, so they have back to that minimally viable marketing.

[Joe Domaleski]: They've got a website and they've got social media.

[Joe Domaleski]: They've got email.

[Joe Domaleski]: They may or may not do.

[Joe Domaleski]: Google pay per click, and maybe they do.

[Joe Domaleski]: And so there's that recognition of that.

[Joe Domaleski]: And so that creates another small data problem, right?

[Joe Domaleski]: Okay, we're thinking about advertising, but we've never done it before.

[Joe Domaleski]: Now we have zero data.

[Joe Domaleski]: Now what do we do?

[Tim Wilson]: But they're not hearing.

[Tim Wilson]: Again, maybe where I'm floating in Columbus, I know analysts who support coffee shops that have three locations.

[Tim Wilson]: I've got a cousin in Colorado who supports a lot of service plumbers and electricians.

[Tim Wilson]: For years now, it's been kind of a local SEO and even a local search engine marketing, because they're saying, why would somebody know Tim's plumbing?

[Tim Wilson]: I need to, when they're searching for plumbing, they hit this, but I pop up, that I show up in their location.

[Tim Wilson]: And my sense has been, in the case of my cousin, she's not an analytics person, she's kind of a web [Tim Wilson]: web design, website, web dev shop, but she's also kind of the content marketer and she's often saying, how do I use the data better?

[Tim Wilson]: My plumber client wants leads and he's paying me for his digital presence and what am I supposed to do?

[Tim Wilson]: A lot of this feels like there's kind of a [Tim Wilson]: between place where the agencies that are providing the service, they're not big enough to be staffing full-time analysts.

[Tim Wilson]: They may not be snowflakes like you who say, well, I'm going to just go get another master's degree to learn about it.

[Tim Wilson]: I feel like I'm seeing much more where there is [Tim Wilson]: a hunger and an interest, but a lot of kind of fear and uncertainty about where does this plug in and then relies on Google's, whatever Google's current term for their local business thing is Google Business Center.

[Tim Wilson]: Is that what it is?

[Tim Wilson]: I don't know.

[Tim Wilson]: I feel like that's been rebranded a few times.

[Tim Wilson]: Yeah, that they just say, well, I'm just going to log into that and hope that [Tim Wilson]: insights emerge and that feels wrong.

[Tim Wilson]: I don't know.

[Joe Domaleski]: Yeah.

[Joe Domaleski]: Well, you know, what's interesting and, you know, of course, we've, you know, gotten what almost halfway through the podcast and not said those dreaded letters AI.

[Joe Domaleski]: But, you know, we'll go ahead and [Joe Domaleski]: Take the opportunity to stick that in here.

[Joe Domaleski]: There is some thought.

[Joe Domaleski]: Google had some pretty big announcements at their marketing conference a couple months ago about making advertising, in particular, a little bit more self-service.

[Joe Domaleski]: So, you know, with AI augmentation, you don't have to know anything about analytics.

[Joe Domaleski]: You go in there and tell it, you know, hey, I'm a plumber and this is my target audience and let it, let it do its thing.

[Tim Wilson]: Yeah, I'll believe it when I see it because, you know, keywords for, you know, 10 years, right?

[Joe Domaleski]: And it's just not, it's not there yet.

[Joe Domaleski]: And, you know, any of you that, you know, that are listening that have ever had to set up an online ad campaign or whatever the user interface is, you know, rather, rather dense and thick, but, but, you know, to your point, Tim, you're, you're absolutely right.

[Joe Domaleski]: And that's one of the things we're trying to do with our firm is find that sweet spot at a small company.

[Joe Domaleski]: They've got that one person freelancer that's kind of advising them on stuff and they get much beyond that and you know now what do i do sort of thing.

[Joe Domaleski]: Versus a big company which may have i gotta tell this story you know i'm working with a client right now who's on the larger end of the spectrum they're actually a national company.

[Joe Domaleski]: And they are just headquartered down here.

[Joe Domaleski]: Now, don't fall out of your chairs when I'm about to tell you.

[Joe Domaleski]: And those of you that are driving while listening, don't drive off the road.

[Joe Domaleski]: Runners don't run off the path and into the bushes while you're listening to the podcast.

[Joe Domaleski]: This is a national company.

[Joe Domaleski]: Somebody set up a dashboard and the marketing manager can't even get access to it.

[Joe Domaleski]: But the executives don't want to change the platform.

[Joe Domaleski]: And so you talk about they have data.

[Joe Domaleski]: It's actually being displayed in a dashboard.

[Joe Domaleski]: They don't want to change the contract.

[Joe Domaleski]: So we may end up going in there and setting up something.

[Joe Domaleski]: to help bring the data to light as kind of a side gig.

[Joe Domaleski]: So it's like, don't mess with what's here, but we're flying blind.

[Joe Domaleski]: And we need something to steer.

[Joe Domaleski]: And I've never seen anything like it.

[Tim Wilson]: So maybe to shift gears a little, I feel like we're kind of laboring in the getting from zero to one data point.

[Tim Wilson]: And maybe it would be useful to say, OK, let's go to where there is [Tim Wilson]: a small data set.

[Tim Wilson]: And we've already talked about two definitions of small, one number of rows, one number of columns.

[Tim Wilson]: And I guess you can have a lot of columns that are sparsely populated.

[Tim Wilson]: So you go through in your article a few different aspects of that.

[Tim Wilson]: But what are if an organization, there is a person and entity that is saying, I have data.

[Tim Wilson]: It is not a lot of data.

[Tim Wilson]: Where should I be starting and what should I be cognizant of?

[Tim Wilson]: Maybe let's not worry about the pitfall so much first, but what can be done effectively with small data?

[Joe Domaleski]: Yeah, that's a great question.

[Joe Domaleski]: use what you have sort of mentality.

[Joe Domaleski]: Some is better than none, more is better than some, but if all you got is some, let's use that.

[Joe Domaleski]: And I think the first step in that process [Joe Domaleski]: We have two interns working with us right now.

[Joe Domaleski]: What's cool is they are also fellow students at other universities.

[Joe Domaleski]: So they're my interns, but I can also say, hey, I'm a fellow student.

[Joe Domaleski]: And we'll put them on projects, okay?

[Joe Domaleski]: And what do you think?

[Joe Domaleski]: Again, they are in master's degree programs in analytics.

[Joe Domaleski]: When we start showing them small business data sets, what do you think their first reaction is?

[Tim Wilson]: We can't get statistical significance with this.

[Tim Wilson]: We don't have...

Yeah.

[Tim Wilson]: Oh, really?

[Joe Domaleski]: Okay.

[Joe Domaleski]: Yeah.

[Joe Domaleski]: The initial reaction is normally, we can't work with that.

[Joe Domaleski]: And it's kind of a teaching point to them.

[Joe Domaleski]: Oh, yes, you can.

[Joe Domaleski]: And you're going to have to because it's all we got.

[Joe Domaleski]: So let's figure out how to make use of it.

[Joe Domaleski]: In the article, I outlined some different techniques and one of the things that we talk about, and again, to somebody who's familiar with analytics, some of these things are [Joe Domaleski]: or, you know, are going to sound familiar.

[Joe Domaleski]: You know, I think the first thing, you know, let's call it gut instinct.

[Joe Domaleski]: Let's call it, you know, naive forecasting, you know, basic benchmarks.

[Joe Domaleski]: I think this is where, you know, experience can help provide some clarity on what you're looking at and what you can do with it.

[Joe Domaleski]: Case in point, okay, we had 50 email opens on the last campaign.

[Joe Domaleski]: Well, Joe, that's just not statistically significant.

[Joe Domaleski]: We can't do anything with that.

[Joe Domaleski]: Or even the corporate client, hey, here's what we know and it's all that we know.

[Joe Domaleski]: And I said, well, that's kind of a starting point.

[Joe Domaleski]: We know something.

[Joe Domaleski]: It's better than nothing.

[Joe Domaleski]: Now, that may be noise, but we can apply some common sense to it and leverage what we know to try to make the most of [Joe Domaleski]: I guess it's kind of like being in the kitchen, right?

[Joe Domaleski]: Being short of ingredients, you know, when in doubt put more flour in there or put an egg in there, you know, that always makes, well, I don't, I don't know how that works, but my wife always figures out a way, you know, okay, you don't have to go to the store.

[Joe Domaleski]: We'll, we'll figure out, I think she learned it from her grandmother.

[Joe Domaleski]: We'll just use the ingredients we have sort of saying.

[Joe Domaleski]: And I think that's, uh, that's, that's true with, uh, you know, with data.

[Joe Domaleski]: Another thing that I like to do is just kind of aggregate data into [Joe Domaleski]: into chunks or clusters or groups.

[Joe Domaleski]: Sometimes that really provides some clarity.

[Joe Domaleski]: Instead of looking at one big blob, let's see what's in common.

[Joe Domaleski]: I'm a Gen Xer, so maybe I fit into a certain category in the marketing data and it's not on the same level as maybe a Gen Z or like my kids like millennials.

[Joe Domaleski]: So, you can do good old-fashioned classification even with a small dataset and get more insight than just looking at spreading it too thin in a big blob.

[Joe Domaleski]: That's been very effective working with little datasets.

[Tim Wilson]: As you were talking, there's an upside to small data is that you can actually [Tim Wilson]: look at the data.

[Tim Wilson]: Like if you're dealing with a million rows, you can do distributions and you can do some EDA and try to get a handle on the data and maybe pull and look at some of it.

[Tim Wilson]: If you have small data, you know what?

[Tim Wilson]: You can look at every one of those leads that came in and see which were garbage versus which ones weren't.

[Tim Wilson]: And I think that goes with the, you know, if the master student says, yes, but you may have selection bias because your form's too long, [Tim Wilson]: Good knowledge to know, but this is back to bad breath or no breath, I guess.

[Tim Wilson]: If you have a small data set, if you have qualitative data, or even if you want to collect qualitative data, [Tim Wilson]: If it's a small amount, you don't have to come up with some fancy natural language processing to try to assess the sentiment.

[Tim Wilson]: You can read 100 comments.

[Tim Wilson]: You can read 10 comments a lot faster.

[Joe Domaleski]: That's right.

[Joe Domaleski]: In fact, ironically, I am taking [Joe Domaleski]: analysis of unstructured data this semester along with the simulation class, which I've done before.

[Joe Domaleski]: I've actually written some on my blog about some sentiment analysis or restaurant reviews, but you're absolutely right.

[Joe Domaleski]: When you have that small data, maybe 50 people opened your email, you can actually know who they are and reach out to each one of them and ask them more about, did you find it useful and that sort of thing.

[Joe Domaleski]: Can't do that if you're dealing with, you know, 100,000 email opens or not.

[Joe Domaleski]: You can't do it easily.

[Moe Kiss]: Yeah.

[Moe Kiss]: And you also mentioned controlled micro experiments.

[Moe Kiss]: Can you tell us a bit more about that?

[Joe Domaleski]: Yeah, you know, that's a fancy, you know, I tried to jazz it up when I put in the article, but I, you know, I think, I think at the end of the day, you know, we were all kids once and I remember I've always been somewhat of a science and math geek.

[Joe Domaleski]: You know, I used to just put things together, little experiments, right?

[Joe Domaleski]: Steal stuff from the kitchen and try not to blow up the house or hook up electrical parts and try not to, you know, short out.

[Joe Domaleski]: Although I think I once did stick a paperclip in a light socket.

[Joe Domaleski]: My dad was not happy about that, but it was a cool.

[Joe Domaleski]: That's what siblings are for, to be blamed for the...

Yeah, and I have a younger brother, Dr.

Chris Domalescu, who has a PhD in this stuff.

[Joe Domaleski]: So, you know, he's a smart guy.

[Joe Domaleski]: But I am the big brother, and don't you forget it, Chris, if you're listening.

[Joe Domaleski]: But, you know, I think micro experiments, again, if you're small business, you're agile, you're flexible, you don't have to have a steering committee, you don't have to have, you know, 20 approvals.

[Joe Domaleski]: just try something out.

[Joe Domaleski]: You know, you can do it in a controlled manner and see what works.

[Joe Domaleski]: You know, whether it's a little A-B test or just, you know, I like to use the term a bake-off, right?

[Joe Domaleski]: You know, here's a sample, you know, you test this and you test this and see which one, you know, seems to work better.

[Joe Domaleski]: And it may not be scientific, but it, you do kind of pick preferences.

[Joe Domaleski]: This happens a lot, you know, on the design side of our agency, too.

[Joe Domaleski]: How do you measure a logo effectiveness?

[Joe Domaleski]: I mean, really, okay, there's been papers written about that, but at the end of the day, you get a small committee.

[Joe Domaleski]: Do we like it or not?

[Joe Domaleski]: You know, what do you like about it?

[Joe Domaleski]: What do you don't like about it?

[Joe Domaleski]: And so, I think in similar manner, you can do that with, you know, other more qualitative marketing things.

[Moe Kiss]: So one of the things that I was listening back to an old episode and it was funny because Tim's Last Call was a podcast that I love.

[Moe Kiss]: It was about choiceology and it was talking about natural experiments.

[Moe Kiss]: And so I wonder if this comes up as well.

[Moe Kiss]: I'm sorry, Katie Milkenman's a genius, but the episode was essentially like looking at past [Moe Kiss]: I know like an example might be like, I don't know, we didn't have $4,000 that month so we just didn't do any marketing.

[Moe Kiss]: And so you use it as like quote unquote a natural experiment to see what would happen if you turned off marketing.

[Moe Kiss]: It's like, have you found that like looking back over historical incidents is like something that you can derive like meaning and direction from?

[Joe Domaleski]: Absolutely.

[Joe Domaleski]: And I think this applies to small or large data sets, right?

[Joe Domaleski]: Data tells a story.

[Joe Domaleski]: One of the interesting things, and I don't know about all of you, your respective organizations or companies, but we saw a massive uptick of business during COVID.

[Joe Domaleski]: Why might that happen when most things were buckling down and cutting costs and we don't know if we're going to be here tomorrow?

[Joe Domaleski]: At first blush, you would think that that would impact everybody equally, but it didn't.

[Joe Domaleski]: We saw almost a 30% increase in our billings to customers.

[Joe Domaleski]: Now, why might you think that it, you know, that happened during COVID?

[Julie Hoyer]: Well, my guess is if you're working with small local businesses, a lot of people, you know, the larger organizations, if I'm remembering correctly, had the supply chain issues or, you know, things shut down, they couldn't have the big warehouses.

[Julie Hoyer]: And I remember there was a big push of like, [Julie Hoyer]: Go buy local, order from a local business, help them out when people can't go into their stores.

[Julie Hoyer]: So I think there was like a little boost for a while, right, from that?

[Joe Domaleski]: There was.

[Joe Domaleski]: I think Julie, that was part of it.

[Joe Domaleski]: I think another part of it.

[Joe Domaleski]: What did we all do?

[Joe Domaleski]: Now, I let my staff work from home.

[Joe Domaleski]: We have two physical offices, but I let them work from home.

[Joe Domaleski]: And they're all creatives, and they like that.

[Joe Domaleski]: They keep some morale up.

[Joe Domaleski]: But everybody worked from home, for the most part, during COVID.

[Joe Domaleski]: And if you were a small retailer a restaurant mom and pop shop or whatever you were invisible if you weren't online so everything pivoted people are driving down the road to see your sign.

[Joe Domaleski]: They are now sitting at home and they're on Facebook or Instagram or whatever.

[Joe Domaleski]: And so they had to pivot their marketing to online.

[Joe Domaleski]: And it was more than an aberration.

[Joe Domaleski]: I mean, it was sustained over about two and a half years.

[Joe Domaleski]: the uh and so as you were suggesting mo you can look back and kind of tell the story at first glance you'd say everybody kind of had a down and like no it actually didn't you know people needed our services more than ever because [Joe Domaleski]: restaurants, we're converting to pickup, right?

[Joe Domaleski]: So, hey, we need you to update our website.

[Joe Domaleski]: And if our website isn't updated, we're invisible.

[Joe Domaleski]: And we're going to do curbside pickup and those sorts of things.

[Joe Domaleski]: And so, yeah, data tells a story.

[Joe Domaleski]: And even if it's a small data set, it is worth looking at that.

[Tim Wilson]: But that does get to one of the pitfalls that I really liked that you had in the article about ignoring context and external factors.

[Tim Wilson]: And this doesn't seem like, to me, it's not tied to necessarily small or [Tim Wilson]: large.

[Tim Wilson]: The number of anecdotes I have of somebody just ignoring an externality and looking at the data and saying, this thing jumped way up or this thing plummeted.

[Tim Wilson]: People were aware that COVID was going on.

[Tim Wilson]: People were aware of tariff stuff, but there's other things that happen.

[Tim Wilson]: Are they aware of what their competitor is doing?

[Tim Wilson]: And are they aware that if your data looks very surprising, [Tim Wilson]: An analyst, the first thing would say, it's probably not a miracle.

[Tim Wilson]: We found the perfect marketing message, and this has caught fire.

[Tim Wilson]: Something might be going on that I may or may not be able to influence.

[Tim Wilson]: This email campaign did wildly better than any of our past email campaigns.

[Tim Wilson]: I need to stop and think about my business and the operating context first and see if I can think about it.

[Tim Wilson]: Again, maybe there you have the nice thing about small data is you can say, I can really go look at what they did.

[Tim Wilson]: Did a bot get a hold of an email or something like that?

[Tim Wilson]: It does seem like there's the same challenge scales down that does go to a little bit of a data fluency that if it's easy to sort of overinterpret the data, the statistician's reaction will be, well, you need more data, that's not necessarily the right answer.

[Tim Wilson]: You just need to be thinking about what's generating the data and what might actually also be going on.

[Joe Domaleski]: Yeah, toward the end of the article, I think I summarized some of these things that I commonly see that are mistakes with small data, and they're not necessarily limited to small data.

[Joe Domaleski]: I think you're right, Tim, they applied other things.

[Joe Domaleski]: Probably the number one thing that I see dealing with small data are people killing campaigns too soon.

[Joe Domaleski]: You know, it's the classic, we've been running this for two months.

[Joe Domaleski]: Why haven't we set sales records?

[Joe Domaleski]: We're gonna kill it.

[Joe Domaleski]: And, you know, we know it takes time for things to work.

[Joe Domaleski]: Looking at the context, which we talked about, you know, another one is kind of the inverse of that, which is, you know, okay, we had a great, we may just made a great sale.

[Joe Domaleski]: It must be the marketing.

[Joe Domaleski]: As much as I would love to claim credit for that, [Joe Domaleski]: Okay, that's a single data point.

[Joe Domaleski]: We can't even draw a line with that.

[Joe Domaleski]: We can just look at it and be amazed.

[Joe Domaleski]: Can we at least have a couple more to kind of draw a thing?

[Joe Domaleski]: And then, of course, people waiting for perfect data, and I think that's the trap many analysts and data scientists fall into is this [Joe Domaleski]: this mythical thing.

[Joe Domaleski]: So I have an intern right now that's working with us.

[Joe Domaleski]: His name's Sam.

[Joe Domaleski]: Great guy.

[Joe Domaleski]: I'm going to have him listen to this.

[Joe Domaleski]: And I want to encourage him.

[Joe Domaleski]: He'll graduate at the end of the year.

[Joe Domaleski]: We're looking at something, some of your listeners probably heard of it, maybe you guys, just a classic marketing mix model.

[Joe Domaleski]: We're looking at the Meridian model.

[Joe Domaleski]: It's an open source model that's put out there.

[Joe Domaleski]: The sample data set that comes with the GitHub package, [Joe Domaleski]: the marketing spend is like $240 million.

[Joe Domaleski]: Now, I don't have a client that even their annual revenues don't get anywhere close to that.

[Joe Domaleski]: And so we're looking at this and Sam was, I had him looking in the model and I had him present this internally to our staff.

[Joe Domaleski]: And I said, we're going to run some of our client data through it.

[Joe Domaleski]: And, you know, okay, a marketing spend on, you know, per month, even for a medium sized client might be $5,000.

[Joe Domaleski]: So, you know, $240 million, $5,000.

[Joe Domaleski]: You know, what do you do with that?

[Joe Domaleski]: Of course, Sam's initial reaction.

[Joe Domaleski]: I'm not sure we have enough data.

[Joe Domaleski]: Oh, we've got enough data.

[Joe Domaleski]: We're going to make the most of it.

[Tim Wilson]: Wow.

[Tim Wilson]: Well, on that note, uplifting as it is, but it's a good segue to, we're going to start heading towards wrap, but [Tim Wilson]: Before we do that, we're going to actually take a quick break with our friend Michael Kaminski from ReCast, which is an MMM and GeoLift platform helping teams forecast accurately and make better decisions.

[Tim Wilson]: Michael's been sharing bite-sized marketing science lessons over the past months and the coming months to help you measure smarter.

[Tim Wilson]: So I'm going to turn it over to you, Michael.

[Michael Kaminsky (Recast)]: The synthetic control method has been called the workhorse of causal inference.

[Michael Kaminsky (Recast)]: Synthetic controls are used to generate causal estimates in situations where large-scale randomization isn't possible or is too expensive.

[Michael Kaminsky (Recast)]: When we're doing causal inference, we're always trying to compare some treatment effects to a counterfactual, what would have happened without the treatment.

[Michael Kaminsky (Recast)]: The synthetic control method is a method for creating a counterfactual.

[Michael Kaminsky (Recast)]: It can be used in experimental contexts where a researcher has intentionally manipulated some treatment, or in quasi-experimental contexts where a researcher is trying to evaluate the impact of some change that wasn't intentionally manipulated.

[Michael Kaminsky (Recast)]: The idea behind a synthetic control is simple.

[Michael Kaminsky (Recast)]: We want to identify control individuals whose pretreatment behavior most closely resembles those of our treated individuals.

[Michael Kaminsky (Recast)]: So the idea behind synthetic controls is to create a weighted average of potential control individuals that best match the treated individuals before the treatment.

[Michael Kaminsky (Recast)]: In the case of geographies, you might imagine the best counterfactual control for Houston is a mix of Austin and New Orleans and Dallas, and that that mix is a better counterfactual than any of those cities individually might be.

[Michael Kaminsky (Recast)]: There are lots of different methods for creating these synthetic controls, and correctly estimating the uncertainty in the causal estimates can be quite tricky.

[Michael Kaminsky (Recast)]: But when utilized in the right experimental context, synthetic controls can help practitioners run statistically powerful experiments even when large-scale randomization isn't possible.

[Tim Wilson]: All right, so if you enjoyed that mini lesson, Michael and the team at Recast have put together a library of marketing science content specifically for Analytics Power Hour listeners.

[Tim Wilson]: For everything from building media mix models in-house to communicating uncertainty to your board, head over to www.getrecast.com slash aph.

[Tim Wilson]: That's www.getrecast.com slash aph.

[Tim Wilson]: All right.

[Tim Wilson]: That was a fun discussion, and I'm not sure if I'm coming away from it more energized or depressed about the prospects.

[Tim Wilson]: Hopefully, it's a call to arms to not dismiss small data from some ivory tower of your ivory tower moe with your canva [Tim Wilson]: You probably do have Yotabytes, the data that you're working with.

[Tim Wilson]: Do you actually know data scale?

[Tim Wilson]: Do you talk Zeta, Exo, Yotabytes?

[Tim Wilson]: You ever asked how much data do we have?

[Tim Wilson]: Yeah, Yotabytes.

[Julie Hoyer]: I thought it was Yotabytes.

[Julie Hoyer]: Where did Yotabytes come from?

[Tim Wilson]: Oh, well, that's your assignment, find out.

[Tim Wilson]: The last thing I like to do on the show is go around the old virtual podcast bar and share a last call, something that might be of interest to our users of any shape or form.

[Tim Wilson]: And Joe, you're our guest.

[Tim Wilson]: Do you have a last call you'd like to share?

[Joe Domaleski]: Yeah, so I'll do this in two small parts.

[Joe Domaleski]: First of all, I just, you know, for folks out there listening to this awesome podcast, [Joe Domaleski]: If you encounter small data, don't be discouraged.

[Joe Domaleski]: You actually, hopefully, you listen to this and you came away with some ideas.

[Joe Domaleski]: You're not alone because sometimes I feel like I'm alone.

[Joe Domaleski]: Am I the only guy who's twiddling bits here?

[Joe Domaleski]: But when I'm not doing graduate school stuff or running a small business, I like to kind of head in the other direction and get my mind off of computers and machine learning and numbers.

[Joe Domaleski]: And I have been reading this very addictive book series, and as usual, I'm late to the party.

[Joe Domaleski]: It is called Dungeon Crawler Carl by Matt Henneman.

[Joe Domaleski]: It is awesome.

[Joe Domaleski]: I understand it's going to be made into a TV miniseries.

[Joe Domaleski]: The premise.

[Joe Domaleski]: How many, how many of you, you all know what Dungeons and Dragons is.

[Joe Domaleski]: Okay.

[Joe Domaleski]: Picture, and this sounds ludicrous, but it's normally the basis of a great concept.

[Joe Domaleski]: Aliens take over the earth and turn it into a dungeon crawl game that's a game show and People are trapped in there and it's like a live D&D and it's being broadcast across the galaxy and the lead car character Carl His cat can talk and her name is Princess doughnut and it is everywhere Dungeon crawler Carl.

[Joe Domaleski]: There's seven books the guy Matt wrote it during COVID and self published it and [Joe Domaleski]: A book publisher's picked it up, so it's a great business success story.

[Joe Domaleski]: You can't put it down.

[Joe Domaleski]: It is hilarious.

[Joe Domaleski]: Is he writing more?

[Joe Domaleski]: He is still writing more.

[Tim Wilson]: Okay.

[Joe Domaleski]: Dungeon Crawler Carl, highly recommended.

[Tim Wilson]: That is, if Michael Helbling was on here, I'm deeply curious to know.

[Joe Domaleski]: I mean, I know there's listeners that are shaking their head going, yes.

[Tim Wilson]: Of course.

[Tim Wilson]: Yeah.

[Tim Wilson]: Awesome.

[Tim Wilson]: That's on my reading list now.

[Tim Wilson]: Julie, what's your last call?

[Julie Hoyer]: My last call does use the dreaded two letters of AI, but it was a recent article that came out from Cassie Kozarkov, one of our faves.

[Julie Hoyer]: Stop it.

[Julie Hoyer]: Was this going to be yours, Moee?

[Moe Kiss]: No, but I did also read it yesterday and I really enjoyed it.

[Julie Hoyer]: Yeah, it was really good.

[Julie Hoyer]: And actually, my favorite part, so the title of this one is the alignment trap, why Gen AI metrics spark debate, not clarity.

[Julie Hoyer]: She has a lot of great subsections in there.

[Julie Hoyer]: And actually, why I wanted to use it as a last call was the fact that me and Tim were actually going back and forth and discussing the article.

[Julie Hoyer]: And it was interesting that she makes so many strong points in there and a lot of good points.

[Julie Hoyer]: But in general, I feel like it's one of those articles that I want to go back and reread.

[Julie Hoyer]: And it sparked a lot of good debate between me and Tim, where I just think if you're [Julie Hoyer]: If you're starting to have conversations about measuring the performance of AI, something we noticed throughout the article and why I want to reread it is like there's kind of two themes going, measuring the efficiency, quote unquote, of like the model itself, the large language model, like the actual process behind it, whichever one you choose.

[Julie Hoyer]: But then there's the larger idea, which I was more honed in on and interested in of [Julie Hoyer]: the actual output, return on investment, did it help you reach your business outcome?

[Julie Hoyer]: Obviously, that's something we talk about so much on the show.

[Julie Hoyer]: AI, GenAI being another tool to help you get there.

[Julie Hoyer]: It's a good read.

[Julie Hoyer]: It's got a lot of good points, and it definitely gets you thinking.

[Tim Wilson]: It just seems like it mixed those together where I'm like, no, part of this [Tim Wilson]: is not hard.

[Julie Hoyer]: Tim's like, let me drag my soapbox over here.

[Tim Wilson]: No, no, no.

[Tim Wilson]: That's what I want to be is out publicly saying that I thought maybe Cassie was a little off on something.

[Moe Kiss]: So I was going to talk about something completely different and now I'm going to pivot.

[Julie Hoyer]: Oh.

[Moe Kiss]: I'm going to pivot completely.

[Julie Hoyer]: You don't want to do the Tim and do both?

[Moe Kiss]: No, the other one's like a three-fer.

[Moe Kiss]: So I had a really interesting conversation with the CEO on the weekend just randomly.

[Moe Kiss]: And we were talking about AI and creativity.

[Moe Kiss]: Look at that little flex.

[Tim Wilson]: She's at a professional sports game just hanging out with the CEO as one does.

[Tim Wilson]: OK, carry on.

[Moe Kiss]: Anyway, we were having a very interesting conversation about AI and creativity.

[Moe Kiss]: Because obviously, that's something that [Moe Kiss]: Canva where I work gives a lot of thought to and fundamentally like humans still being at the centre of it and I was talking about [Moe Kiss]: I was talking about how I use AI for generating ideas and just how I feel that sometimes I'm getting diverse perspectives.

[Moe Kiss]: And he did a game with me that completely stumped me.

[Moe Kiss]: And I now am like, does everyone know this?

[Moe Kiss]: Is this a thing that is known or is this a thing that is not known?

[Moe Kiss]: And I am super, anyway.

[Moe Kiss]: So we sat there and he goes, there was him and he had his son with him.

[Moe Kiss]: He goes, open your AI tool of choice, right?

[Moe Kiss]: Claude, church, APT, whatever.

[Moe Kiss]: And we had different ones.

[Moe Kiss]: And it goes, you had to give it a prompt to give you a number between one and 10.

[Moe Kiss]: Has everyone heard of this?

[Moe Kiss]: Everyone gets the same number, regardless of the tool.

[Moe Kiss]: And then it goes, give me another number.

[Moe Kiss]: And then you go, give me another number.

[Moe Kiss]: And we wrote our prompts slightly differently because each person wrote it in their own language.

[Moe Kiss]: And you got the same number.

[Moe Kiss]: And then it said, give me another number.

[Moe Kiss]: And you get the same number.

[Moe Kiss]: And I was just sitting there like brain exploding because I think in my mind, the way I've been thinking a lot is that companies that are leaning in really heavily here are going to have this advantage.

[Moe Kiss]: And I guess kind of where I'm landing now is like, maybe that advantage will plateau at some point if I'm thinking that I'm using some of these tools for more [Moe Kiss]: Hypothesis generation and things like that, like maybe there's going to come a point where a plateau is, but it was just such an interesting exercise to go through that I hadn't.

[Moe Kiss]: And I was like, is this a no known that everyone's aware of?

[Tim Wilson]: Can we all do that right now?

[Tim Wilson]: Because I've got caught up and I'm asking.

[Moe Kiss]: All right.

[Moe Kiss]: Give me a number between one and 10.

[Tim Wilson]: But we can kind of frame it as long as it's asking clearly that question, I can put, give me an integer.

[Joe Domaleski]: So we all type that in.

[Joe Domaleski]: Give me a number between one and 10.

[Tim Wilson]: But it can be something.

[Tim Wilson]: I'll give me an integer that falls between one and 10 inclusive or something like that.

[Tim Wilson]: Sure, Tim.

[Moe Kiss]: I don't know.

[Moe Kiss]: That would be how the team phrases it.

[Moe Kiss]: You said this is a line of experiments.

[Tim Wilson]: I'm not trying to break this.

[Tim Wilson]: Yeah, I'm curious.

[Tim Wilson]: Are we ready?

[Tim Wilson]: Are we all going to go?

[Julie Hoyer]: Wait, wait, wait.

[Julie Hoyer]: Oh, hang on.

[Julie Hoyer]: I'm struggling over here.

[Julie Hoyer]: OK.

OK.

[Tim Wilson]: Yes.

[Tim Wilson]: Oh my God.

[Tim Wilson]: So Moee's and Tim's and Joe, you got the same thing?

[Tim Wilson]: Yeah.

[Tim Wilson]: Same number.

[Tim Wilson]: Okay.

[Tim Wilson]: And then give me another number.

[Julie Hoyer]: Julie.

[Julie Hoyer]: Yeah, I got seven.

[Julie Hoyer]: Sure.

[Julie Hoyer]: How about seven?

[Julie Hoyer]: That's how it answered.

[Tim Wilson]: Oh.

[Tim Wilson]: Well, that's.

[Tim Wilson]: Oh.

[Tim Wilson]: What?

[Tim Wilson]: What?

[Tim Wilson]: I got four.

[Julie Hoyer]: I got three.

[Tim Wilson]: I got three.

[Moe Kiss]: Okay.

[Moe Kiss]: So this is interesting.

[Moe Kiss]: I got four as well, Tim, but I've done this three times now and every other time I've gotten three is the second number.

[Julie Hoyer]: I even said, the way I prompted it after seven was cool.

[Julie Hoyer]: Can I have another?

[Moe Kiss]: I tried to be very casual.

[Moe Kiss]: And then normally the third number is also the same.

[Moe Kiss]: So the thing that's very interesting about this is like, [Moe Kiss]: just especially with the intersectionality of creativity, like, are we actually generating more creative ideas or?

[Moe Kiss]: No, well, Jesus Christ, no.

[Julie Hoyer]: Isn't it all statistically, you know what I mean?

[Julie Hoyer]: It's all like the statistics of what is most likely in the combinations of the words and the, I mean, I don't...

I mean, it's work speaking a little bit, but...

There is also a very interesting article about Workslop that we can talk about next time on the show.

[Tim Wilson]: Yeah.

[Joe Domaleski]: Oh, yeah, I slop.

[Joe Domaleski]: Yeah.

[Tim Wilson]: I have a whole rant that was recorded that I never put it up.

[Tim Wilson]: Val said that I could or should, but it was like six minutes and I was like, but who is this for?

[Moe Kiss]: So Tim, after our live experiment on the show, over to you.

[Moe Kiss]: What's your last call?

[Tim Wilson]: So I'm going to go very simple with, and this was a kind of learned about this podcast called Lost Women of Science.

[Tim Wilson]: Heard about it through 99% invisible.

[Tim Wilson]: And basically the premise is the hosts go back with people that you haven't necessarily heard of and they kind of [Tim Wilson]: bring them up and kind of talk through them.

[Tim Wilson]: So specifically the one I listened to was June Bacon Bursey, which was the weather expert who answered the $64,000 question.

[Tim Wilson]: And it's just, it is a well done.

[Tim Wilson]: I'm kind of hooked on the, listen to a couple other episodes since then.

[Tim Wilson]: But if you're into kind of history and people who dive deep and kind of [Tim Wilson]: go a little beyond the Wikipedia entry on somebody and they tend to have fascinating stories.

[Moe Kiss]: Nice.

[Moe Kiss]: I just put that on my list.

[Moe Kiss]: There's also a Spanish version, which is very cool too.

[Tim Wilson]: That is true.

[Tim Wilson]: Yes.

[Tim Wilson]: They recorded it in both.

[Tim Wilson]: Yeah.

[Tim Wilson]: I made the note to do it a while back and I'd forgotten that.

[Tim Wilson]: Well, this has been a pretty interesting discussion.

[Tim Wilson]: And I think we all now have a bunch of reading and listening material from the last calls alone.

[Tim Wilson]: So, all right.

[Tim Wilson]: But Joe, thank you for coming on, taking a break from your running a company and being a student.

[Joe Domaleski]: Well, thanks for sticking up for the small people, the little people, the little data people.

[Tim Wilson]: If you have enjoyed the show, please leave us a review or a rating on whatever platform you listen to.

[Tim Wilson]: I think Joe mentioned before we were recording that he has a podcast sticker on his laptop, and you too can do a sticker.

[Tim Wilson]: If you go to analyticshour.io.

[Tim Wilson]: click on the global nav, fill out a little form, we'll happily send you a sticker or two or three.

[Tim Wilson]: You can also reach out to us on the LinkedIn, any one of us individually, or the company page, or reach out to us on the Measures Slack, or you can just send us a good old email at contact at analyticshour.io.

[Tim Wilson]: So regardless, if you are analyzing [Tim Wilson]: Yota bytes?

[Tim Wilson]: Yota bytes, not Yota bytes.

[Tim Wilson]: Okay, Yota.

[Tim Wilson]: Think Star Wars, not Seinfeld.

[Tim Wilson]: Yota bytes of data, or literally dozens of rows of data, or whether you're just trying to explain and define what data is to one of your clients for Julie Hoyer and Moee Kiss.

[Tim Wilson]: I just want to encourage you to keep analyzing.

[Announcer]: Thanks for listening.

[Announcer]: Let's keep the conversation going with your comments, suggestions, and questions on Twitter at @analyticshour on the web at analyticshour.io, our LinkedIn group, and the Measure Chat Slack group.

[Announcer]: Music for the podcast by Josh Grohurst.

[Charles Barkley]: So smart guys wanted to fit in.

[Charles Barkley]: So they made up a term called analytics.

[Charles Barkley]: Analytics don't work.

[Charles Barkley]: Do the analytics say go for it, no matter who's going for it?

[Charles Barkley]: So if you and I were on the field, the analytics say go for it.

[Charles Barkley]: It's the stupidest, laziest, lamest thing I've ever heard for reasoning in competition.

[Tim Wilson]: Jack, it's perfect record if you're really feeling.

[Tim Wilson]: I'm not jinxed to this.

[Tim Wilson]: I'm happy.

[Tim Wilson]: That's a gamble.

[Joe Domaleski]: They've gotten better because I'm a student there now.

[Tim Wilson]: I was going to ask you to put in your address so we could send you a little something, but I'm pretty sure if my writers are right.

[Tim Wilson]: Is that right?

[Joe Domaleski]: Yeah, because I have your face on my laptop.

[Tim Wilson]: Yeah.

[Tim Wilson]: We will not send you anything else with my face on it.

[Julie Hoyer]: You don't hear that often.

[Tim Wilson]: I know.

[Joe Domaleski]: Well, the quintessential analyst, you know, that was to, it's right up there with the rest of my stickers.

[Julie Hoyer]: Oh my gosh, Heldl is going to be so happy to hear that.

[Julie Hoyer]: It's going to make Heldl weak.

[Julie Hoyer]: I'm just going to try to do it all.

[Tim Wilson]: See how it goes.

[Tim Wilson]: Show me Tim.

[Tim Wilson]: Yeah, just me.

[Tim Wilson]: It's all about me.

[Tim Wilson]: Rock Flag and Yodobites.

[Tim Wilson]: I forgot to actually come up with something.

[Tim Wilson]: It had to be that.

[Tim Wilson]: It had to be that.