Episode Description
Traditional Googlebot crawling is falling while AI retrieval bots climb. Dan Kurtz of Miss Pepper AI reads it off his own server logs, and his conclusion is not that SEO is over. It is that we have done this before. We have been multi-purposing content into infographics, reels and press releases for twenty years because humans have preferred content types. Bots have one too.
Key takeaways
- Bots are a customer segment, not a technical problem. They have a format preference, and serving it is ordinary multi-purposing.
- The downside of publishing a machine-readable companion is bounded. Worst case, the crawler ignores it.
- Linking to the file is the mechanism. A companion file nobody links to is a file nobody fetches.
- Deliverable-count retainers are a symptom. Hitting a monthly quota is not the same as paying attention.
- An SSL certificate reissue may force AI recrawls. Offered as an untested hypothesis, with the caveat attached.
- The argument: bots as a customer segment
- What the crawl data actually shows
- The agentic flywheel
- Why he refuses deliverable-based retainers
- Chapters and timestamps
- People, tools and sources mentioned
- Questions this episode answers
- Go deeper
As SEOs and content marketers, we’ve been doing that for decades. We call it multi-purposing. Bots have a preferred content type. Technically, they’re just an additional customer.
— Dan Kurtz, 37:42
The practical version is a plain-text or markdown file sitting alongside the HTML page. Dan’s case for doing it is that the risk is asymmetric:
I would absolutely spend the time to go ahead and do that and invest in it, because the worst thing that can happen is they pass it. They fly over it.
— Dan Kurtz, 37:42
Then the step almost everyone skips. “That’s just a matter of directing them to it next time they come to the website and saying, hey, this file exists.” We took that literally and published machine-readable companions across twelve episodes, linked from every post. One thing we found by accident: our own flagship site had been returning 429 to GPTBot on every page.
Not everyone agrees. Malte Landwehr makes the opposite case, that a second URL carrying the same content is not cost-free, and would serve markdown by user agent at the same URL instead. Two practitioners, two defensible positions, which is usually the sign that the question is open rather than solved.
What the crawl data actually showsTraditional Google bots down, I think something like sixty percent crawling sites, but AI bots are up eighty percent for doing the crawling and retrieval. It’s just looking at server logs and stuff like that, you know, got my own sites that I can see those kinds of tests on.
— Dan Kurtz, 6:59
He frames it as his own telemetry rather than an industry benchmark, and that distinction matters. The independently verified figure is Cloudflare’s: automated traffic overtook human traffic on the web in June 2026.
None of this appears in Search Console. Bot requests are not impressions. If you want to know which crawlers actually reached you and what they got, the answer is in your access logs, and we wrote up the log-analysis method for auditing AI bot traffic as a standing SOP.
The SSL trick, with its caveat
If you go ahead on your hosting provider and you go ahead and you reissue a new SSL certificate, it will force all of the AI bots to recrawl your stuff all over again... I don’t know what the cadence is, I don’t know what the frequency is.
— Dan Kurtz, 38:09
Treat that as a hypothesis worth testing rather than a tactic. He says so himself.
The agentic flywheelEverything we do for SEO, I’ve got a dashboard that’s got an API. The thought leadership platform for the exec, that’s got an API. The call and appointment setting platform, that’s got an API. So now all of these all talk to each other and they’re all passing baton data between each other.
— Dan Kurtz, 26:18
Sales talks to SEO talks to marketing talks to appointment setting. He expects to publish results on what that does to a business around Q1 2027.
One component is an A/B loop that hands Microsoft Clarity and SEO Gets data to Claude and lets it rewrite the page. Worth a warning before you build the same thing: Clarity’s regex filters silently return nothing, and Search Console and GA4 run a third engine again. Our pattern compatibility checker tells you which of the three will actually accept a given filter, and the regex library has the patterns themselves organised by the question they answer.
Why he refuses deliverable-based retainersDeliverables in SEO means that you’re not paying attention. It’s you’re focused on making sure you hit that quota every month.
— Dan Kurtz, 13:25
His alternative is an enablement layer: AI first, humans working on top of it, and no scope creep pushed down onto whoever is doing the work.
Chapters| Time | What happens |
|---|---|
| 0:14 | Miss Pepper AI’s origin, and the 2012 Google penalty that killed the first version |
| 2:15 | Seth Godin’s Linchpin, then Black Hat World |
| 6:59 | Googlebot down 60%, AI bots up 80%, read off his own server logs |
| 13:25 | “Deliverables in SEO means that you’re not paying attention” |
| 26:18 | The agentic flywheel: an API between every function |
| 37:42 | Bots as a customer segment with a format preference |
| 38:09 | Linking to the file, and the SSL reissue trick |
| Entity | What it is |
|---|---|
| Miss Pepper AI | Dan’s agency and product, built on an AI enablement layer beneath human work |
| Data for LLMs | His approach to publishing structured data straight to the models |
| Microsoft Clarity | Behaviour analytics feeding his A/B rewrite loop |
| SEO Gets | Search Console data layer in the same loop |
| Cloudflare bot report | The independent source for the crossover claim |
| Seth Godin, Linchpin | Named as the turning point early in his career |
- Are AI crawlers actually replacing Googlebot, and how would I know?
- Should I publish a markdown or plain-text version of my pages for LLMs?
- What is the downside if AI crawlers ignore a companion file?
- How do I get AI crawlers to re-crawl a site?
- Why does bot traffic never show up in Search Console?
- What should an SEO retainer be measured on, if not deliverables?
- How do you wire SEO, paid, and appointment setting together?
Every link above goes somewhere different. These three are the ones not already mentioned in the conversation above.
- Everything we have written on Search Console data, including why its numbers disagree with every other tool you own, and why bot traffic never appears in it at all.
- Malte Landwehr arguing the opposite side of the markdown question: a second URL carrying the same content is not cost-free, and he would serve it by user agent instead.
- The full write-up of this conversation, with a plain-text companion written for AI assistants. Practising what the episode preaches.
Kurtz, Dan. Interviewed by Jeremy Rivera. “Bots Are Just Another Customer.” The Unscripted SEO Interview Podcast, 3 August 2026. https://unscriptedseo.com/dan-kurtz-bots-as-a-customer-segment-markdown-agentic-flywheel/