One Word Changed Every Answer: Ben Wills on Testing LLMs Directly

August 19
39 mins

Episode Description

Ben Wills ran the same hotel-recommendation prompt over and over and changed exactly one word in it: the car he was flying to Los Angeles to pick up. Honda Civic, BMW, Ferrari. The hotels moved with the car, and they moved across every ChatGPT model and every level of thinking he tried.

Ben started in SEO in 2001 at KeywordRanking alongside Andy Beal and Garrett French, co-founded the link building company ontolo with Garrett, then disappeared into 13 or 14 years of pure engineering. Large-scale crawlers and scrapers written in C by hand. Embedded firmware for an ESP32 inside high-end audio equipment. He is back in marketing now, building AI search visibility tooling at OppAlerts, and his argument is that LLMs finally hand SEOs the thing search engines never did: a fast, direct, testable feedback loop.

In this episode

  • The one-word experiment: hold the prompt constant, swap the car, and watch the hotel tier move with it
  • Why you can test an LLM but never could test a search engine: no waiting for the index, no personalization to strip out
  • The persona is a ranking factor: 100 industries, 10 personas each, 1,100 personas across the July analysis
  • Backlinks, ranking strength and LLM visibility: a decent correlation, with a full page of caveats in front of it
  • How Ben actually uses a correlation: as the anecdotal evidence that tells you which experiment to run
  • Isolating the model: thinking off, fan-out off, then turn each one up a step at a time
  • AI slop as an accountability question: producer standards versus consumer appetite, and whether his own reports qualify
  • Inside OppAlerts: historical PageRank from Common Crawl, news-specific link gap analysis, 20,000+ taxonomy categories
  • The Yandex source leak: a week in a Slack channel with Mike King and Russ Jones, and what the category vectors showed

Chapters

  • (00:00) Getting back into the industry after 13 years of engineering
  • (01:32) AI is taking your tasks, not your job
  • (02:58) The one-word experiment: Honda Civic, BMW, Ferrari
  • (05:42) Why you can test an LLM but never could test a search engine
  • (06:39) Twiddlers, patents and the black box
  • (08:22) Ann Smarty, fan-out queries and the return of search principles
  • (10:23) 100 industries and 1,100 personas: the July correlation study
  • (12:45) Correlation is not causation, and Ben's own asterisks
  • (14:29) Isolating the model: thinking levels off, fan-out off
  • (16:25) Engineer brain vs. marketer brain vs. business brain
  • (20:49) AI slop and where the accountability sits
  • (25:03) KeywordRanking, ontolo, pure C crawlers and ESP32 firmware
  • (27:46) news.oppalerts.com and building marketing tools
  • (29:32) Historical PageRank, contact data and campaign personas
  • (31:30) News link gap analysis and 20,000 taxonomy categories
  • (33:36) Vectors, the Yandex leak and CatBoost
  • (37:01) Homework: go build a search engine
  • (38:46) Where to find Ben Wills

Notable quotes

"One of the most important things for people to understand about the prompts is how one word can completely change the results, the response that you get back."
- Ben Wills
"The search principles are gonna come right back. You gotta have good quality content. There's gotta be a lot of it distributed across the web. You gotta have backlinks."
- Ben Wills
"Ann Smarty said ... the fastest way to get into the LLM responses is via SEO. And that's the highest leverage piece that you have to get in the responses."
- Ben Wills
"You'll learn more about search in that hour and a half than anything else."
- Ben Wills

Resources & links

About the show

Unscripted SEO is hosted by Jeremy Rivera. Full episode archive at UnscriptedSEO.com.

See all episodes