Join us today as we take a deep dive into the technical aspects of Robots.txt, Google's crawlers, Webmaster authority, and what you can do to control the creepy crawlers of the wide-wide web!
Yesterday
Why AEO Starts With Better SEO Fundamentals
AI Overviews now appear in 40% of Google searches, up from 6% in six months. Lucas Tieleman, co-founder of Optiversal, helps enterprise retailers turn static product catalogs into machine-readable content that performs across Google, ChatGPT, and Google Merchant Center. He breaks ... Show More
55m 42s
Aug 20
One thing AI still can't do well
AI agents still can't replicate implicit, high-frequency human communication around company strategy. Thenuka Karunaratne, founder of sunbeam, the autonomous SEO agent, and daydream, the AI-native SEO agency that identifies search patterns and creates content at scale, breaks dow ... Show More
1m 42s
May 2025
How Googlebot crawls the web
<p dir="ltr">In this episode of Search Off the Record, Martin and Gary from the Google Search Relations team take a deep dive into how Googlebot and web crawling work—past, present, and future. Through their humorous and thoughtful conversation, they explore how crawling evolved ... Show More
32m 41s
Aug 2024
Crawling smarter, not harder
In this episode of SOTR, John Mueller, Lizzi Sassman, and Gary Illyes talk about misconceptions around crawl frequency and site quality, what's challenging about crawling the web nowadays, and how search engines could crawl more efficiently. Resources: Episode transcript → https: ... Show More
40m 11s
May 2025
The Dangers of A.I. Flattery + Kevin Meets the Orb + Group Chat Chat
<p>This week we dig into the ways chatbots are starting to manipulate us, including ChatGPT’s sycophantic update, Meta’s digital companionship turn and a secret experiment run on Reddit users. Then Kevin reports back from the unveiling of a new eye-scanning orb. And finally, we’r ... Show More
1h 7m