Join us today as we take a deep dive into the technical aspects of Robots.txt, Google's crawlers, Webmaster authority, and what you can do to control the creepy crawlers of the wide-wide web!
Oct 6
Onsite vs offsite as AI search continues to drive change
Off-site signals now outweigh on-site control in AI search. Siiri Lee, Senior Director of Global Direct-to-Consumer at MANSCAPED, brings two decades of experience scaling DTC and omnichannel ecommerce through conversion optimization, personalization, retention, and loyalty progra ... Show More
1m 59s
May 2025
How Googlebot crawls the web
<p dir="ltr">In this episode of Search Off the Record, Martin and Gary from the Google Search Relations team take a deep dive into how Googlebot and web crawling work—past, present, and future. Through their humorous and thoughtful conversation, they explore how crawling evolved ... Show More
32m 41s
Aug 2024
Crawling smarter, not harder
In this episode of SOTR, John Mueller, Lizzi Sassman, and Gary Illyes talk about misconceptions around crawl frequency and site quality, what's challenging about crawling the web nowadays, and how search engines could crawl more efficiently. Resources: Episode transcript → https: ... Show More
40m 11s
May 2025
The Dangers of A.I. Flattery + Kevin Meets the Orb + Group Chat Chat
<p>This week we dig into the ways chatbots are starting to manipulate us, including ChatGPT’s sycophantic update, Meta’s digital companionship turn and a secret experiment run on Reddit users. Then Kevin reports back from the unveiling of a new eye-scanning orb. And finally, we’r ... Show More
1h 7m