Google launched a new desktop googlebot sometime after midnight GMT on February 29, 2024. The new crawler was first discovered in server logs by SEO and indexation researcher, Carolyn Holzman.
"Since I started observing and notating googlebot crawlers as part of the indexation research project, other than updates in the chrome builds of the bots, I have never seen a wholesale renovation of the crawler strings," said Holzman.
The new desktop google string appears like this - Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.6167.139 Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html).
Previously it was styled as this - Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/121.0.6167.139 Safari/537.36
The indexation research project is where Holzman has honed her expertise about Google's indexation system and the crawler agents associated with it over the past two and a half years.
The project began in August 2021 and is comprised of daily publishing a new content page with two keywords on it. The data consists of daily measurements of when a particular test page has completed its path through the indexation system. Other events such as search console page report updates as well as Google's various updates are overlayed on the measured data.
The indexation research is often a topic on "Confessions of an SEO" podcast where as host, Holzman often refers to something from the research. Season 4, Episode 7. Forced Indexing is the most recent podcast which features the topic of indexation.
An unexpected outcome of this research was the appearance of the Google Search Status Dashboard shortly after appearing on SEO Fight Club discussing the sub-systems of the indexation system.
Holzman replied, "Prior to the introduction of the dashboard, I was a lone voice in the room making declaritive statements about the indexation system's subsystems, crawling, indexing and serving. Initially, those were the three sections Google had on its dashboard and sometime later the "ranking" system was added which ultimately became the replacement page for the old list of all Google updates page."
Previously the results of her research where shared publicly as Crawl or No Crawl Reports in the form of daily videos on Youtube. In November 2023 Crawl or No Crawl was moved to Substack where members pay a monthly fee to receive insider information on Google's current state of its indexation systems.
Currently the testing revolves around indexation using the Google Indexation API. The purpose of this test phase is to confirm if requests made via the API move through the full indexation systems in the same way as manual indexation requests made via search console.
In addition to documenting the time it takes content to move through the system, the research data also notes when various crawler agents' chrome builds are updated. At the time of this announcement the latest observed chrome build for googlebots and inspectiontool crawlers is Chrome/121.0.6167.139.
"Sometimes people new to my research assume I'm talking about the version of Chrome of the browers on our computers when I speak about chrome build updates. Instead I'm referring to the version of chrome that is used by Googlebots to "take pictures" of our content, both html and javascript before turning it all into html to score it in order to rank it," Holzman continued.
At the end of October 2023, Google announced as part of the mobile first indexing intiative being complete was the semi-retirement of the legacy desktop Googlebot. Within the announcement was confirmation that "we'll continue to reduce our crawling with legacy desktop Googlebot as much as possible. This helps to save resources for site-owners, and for us."
Those interested in joining the Crawl or No Crawl Report to learn more about the implications of this change can visit Crawl Or No Crawl on Substack. Those with other inquiries or questions regarding indexation or problems with indexation can visit the American Way Media Index Detector page for more information. For seo consultations or questions of a forensic nature, please reach out here: American Way Media.
https://www.youtube.com/watch?v=rVGSIy02S7U
###
For more information about American Way Media, contact the company here:
American Way Media
Alison Kane
512-853-9484
akane@americanwaymedia.com
2309 Broughton Ct Suite B
Austin, TX 78727
