Google may reduce the frequency of crawling web pages as it becomes more aware of crawling and indexing sustainability.
This topic has been discussed by the Google Search Relationships team, consisting of John Mueller, Martin Splitt, and Gary Ellis.
Together, in the latest episode of the Search Off the Record podcast, they discuss what to expect from Google in 2022 and beyond.
Among the topics they cover are crawling and indexing, which SEO pros and website owners say they haven’t seen much of over the past year.
This will be a major focus for Google this year as it aims to make crawling more sustainable by conserving computing resources.
Here’s what that will mean for your website and its performance in search results.
Crawling and indexing sustainability
Since Googlebot’s crawling and indexing happens by default, it’s not something you would think has an impact on the environment.
Ellis draws attention to this problem when he says that computing is generally not sustainable:
“…what I mean is that computing, in general, is not really sustainable. And if you think about Bitcoin, for example, Bitcoin mining has a real impact on the environment that you can actually measure, especially if the electricity comes from coal plants or other plants less sustainable.
We’re carbon neutral, because I don’t know until 2007 or something like that, 2009, but that doesn’t mean we can’t reduce our environmental footprint. And crawling is one of those things that we’ve been able to slice off some low-hanging fruit early on.”
Low fruits, in this case, indicate unnecessary web crawling. Such as crawling web pages that have not been updated recently.
How will Google make crawling more sustainable?
Ellis explains that web crawling can be more sustainable by reducing it Update crawls.
There are two types of Googlebot crawls: crawling to discover new content and crawling to refresh existing content.
Google is considering reducing the crawl for refreshing content.
Ellis continues:
“…one of the things that we do, and we may not need to do that very often, is refresh the crawls. Which means that as soon as we detect a document, a URL, we go and crawl it, and then, eventually, we’ll come back to the URL This… This is the crawl update.
And then every time we go back to that URL, this will always be the crawl update. Now, how often do we need to refer to this URL? “
He goes on to give an example of some websites that require a high number of refresh crawls for some parts of the site but not others.
A website like the Wall Street Journal is constantly updating its homepage with new content, so it deserves a lot of crawls for the update.
However, WSJ is unlikely to update its About page frequently, so Google doesn’t need to keep doing refresh crawls on these types of pages.
So you don’t have to go back there often. And a lot of times we can’t estimate this very well, and we definitely have room for improvement in our update crawls. Because sometimes it seems wasteful that we’re hitting the same URL over and over again.
Sometimes we get to 404 pages, for example, for no good reason or no apparent reason. And all of these things are basically things that we can improve on and then reduce our footprint even more. “
- See: Google has two types of crawling – discovery and update
If Google has to reduce update crawls, which they aren’t 100% sure of, here’s the impact it could have on your website.
What does a low crawl rate mean for your website?
There is a belief that a high crawl rate is a positive SEO signal, even if you don’t update your content as often as Google crawls it.
This is a misconception, Ellis says, as content will not necessarily rank better because it is more crawled.
Mueller:
“So I think this is kind of a misconception that people think if a page is crawled more, it will rank higher. Is it true that this is a misconception, or is this actually true?”
Ellis:
“It’s a wrong idea.”
Mueller:
“Okay, so there’s no need to try to force a re-crawl on something if it really hasn’t changed. You’re not going to get a better ranking.”
Again, it’s not confirmed that Google will reduce update crawls, but it’s an idea the team is actively considering.
If Google follows this idea, it wouldn’t be a bad thing for your website. More creep does not mean a better ranking.
Further, the idea is to know the pages Need Crawl refresh and pages that don’t. This means that the pages that you change often will continue to refresh and update in search results.
For more details on how Google plans to do this, listen to the full discussion in the podcast below (starting at the 2:40 mark):
Featured image: Alena Veasey/Shutterstock
