SEO

Everything We Know About IndexNow So Far

We hope you’ve heard of IndexNow by this point.

We covered its announcement in October and in December I had Fabrice Cannell, Principal Program Manager for Microsoft Bing, at the SEJ show to talk about this revolution in how search engines discover new and updated content.

But what exactly is it, and what do you need to know about it for SEO?

Here’s what we know so far.

Why do we need the index now

Search engines have been discovering content the same way for a long time.

Way back in 1993, JumpStation It became the first Internet discovery tool to send bots to follow (or crawl) links from website to website and create a content map or index.

Within the next few years, other search engines adopted this process as a way to find new content.

This process is built into an algorithm that also takes into account content, relevance, and other factors – and search engine optimization as we know it was born.

It didn’t take long before SEO savvy web publishers realized the importance of indexing in securing high search engine rankings. In fact, without indexed content, a website will not generate organic traffic.

But here’s the catch: unless you’re a large, high-traffic site, it can take days or even weeks before a search engine crawls your site and indexes new pages. Heck, even if you’re a large e-commerce business or marketplace site, it can take weeks of engines to catch up on new arrivals or out-of-stock products, too — especially when you rely on discoverability with basic crawls.

Meanwhile, you may have lost important traffic to competitors.

To streamline this process, Microsoft Bing launched a new initiative in October 2021 called IndexNow.

So what is the index now?

IndexNow is an open source protocol that allows website publishers to be instantly indexed across participating search engines, and results updated based on the latest content changes.

Simply put, it is a simple ping command that notifies the search engines that the URL and its contents have been added, updated or deleted.

By eliminating the need for heuristic crawls, IndexNow seeks to improve the overall efficiency of the Internet.

How does IndexNow work?

There are two different ways that search engines obtain index data: pull and push.

Pull indexing occurs when a search engine visits your site to request web pages and collects data from the server.

This is how search engines traditionally work.

Batch indexing is when a web publisher or content management system notifies the search engine that pages have been added, deleted, or other changes made.

This is the really groundbreaking thing about IndexNow – it allows all URLs submitted to any IndexNow-enabled search engine to be submitted at the same time to all other search engines using the protocol.

You can read more about the technical aspects of how IndexNow works here.

What search engines are using the index now?

At the time of publication, Bing and Yandex were using IndexNow.

Reportedly, a handful of others are testing the protocol, according to an unnamed expert.

So far, DuckDuckGo has not announced any plans to certify IndexNow.

However, it should be noted that the privacy-focused search engine draws up to 400 sources as well as uses its own DuckDuckBot web crawler to compile its search results.

Considering that Bing and Yandex are among those sources, you can see updated content appearing in DuckDuckGo faster because of IndexNow as well.

Even Bing’s biggest competitor, Google, and the most important of all search engines, has confirmed that they will be testing the IndexNow protocol.

And while they haven’t gone so far as to announce whether or not they’ll adopt him, the fact that they’re giving him a try shows recognition of his potential.

Google’s top concerns about IndexNow seem to be related to sustainability and efficiency.

Google currently uses HTTP/2, which is an efficient underlying protocol for data transfer for more than half of all crawls, and may eventually decide that this is the best fit for their needs.

Or they may decide to build their own alternative if they decide an API-based approach is needed – and they’re not sure they’ll do that.

But not everyone is on board.

There are a lot of people in the SEO community who don’t see the need for this protocol, and instead claim that XML sitemaps actually accomplish what IndexNow is trying to do.

You can read more about the debate here.

How do you use the index now?

Notifying search engines of updates to your web content using IndexNow is a straightforward process:

  1. Generate an API key – This is served along with the URLs to ensure domain ownership. You can do this using an online key generator such as located here.
  2. Host API key – Your API key is hosted on the root directory in txt format.
  3. Send URLs with parameters You can either submit URLs individually or in bulk. Submit your main location as a parameter.
  4. Check your submissions – With the Bing Webmaster Tools portal, you can check and discover submitted URLs.

For examples and more in-depth instructions, you can visit IndexNow.com.

More advanced users can use Python for bulk indexing and URL submission automation.

This requires a basic understanding of Python syntax and knowledge of Python libraries and packages.

We have a handy step-by-step guide to walk you through the process here.

What tools sync with the index now?

Any website developer can use the process outlined here to take advantage of IndexNow, but what’s really interesting is that you don’t have to know HTML from HDMI to reap the benefits.

A number of large companies, including LinkedIn, MSN, and GitHub, are all planning migrations to IndexNow.

In addition, other search solution providers such as BotifyAnd OnCrawl, Onely, and Yext are in the process of adopting this protocol.

Content management systems (CMS) have jumped in, too.

WordPress already offers a Index is now plugin. Duda supports This proactive approach to web crawling, and Wix plans to integrate soon.

Bing recently announced Rank Math and All In One SEO integration with IndexNow as well.

Content Delivery Networks (CDNs) that support IndexNow include Cloud Flair And Akamai.

Through their own proxy servers, this will speed up data delivery and bring it closer to users.

What are the benefits of using the index now?

In addition to emphasizing the evolution of indexing from pull to push, the primary benefit of IndexNow for publishers is the elimination of the time between making updates and being discovered by search engines.

By allowing webmasters to notify all participating URLs with a single API call, search engine content discovery is simplified, which in turn makes the web more efficient.

This is beneficial for site owners, as the increased requirements on the server are reduced.

Search engine spiders no longer need to perform heuristic crawls to determine if a page has been refreshed.

This allows servers to run more efficiently without excessive site load, which in turn reduces energy usage.

And unlike an XML sitemap, which only redirects SEO crawlers to pages within your website, IndexNow allows you to report status code pages other than 200 to search engines.

Index now frequently asked questions

What happens after I submit the URL?

IndexNow transmits your changes to search engines, which in turn apply them to page rankings.

Using IndexNow does not guarantee that web pages will be crawled or indexed and it may take time for changes to be reflected.

How many URLs can I submit using IndexNow?

You can provide search engines with a list of up to 10,000 URLs with a single API call.

Will I rank better in search results if my content is indexed faster?

Yes, the faster your pages get indexed, the faster competition for high-ranking search engine sites starts.

What does all this mean?

The march of technology is relentless.

As such a vital part of the modern world, it is only natural that changes in the Internet, especially search engines, should be a part of it.

By shifting the process of indexing sites away from crawling and into something faster and more efficient, IndexNow appears to be on the cusp of a revolution.

And the fact that companies like MSN, GitHub, and WordPress have all joined seems to indicate that this protocol will soon become the norm.

More resources:

  • Bing announces All In One SEO and IndexNow integration
  • The index is now causing mixed reactions, including skepticism
  • Advanced Technical SEO: A Complete Guide


Featured image: ra2 studio / Shutterstock

Related Articles

Leave a Reply

Your email address will not be published. Required fields are marked *

Back to top button