This client created 1,500+ subdomains for insurance agents and were facing some serious issues getting visibility and visits from their new subdomains. Their request was to index all the pages on their subdomains and automate that indexing in the future, as they were under the impression that there are some APIs available to do so.
My initial concern is that the client is confusing “index” with high ranking. Page indexing is done automatically by Google when your site is crawled. Meaning, they gather information about your page(s) from their crawl and save it for future use for when it would be handy to serve to a user. There are many variables involved with how and when your site is crawled and indexed, for example, site ranking, visits, internal links, external links, standards compliance, and so on. Indexing and speed of indexing mostly come down to how Google accesses your pages, which means links internally on your site and externally on other websites. How Google serves it (ranking) comes down to relevance, semantics, value, etc.
Can you automate site indexing and all new pages created in the future? What APIs exist for us to do this?
I’m just going to say ‘no’ and leave it there. Google has stated that that automating access or ‘submission’ to them in any way is a definite ‘no’ in their book. So please don’t do it.
There aren’t any automated “indexing” APIs or tools to use that I would recommend, because that would require using the same automated ‘submission’ schemes or low quality link tactics that violate Google’s quality practices, and would do more harm. There are tactics that you can put into place to help get individual pages indexed quickly, but not all at once and in perpetuity. For instance, the fetch and render tool in Google Search Console, posting from Google My Business, building solid XML sitemaps and sitemap indexes, well structured and linked sites, social shares, backlinks, etc.
XML sitemaps and sitemap indexes are helpful to instruct Google how to crawl your site, but they’re only a small piece of the crawling and indexing puzzle. There are many different variables that determine how often Google crawls your site and how easy it is for Google to index your pages. For example, a well shaped site to maximize Google’s crawl budget by leading crawlers to your most valuable and important pages on your site will help.
If you’re looking for more information and details about this, I highly recommend reading Paddy Moogan’s guide to crawling, indexing and ranking. This is a pretty thorough guide, and will help you to put this process together in sequence.
