The New Google AI Crawler: What You Need to Know

The Google AI crawler which is being searched for is Google-CloudVertexBot. The site will be crawled only on the request of the site owner, to create Vertex AI Agents, and as per the Google documentation, it has no impact on Google Search or on any other Google product. Googlebot will continue to index the site. There is no need to make changes to your SEO due to this crawler.

Googlebot alone accounted for 4.5% of the HTML request traffic on the Cloudflare network in 2025, compared to 4.2% by all other AI bots combined (Source).

Search Engine Journal announced on August 21, 2024 that Google had included Google-CloudVertexBot in its crawler documentation, so the “new” crawler is actually over two years old.

This guide will explain what the crawler does, how it differs from Googlebot, and what to check in your robots.txt.

What Is Google’s AI Crawler?

Google identifies the bot as Google-CloudVertexBot, while Google documents that the crawler crawls sites based on requests from site owners to build Vertex AI Agents (Source). 

Vertex AI is the Google Cloud platform for building applications using AI technology, and the crawling retrieves pages to be used by a customer’s agent on that particular customer’s website.

Key Differences From Other Google Crawlers

Googlebot creates search indexes for Google and crawls the web according to its own schedule. Google-CloudVertexBot crawls on demand and is a product of its own.

These are both common crawlers, and according to Google, all common crawlers follow the robots.txt protocol when crawling automatically. The crawler that Google uses for search is Googlebot, which also feeds Discover, Images, Video, and News.

What the Crawler Does Not Do?

Google’s documentation states that Google-CloudVertexBot is a bot that targets Vertex AI Agents only and has no effect on Google Search or any other product, making it a service crawler for just one Google Cloud product.

This takes away the reason behind many claims that you may have come across, which include claims about how the crawler helps in indexing, uses machine learning to analyze content for ranking, or even indicates an algorithm update.

Implications for SEO

Your search rankings do not depend on this crawler. No SEO change is needed. Keep doing the work that earns rankings: clear answers, accurate facts, fast pages, and crawlable internal links, as covered in our 10 best SEO practices of 2026 guide.

Key Features of the Google-CloudVertexBot Crawler

Four details from Google’s documentation explain how this Google crawler behaves.

• Request-based crawling: Google says site owners request the crawls that build Vertex AI Agents.
• User agent token: Google-CloudVertexBot.
• Robots.txt: Google says its common crawlers always obey robots.txt rules when crawling automatically, so a Disallow rule for this token stops the crawls.
• Scope: Vertex AI Agents only, with no effect on Google Search.

According to Search Engine Journal, advanced indexing of websites, which is the Vertex AI service that employs this crawler, necessitates domain verification and indexing quotas, thereby fitting into the site owner request model.

It should be clear that if you see a visit by this user agent, it means that the site owner requested a crawl by this crawler; therefore, you should inquire within your team about Vertex AI usage before considering it unknown traffic.

Impact on SEO and Indexing

Changes in Indexing Priorities

Indexing priorities do not change because of this crawler. Google states that Google-CloudVertexBot has no effect on Google Search, so your content, structured data, and page speed matter for the same reasons as before.

Google Crawlers Compared

The table below lists five Google crawlers and what each affects.

The first four rows come from Google’s common crawlers documentation, the AdsBot row comes from Google’s special-case crawlers documentation, which says special-case crawlers may ignore robots.txt rules, so a robots.txt block is not a security control.

How to Optimize for Google Crawlers

• Keep the pages you want indexed open to Googlebot in robots.txt, and test any change with the robots.txt report in Search Console before you publish it.
• Link every important page from at least one crawlable internal link.
• Keep your XML sitemap current.
• Use structured data that matches the visible content of the page, and validate it with the Rich Results Test before release.

What This Means for Webmasters?

Practical Steps for Adapting

Let’s start with robots.txt, in which you define rules for each Google crawler token.

Google imposes a robots.txt size limit of 500 KiB, ignores the content beyond that limit, and caches the file for up to 24 hours, so give yourself at least a day to see the effect of a rule change (Google Search Central).

The robots.txt file of Content Whale has one token for all crawlers, allows the website, and disallows UTM links, paginated archive pages, and some internal directories, without any specific rules for Google-Extended or Google-CloudVertexBot tokens.

Choose Google-Extended intentionally: a Disallow rule for that token means Google won’t use your crawled content to train future Gemini models or provide grounding, and Google states that it doesn’t impact your indexing in Google Search.

Leave Googlebot allowed on every page you want in search results.

Security Considerations

Google notes that user agent strings can be spoofed, so a request that claims to be Google-CloudVertexBot may come from anyone. Verify suspicious traffic with Google’s guide to verifying Google crawlers before you block or allow it.

Keep private pages behind a login, because a robots.txt rule controls crawling only, and Google may still index the URL of a disallowed page.

Final Thoughts

Google-CloudVertexBot is the Google AI crawler, a request-based crawler for Vertex AI Agents that Google says has no effect on Google Search, so it gives you no reason to change your SEO plan, rewrite your content, or add markup for it.

Review your robots.txt once, decide on Google-Extended, and verify any traffic that claims to be a Google crawler. Document each decision.

Connect with Content Whale for an SEO content plan that follows how Google crawls and ranks pages.

FAQs

Why is the Google AI Crawler important?

Google-CloudVertexBot crawls websites upon request of the owners for the creation of Vertex AI Agents while Googlebot crawls for Google search.

What makes the Google AI crawler different from Googlebot?

Googlebot crawls pages for inclusion into Google’s search indexes. Google-CloudVertexBot crawls only on request for one particular product and Google claims it has no effect on Google Search according to its documentation.

Will Google AI Crawler impact my page rank?

Not necessarily. According to Google documentation, Google-CloudVertexBot has no effect on Google Search.

What can I do to get ready for the Google AI Crawler?

Nothing specific. Keep your pages crawlable for Googlebot and review your robots.txt for the tokens you care about.

Is the Google AI crawler the same as Google-Extended?

Not quite. Google-Extended is an independent robots.txt directive that determines if Google can leverage the content that it has crawled to train its future Gemini models and to provide grounding, whereas Google-CloudVertexBot is an independent crawler.

Which Google crawler builds the Google search index?

It’s Googlebot. It gathers content for creating Google search indexes and serves as a source for Discover, Images, Video, and News.

What is a Google Web Crawler?

Google Web Crawler is software that fetches webpages for a Google product. Googlebot creates Google search indexes, and there are additional Google crawlers for Ads, testing, and AI products.