Why User-Triggered Fetchers Matter
Sometimes, a search engine’s crawlers don’t come to your website automatically. Instead, they are triggered by an action you or another user takes. These are called user-triggered fetchers. They don’t crawl your site for indexing like a regular Googlebot; instead, they fetch specific pages to help with tasks such as testing, verification, or previews.
User-triggered fetchers are important because they allow site owners and SEOs to:
- Test how Google sees a page (without waiting for a scheduled crawl).
- Verify site ownership in tools like Search Console.
- Check if structured data is working.
- Preview how content might appear in Google services.
Google’s User-Triggered Fetchers
Here are the main fetchers you may encounter:
Google Site Verification
This fetcher is used when you are verifying site ownership in Google Search Console. It is a one-time fetcher that only appears when you request it.
Google-InspectionTool
This fetcher runs when you use tools like the URL Inspection tool, Rich Results Test, or Mobile-Friendly Test. It fetches a page on-demand to check its status.
AMP Crawler
This fetcher is used to validate and preview your AMP pages. It ensures that your AMP pages are working correctly and are ready for a search engine’s index.
Safe Browsing
This fetcher fetches URLs to check if content is safe for users. It is a security-related fetcher that helps keep the internet safe for everyone.
AdsBot
This fetcher checks landing page quality for Google Ads. It ensures that your landing pages are working correctly and are relevant to your ads.
Key Things to Know
- These fetchers don’t crawl for indexing—they only appear when you use a Google tool or request a check.
- They are on-demand crawlers, triggered by specific user actions.
- Blocking them in
robots.txtcould break important Google services (like site verification or ads checks). - They don’t affect your crawl budget since they aren’t part of regular crawling.
If you want to learn about ongoing bots that do index your content, check our List of Google’s Common Crawlers
How to Verify a User-Triggered Fetcher
Just like with other Google crawlers, you can confirm if they’re genuine by checking their IP:
Step 1: Find the crawler’s IP address
Look at the IP address in your server logs.
Step 2: Do a reverse DNS lookup
Run a reverse DNS lookup on that IP. The result should point back to a domain ending in googlebot.com or google.com.
Step 3: Confirm with a forward DNS lookup
Take that domain name and look it up again to see if it resolves back to the same IP address. If it matches, the crawler is a genuine Google crawler.
