When we think about Google crawlers, most people only know about Googlebot, the main crawler for indexing web pages. However, Google also uses other crawlers that serve different purposes. One of these is Feedfetcher, a special crawler that retrieves content from RSS and Atom feeds.
In this, we will explain what Feedfetcher is, how it works, and why you should not block it in your robots.txt file.
What is Feedfetcher?
Feedfetcher is a Google user agent that collects RSS and Atom feeds when users add them to products like Google News or Google Podcasts. Unlike Googlebot, Feedfetcher does not crawl entire websites. Instead, it only fetches the feed content that users subscribe to or request.
This makes Feedfetcher a user-triggered fetcher rather than a traditional crawler. Its activity is based on user actions, not on Google’s indexing process.
User Agent Details
| User Agent | User-Agent String | Purpose |
|---|---|---|
| Feedfetcher | Feedfetcher-Google (+http://www.google.com/feedfetcher.html) | Retrieves RSS and Atom feeds for Google products and user subscriptions. |
Why Feedfetcher Matters
Even though Feedfetcher is not used for search indexing, it still plays an important role for site owners and SEO professionals:
1. Distributes Your Content to Google Services
If your site publishes an RSS or Atom feed, Feedfetcher ensures that updates are delivered to Google products where users subscribe to your content.
2. User-Centric Activity
Unlike indexing crawlers, Feedfetcher is triggered when a user adds your feed to a Google product. This means its activity depends on your audience.
3. Does Not Affect SEO Directly
Feedfetcher does not impact your site’s search rankings, but it does help in content distribution and visibility through feeds.
Best Practices for Feedfetcher
To make sure your feeds work correctly, follow these practices:
-
Do not block Feedfetcher in your robots.txt file. Blocking it will prevent Google products from accessing your feeds.
-
Maintain a clean and valid RSS or Atom feed so that content is fetched correctly.
-
Check server logs for
Feedfetcher-Googlerequests to monitor activity. -
Optimize your feeds with clear titles, descriptions, and structured data to improve how content appears in Google services.
