Robots Meta Tags

Have you ever wondered how to tell a search engine not to look at a certain page on your site? That’s what a robots meta tags are for. It’s a small piece of HTML code that you place in a page’s header. It gives search engines instructions on how to handle that specific page.

For example, if you want to tell a search engine not to index a page, you would use this tag:

<meta name=”robots” content=”noindex”>

You can also use this tag to give search engines other commands, like not to follow the links on a page. You can add more directives by separating them with a comma.

What’s the Difference Between the robots.txt File and Robots Meta Tags?

You might get confused between the robots.txt file and a meta tag. Here’s a simple way to think about the difference:

• robots.txt file: This is a file for your entire website. It tells search engines where they are allowed to crawl.
• Robots meta tag: This is a tag for a single page. It tells search engines how to handle that page, like whether to index it or not.

The robots.txt file is like a fence around your property. A robots meta tag is like a sign on a door that says, “Do not enter.”

When You Should Use the Robots No-Index Meta Tag

<meta name=”robots” content=”noindex”>

You should use this tag for pages you don’t want to show up in search results. This is useful for a few reasons:

• Private Pages: You can use it to protect sensitive content, like a private member area or an exclusive offer.
• Thin or Duplicate Content: If you have pages that are too similar, you can use a noindex tag on one of them to prevent search engines from getting confused. 
• Confirmation Pages: You can use it to keep a confirmation or thank-you page out of search results.

How to Use Different Robots Meta Tags

There are a few other meta tags you might need to use.

How to use the Meta Robots “noindex,follow” Tag

<meta name=”robots” content=”noindex,follow”>

If you want search engines to not index a page but still follow the links on it, you would use this tag. This is useful for pages you’ve replaced but still have valuable links on them.

Meta Robots “noindex,nofollow” and “none” Tags

<meta name=”robots” content=”noindex,nofollow”>

Use this tag to tell search engines not to index a page and not to follow the links on it. The <meta name=”robots” content=”none”> tag has the same effect.

Meta Robots “noarchive” Tag Use Cases

<meta name=”robots” content=”noarchive”>

Search engines will often save a copy of a page. If you don’t want a search engine to save a copy of your page, you can use this tag. This is useful for pages that are updated frequently.

Skip the Snippet with the “nosnippet” Tag and the “data-nosnippet” Attribute

<meta name=”robots” content=”nosnippet”>

When your page shows up in search results, a short description, or snippet, appears below the title. If you don’t want a search engine to display a snippet for your page, you can use this tag.

“Noimageindex” Meta Tag

<meta name=”robots” content=”noimageindex”>

Use this tag if you don’t want the images on your page to be indexed by search engines.

“Notranslate” Meta Tag

<meta name=”robots” content=”notranslate”>

If you don’t want a search engine to offer to translate your page, you can use this tag.

What about the X-Robots Tag?

The X-Robots tag is used for files that aren’t HTML, like images and PDFs. It is placed in your website’s HTTP Response header. It is also a great way to give directives to a large group of files at once.