Meta Robots Tag | How to Use the Meta Robots Tag for SEO

Using the Meta Robots Tag for SEO

You can use the meta robots tag to control where and how Google and other search engine crawlers move around your website and pass link juice from page to page. If that sounds familiar to another text file on your website, stay tuned for more...

In this guide, we'll cover

1. What the meta robots tag is and why it matters
2. How you use the robots meta tag for SEO
3. The benefits of using the robots meta tag

What Is The Robots Meta Tag and Does It Matter?

The robots meta tag is an HTML tag that goes the head tag of a page and provides instructions to bots. Like the robots.txt file, it tells search engine crawlers whether or not they are allowed to index a page.

To find the meta robots tag on a page, simply right click on a web page, click "View Source" and then do a Find for "robots". It will look something like this:

<meta name="robots" content="noindex" />
<meta name="googlebot" content="noindex" />
<meta name="googlebot-news" content="noindex" />
<meta name="slurp" content="noindex" />
<meta name="msnbot" content="noindex" />

In this example, the top line gives applies to all search engine bots while the 4 lines after that give apply to specific user-agents. In this specific example, the meta robots tag is telling search engines not to index the page. However, the bots are free to follow the links they find on the page.

The robots meta tag matters because it adds an extra layer of protection to the robots.txt file. When a crawler follows an external link and lands on one of your pages, it can still crawl and index that page because it hasn't seen the robots.txt file.

The robots meta tag prevents this crawling and indexing from happening.

                           How is meta robots different from robots.txt?                      

             The meta robots tag applies only to the page that contains the tag. Robots.txt files apply to your entire website.          

How does the robots meta tag work?

As you can see, there are two parts of the tag: name="" and content="".  

Read the guide to search engine crawlers and crawling to learn more about how they work.

The name part specifies the user-agent of the bot you 're instructing, just like the user-agent line in a robots.txt file. Unlike robots.txt, you don 't use a wildcard character to include all bots. For that, you just write "robots".

Hence the meta robots tag name.

The second part, content="" is where you tell the bots what to do.

What robots meta tag values are there?

There are a lot of different values you can add to the content field in the robots tag. Each of these values does something different:

• Index: Tells search engines to index the page. This might seem pointless at first glance since  "Index" is the default, but can be useful if you want only a particular group of search engines to index the page.
• NoIndex: Tells search engines not to index the page so it won 't appear in search results.
• NoImageIndex: Tells search engines not to index the images on a page. However, if someone adds that image somewhere else on the web, Google will still index it and show it in image results.
• None: This works as a shortcut for "noindex, nofollow". It tells search engines to ignore the page and pretend they 've never seen it.
• Follow: Tells search engines to follow the links it finds on the page. Like with "Index", this is the default status when a bot doesn't find a meta robots tag that applies to it.
• NoFollow: Tells search engines not to follow any links on a page at all. You can add this value to an individual link as well.
• NoArchive: Tells search engines not to show cached copies of the page.
• NoCache: The same thing as  "NoArchive" except used by MSN/Live.
• NoSnippet: Prevents search engines from showing a snippet for this page in search results. It also prevents them from caching the page.
• NoTranslate: Tells search engines not to offer translated versions of the page in search results.
• Unavailable_after: Tells search engines not to display the page in search results after a certain date.
• NoYDir: This tells the search engine not to use the Yahoo! Directory page description in the search snippet.
• NoODP: Prevents search engines from using the description of the page from DMOZ in the search snippet. ODP is the community that runs and maintains the DMOZ directory.

Those last two values  "NoYDir and NoODP" don 't have a use these days. Neither the Yahoo! Directory or DMOZ exist anymore. However, you might still see them while out and about on the web.

Complicating things a bit here is the fact that not all search engines support all values. So here 's a handy table that breaks that down:

 Which values are recognized by which search engines?            Value      Google      Bing      Yandex                index      Yes      Yes      Yes              noindex      Yes      Yes      Yes              none      Yes      Doubt      Yes              noimageindex      Yes      No      No              follow      Yes      Doubt      Yes              nofollow      Yes      Yes      Yes              noarchive/nocache      Yes      Yes      Yes              nosnippet      Yes      No      No              notranslate      Yes      No      No              unavailable_after      Yes      No      No              noodp      No      No      No              noydir      No      No      No    

Using commas, you can create multi-directive meta tags instead of creating one tag for each directive. In fact, you 'll see this quite often as many robots meta tags use  "noindex, nofollow" values:

<meta name="robots" content="noindex, nofollow" />

How Do I Use the Robots Meta Tags for SEO?

Making sure certain low-value pages stay out of Google 's index and search results is just as much a part of SEO as getting pages into search results. Noindexing low-value pages can help increase what 's known as your site 's  "crawl demand", which can help get your site crawled more often.

Meta robots also adds an extra layer of protection for pages you 've blocked via your robots.txt file. Those pages can still wind up indexed if Google lands on it from an external backlink. Noindexing the page will prevent this.

Using the robots meta tag to stop a page from being indexed and links from being followed looks like this:

<meta name="robots" content="noindex, nofollow">

The noindex and nofollow values are the two most-used values used in the meta robots tag. However, the other values listed above also have SEO value:

• NoImageIndex: This instructs search engines to not crawl the images on a page.
• None: This is equal to using "noindex, nofollow" combined into one value. Search crawlers won't index the page or follow any links.
• NoArchive: Stop search engines from showing a cached version of your page. Make sure people are always seeing the latest version of your content. MSN/Live uses "NoCache" instead of "NoArchive".
• NoSnippet: This stops search engines from displaying your site's snippet in search results and from showing a cached version of the page.

If the whole point of SEO is getting pages into search results, how on Earth does the meta robots a page help SEO?

1. It prevents any private files or folders from being indexed and displayed in search results. It 's generally advisable not to publish this content to your site at all, or to password protect it. However, if for some reason you have to put it on your site, the robots meta tag will keep it out of Google.
2. It helps search engines to crawl your site more efficiently. Search robots have limited crawl budgets, so they could theoretically spend all their time crawling pages you don 't really care about ranking, while ignoring your most important ones. Blocking indexing of these unimportant files will help guide crawlers to your more valuable pages.
3. If you 've got a page that 's acquired a lot of link juice, but you don 't want it indexed, use the follow directive to pass that link juice to other pages of your site.

While it 's never advisable to publish sensitive information on your website, it sometimes happens. Blocking these URLs via robots.txt is telling anyone who reads it that they should take a look at those pages. Adding  "noindex" to a meta robots tag will keep that page out of search results without listing it where someone could find it.

The most important part of using the robots meta tag is to make sure you 're using it correctly. It 's not unheard of for an entire site to get deindexed because someone accidentally added the robots noindex tag to the entire site. So understanding how the robots meta tag works is absolutely vital for SEO.