In this post, we will discuss the usage and advantages of the robots meta tag, which is available within the head tag of an HTML page. HTML can be generated either by writing plain HTML in an HTML file or dynamically generated by a website.
What is robots meta tag?
It is an optional tag available within the head tag of every web page. It directs the search engines and other services to crawl or index the web page.
What are the advantages of using robots meta tag?
- It is part of The Robots Exclusion Protocol (REP) and instructs search engines to crawl and index the page.
- It increases the site ranking.
- It's used for individual pages as compared to robots.txt file explained in How To Use SEO Friendly robots.xt File.
- We can use both robots.txt file and robots meta tag on the same page. The instructions available in robots.txt will be preferred by some of the search engines for noindex in case meta tag allows indexing and vice versa.
- The meta tag can be used to completely block the individual page from indexing, where robots.txt allows indexing of all pages.
What are the possible values for robots meta tag?
We can use the combination of below-listed values for robots meta tag:
- index - Allows search engines to crawl and index the page.
- noindex - Instructs the search engines and robots to avoid crawling and indexing. The search engines won't show the page in their search results.
- follow - Instructs search engine to further crawl the links available on the page. Though it might not be true always and search engines may or may not crawl the links.
- nofollow - Disallow search engines to follow other links and crawl only the current page. It depends on search engines to respect the page preference.
- noarchive - Disallows search engines to show the cached copy of the page in search results.
What are the possible combinations for robots meta tag?
We can combine the possible values for robots meta tag as listed below:
index, follow or all - It allows indexing the current page and links available within the page. It can be used as shown below:
<meta name="robots" content="index, follow"> OR <meta name="robots" content="all ">
noindex, nofollow or none - It completely blocks the search engine and robots to crawl and index the page. Though certain malware won't respect these values. It can be used as shown below:
<meta name="robots" content="noindex, nofollow"> OR <meta name="robots" content="none">
This is how can use the meta robots tag most efficiently.
