Sometimes you don’t want a page to be indexed by search engine robots. The reason could be as simple as you dont want people to find a page as it is not important. For example, I have a website portfolio on my website and I display all items on one page. I have put single portfolio item links with each item but I dont want search engine to index those subsequent pages at all. My SEO guy recommended this and I dont want to gointo more details about the SEO stuff here.
Hence what I want is to not let the search engine robots follow the single portfolio item links placed on portfolio page and not index the single items page itself.
You may know about the “noindex” and “nofollow”. Noindex does exactly the same it sounds. When Google bot or any other search engine bot crawls your website it detects all the links on a page and it lists all the in-site links and follow them one after another, once it has finished with current link. So adding a nofollow stops Google bot indexing that link for further crawl.
To instruct Google bot to not follow a link for crawling, you add a “rel” attribute with “nofollow” value
<a href="https://devarticles.in" rel="nofollow">Web Development Tips and Tricks</a>
Next thing is, how to stop a page being indexed from Google searches. For some reason Google bot may end up on a page which you dont want to be indexed. It could be an old link somewhere on your page which does not have a “nofollow” attribute value added or anything. The thing is you dont want it been indexed. Ok, there is another important trick.
To stop a page being indexed by Google bot you had a <meta> tag to <head> section of your web page. Here is the tag:
<meta name=”robots” content=”noindex” />
This meta tag takes multiple values so you may instruct robots to do any of the following while setting “robots” meta tag.
<meta name="robots" content="noindex, follow">
<meta name="robots" content="index, nofollow">
<meta name="robots" content="noindex, nofollow">
The value of content attribute is quite self explainatory.
