" Difference Between Robots.txt and Robots Meta Tag

Difference Between Robots.txt and Robots Meta Tag

You are aware that if you want to hide something from Google, you use either robots.txt or robots ‘noindex’ and ‘nofollow’ tag. Pages that don’t add any value to readers or crawlers are more likely to be ignored by Google. Pages that can be marked as duplicate one are also gets devalued by Google. Web masters hide such pages from crawling with robots.txt or robots meta tag. But what is the difference between robots.txt and robots meta tag?

Difference between robots.txt and robots meta tags when it comes to indexing

You are likely to hide old or outdated files, unused files, folders, backup folders, zip folders by using robots.txt. Still, there will be a possibility that it can be seen on Google. How? Let’s say, you have hide a unwanted webpage on your site with ‘disallow’ command. But this page is linked from some other article or pages on your site which is crawled and indexed by Google, then Google will able to find your disallowed or blocked page with the help of some other source.

In other words, robots.txt file just hides the file and tries to make sure Google doesn’t crawl it. It doesn’t prevent Google from indexing, if Google finds that file or page from somewhere else. For example, as I said; if it is mentioned with link on some other indexable and crawl-able webpage.

Syntax for robots.txt –

User-agent: *
Allow: /
Disallow: /old_file.html
Disallow: /unused-folder/
Disallow: /backup-file.zip

On the other hand, robots meta tag with nofollow and noindex attributes on the page prevents Google from indexing it and from showing it on Google. Though Google can see the page, it doesn’t crawl it and submit its content for indexation. Google doesn’t pass any ranking or SEO value to it.

Syntax for robots meta tag –

<meta name=”robots” content=”noindex,nofollow”>

With robots meta tag, even after crawler encounters with some page that you don’t want to indexed will not be visible on Google.

Conclusion, robot.txt doesn’t show files to Google for crawling, but if Google encounters with them, it might indexed that file on Google. However, robots meta tag with ‘noindex’ and ‘nofollow’ attributes doesn’t index files even after crawling.

Profile photo of Prabodhan Patil

Prabodhan Patil

Working with Indigo Consulting (a Publicis Groupe Company) since 2015 - helping businesses to grow. Specialist in Search Engine Optimisation, SEM , SMO strategies. Expert in techniques and strategies acceptable by the search engine industry.