SEO is very buzzing word now and everyone wants to jump in to SEO Training. I will advise them to check all the details before joining any SEO Training Institute like Course Details, Placement Partners and Even Trainers who are going to teach them, Live Training and certification they are getting etc. Don’t feel that I am trying to promote our institute; this is generic advice for all. If you think that why you should Join DIIM for SEO Training then just read the title, we are focusing on all micro details like Robots.Txt, it is really small file but it will create a great impact on your SEO results. So here we go to know all aspects about Robots.txt

What is Robots.txt

For those who don’t know what is robots.txt, it is a text file which instructs search engine robots to crawl a page or not. Any CMS (content management system) based sites are having admin module online and that should not be crawled by Search Engines, using robots.txt you can block that part to not get it crawl.

Configure Rotots.txt

Robots.txt is not only for blocking content but it is having various use as follows:

  • You can block specific directory to get it crawled, even specific file can be blocked.
  • You can block particular search engine to stop the indexing of the website.
  • You can block specific file types to get it crawled by search engine eg. All PDF files.
  • You can pass sitemap (XML) location via robots.txt

Why you need proper Robots.txt

  • To have proper URL Canonicalization.
  • To avoid Duplicate content.
  • Drive traffic to proper pages
  • Decrease bounce rate.

If you site is on wordpress and you want to improve your robots.txt then check our robots.txt file.

I hope you have got some new dimension about Robots.txt, if you still have any doubt then kindly comment here. We will reply your queries. Don’t forgot to follow us on Facebook and Twitter You also can subscribe for our Newsletter to get auto update direct in your mail box.

Showing 2 comments
  • Prince Ramgarhia

    great article and surely gonna edit my robots.txt for better seo..

  • Amarnath Chakraborty

    Yes right but robots.txt also help google robots to crawl your website and index your webpages much faster ! You will also have a choice to disallow robots to access your secret file or folder from appearing in google!

    Nice info thanks..

Leave a Comment