Skip to content

Tag: Google

Robots.txt demystified

A lot has already been written about creating and using the robots.txtfile. However, some of the articles often tend to be too cryptic or “techie” in nature, leaving a lot of newbies and tech unsavvy people totally overwhelmed with all that information.

Came across an article by Vanessa Fox that seeks to demystify robots.txtin a clear, easy to understand manner, with lots of examples and links. This is a must read for everyone who wanted to know more about robots.txt and the Robots Exclusion Protocol.