A lot has already been written about creating and using the robots.txtfile. However, some of the articles often tend to be too cryptic or “techie” in nature, leaving a lot of newbies and tech unsavvy people totally overwhelmed with all that information. 

Came across an article by Vanessa Fox that seeks to demystify robots.txtin a clear, easy to understand manner, with lots of examples and links. This is a must read for everyone who wanted to know more about robots.txt and the Robots Exclusion Protocol.

Thanks, Vanessa.

Atul KarmarkarInternetGoogle,robots.txt,SEOA lot has already been written about creating and using the robots.txtfile. However, some of the articles often tend to be too cryptic or 'techie' in nature, leaving a lot of newbies and tech unsavvy people totally overwhelmed with all that information.  Came across an article by Vanessa Fox that seeks to...Around Pune, Technology, Startups, Investing, Food