robots.txt Generator
June 7th, 2007
In case you are still not familiar with the concept of robots.txt and how to use it to avoid possible block to certain important spider activities or simply you don’t know how to build one…
This site is the 1st actual site that took the liberty to provide a service to generate a fully working robots.txt from your actual site structure… it works in a different way with CMS driven sites that static sites.
www.clickability.co.uk/robotstxt.html
The idea is in the steps to create it and what to allow or disallow. It would be a great idea if this can be a built as a plug-in for Wordpress that can easily lists all the content of your blog and then you can select what levels to disallow or control!
Leave a Reply