If you're looking for information about robot.txt files you should read "robots.txt Adventure" at nextthing.org. The article is packed with original research and features some interesting facts and figures.
As for my own robots.txt, it's a pretty standard two line affair:
User-agent: *
Disallow:
CrunchBang - A weblog & personal resource about coding, design & GNU/Linux.
Subscribe to RSS feeds | What am I doing? | Interesting stuff elsewhere on the interweb
Ubuntu is an operating system consisting of free and open source software. With Ubuntu you can surf the web, read email, create documents, spreadsheets and more!
Add Your Comment
Use the form below to add your comment. Markdown syntax is available. Note, comments are moderated by me for spam filtering. Alternatively, feel free to contact me privately.