Re: robots.txt: Good, Bad, Ugly?

Allodoxaphobia wrote:
On Fri, 22 Sep 2006 01:16:27 GMT, Barton L. Phillips wrote:
In reality I have not seen any problems relating to bots. I check my logs for user agents and look at who/what is reading my robots.txt and have not seen anything that alarmed me to date.

Errr, ummm. Bad bots probably won't bother reading robots.txt.

Yes sorry that was not really what I was trying to say. You are right of course, and I do other analysis to try to see what "bad" bots are doing. I think buck's program might help as log files are quite hard to analyze for this type of violation.

Barton L. Phillips
Applied Technology Resources, Inc.
Tel: (818)652-9850