Re: robots.txt: Good, Bad, Ugly?

Allodoxaphobia wrote:
On Fri, 22 Sep 2006 01:16:27 GMT, Barton L. Phillips wrote:
In reality I have not seen any problems relating to bots. I check my logs for user agents and look at who/what is reading my robots.txt and have not seen anything that alarmed me to date.

Errr, ummm. Bad bots probably won't bother reading robots.txt.

Jonesy
Yes sorry that was not really what I was trying to say. You are right of course, and I do other analysis to try to see what "bad" bots are doing. I think buck's program might help as log files are quite hard to analyze for this type of violation.

--
----------------
Barton L. Phillips
Applied Technology Resources, Inc.
Tel: (818)652-9850
Web: http://www.applitec.com
.

Relevant Pages

  • Re: Saving passwords without prompting
    ... > Ralph Wade Phillips wrote: ... >> wait, don't finish reading this, go change it NOW. ... If he isn't, and the logs aren't available, I'd ... > If you know whomever you suspect is reading your mails, ...
    (uk.comp.homebuilt)
  • Re: Separating AD Log and Data Files...Benefits out weigh the costs?
    ... OS on RAID1 (a lot of reading and writing) ... For the DCs in your scenario it should be more than enough to have: ... OS, AD DB, AD LOGS, AD SYSVOL on RAID1 ...
    (microsoft.public.windows.server.active_directory)
  • Re: Having problem with my FRead in my program
    ... > I am having problem reading some data in a file! ... > Then everything under is another type of structure which contains logs ... > I have no problem writing and reading the first part, ... > fseek(ScaleFile, 0, SEEK_END); ...
    (microsoft.public.windowsce.embedded.vc)
  • Re: Firefox overtakes IE6
    ... By reading the logs of one particular site, ... then green must be a vague term. ... If web stats are 'vague', ...
    (uk.people.silversurfers)
  • Re: Saving passwords without prompting
    ... But that won't keep anyone else from accessing your account if they know what the password is. ... Unless a network admin somewhere along the mails route takes a look, or the POP3 box is brute forced,, your email should be pretty safe. ... If he isn't, and the logs aren't available, I'd suggest planting a link to a website that logs are available for. ... If you know whomever you suspect is reading your mails, it should be easy enough to get them to visit a particular link.. ...
    (uk.comp.homebuilt)