Re: robots.txt: Good, Bad, Ugly?

Dave {Reply Address In.sig} wrote:
Barton L. Phillips wrote:
Dave {Reply Address In.sig} wrote:
All the reputable search engines honour the robots.txt file. The
malicious ones are the ones that ignore it, and yes, you do need other
means of restricting them.

Could you expand on "other means". I use .htaccess and .htpasswd etc. but what else do you have in mind?

It depends on how sophisticated you want to get. I came across
http://www.neilgunton.com/spambot_trap/ when browsing one day and it
looked like fun.

This looks a lot like what Buck sent me in a zip file. I am examining both and may (probably more for fun than need) implement something (at least for a little while and look at the logs).

In reality I have not seen any problems relating to bots. I check my logs for user agents and look at who/what is reading my robots.txt and have not seen anything that alarmed me to date. However, buck's logic will give me a little bit better log to look at.

Thanks

--
----------------
Barton L. Phillips
Applied Technology Resources, Inc.
Tel: (818)652-9850
Web: http://www.applitec.com
.

Relevant Pages

  • Re: Log files filling disk up
    ... Try using NTBACKUP to create an online backup. ... "Dave Marshall" wrote in message ... I guess that's the way to clear logfiles etc ... > clear logs at will? ...
    (microsoft.public.exchange.admin)
  • MUni at Mammoth...
    ... and it was a very interesting and mostly fun ... *really* bad UPD I had trying to ride down ... some logs. ... It's also in the vid. ...
    (rec.sport.unicycling)
  • Re: All events showing in each event log
    ... Dave Patrick ....Please no email replies - reply in newsgroup. ... It also adds a new registry key in the same ... | other logs. ... | rebuilt themselves on reboot, ...
    (microsoft.public.windows.server.general)
  • Re: Roaming Profiles Setup Walkthrough?
    ... > What does YMMV stand for? ... > "Dave the Clueless" wrote in message ... >> may not be able to use most apps on the client he logs into. ... >> this was my solution to the roaming profile problem. ...
    (microsoft.public.windows.server.sbs)
  • Re: Logging SuperDuper! Backups
    ... I've been a SuperDuper! ... back and forths with Dave and Bruce, especially during the period they were ... Bruce, but my recollection was that only on a very few occasions did I need ... if the interface access to the logs can be maintained while satisfying ...
    (comp.sys.mac.apps)