Re: robots.txt: Good, Bad, Ugly?

From: "Barton L. Phillips" <barton@xxxxxxxxxxxx>
Date: Thu, 21 Sep 2006 17:17:47 GMT

Dave {Reply Address In.sig} wrote:

All the reputable search engines honour the robots.txt file. The
malicious ones are the ones that ignore it, and yes, you do need other
means of restricting them.

Could you expand on "other means". I use .htaccess and .htpasswd etc. but what else do you have in mind?

--
----------------
Barton L. Phillips
Applied Technology Resources, Inc.
Tel: (818)652-9850
Web: http://www.applitec.com
.

Follow-Ups:
- Re: robots.txt: Good, Bad, Ugly?
  - From: Dave {Reply Address In.sig}

References:
- robots.txt: Good, Bad, Ugly?
  - From: caveman
- Re: robots.txt: Good, Bad, Ugly?
  - From: Dave {Reply Address In.sig}

Prev by Date: Re: robots.txt: Good, Bad, Ugly?
Next by Date: Re: UDP Port 28711
Previous by thread: Re: robots.txt: Good, Bad, Ugly?
Next by thread: Re: robots.txt: Good, Bad, Ugly?
Index(es):
- Date
- Thread