Re: robots.txt: Good, Bad, Ugly?

caveman@xxxxxxxxxxxxxxxxx wrote:
Do civilized modern web crawlers still use robots.txt? In a recent
debate a friend was suggesting that robots.txt should not be used
anymore since there are other means of authorizing/restricting access
to a web site. After the debate I was left with the impression that
only malicious individuals seek the contents of robots.txt.

Opinions?

All the reputable search engines honour the robots.txt file. The
malicious ones are the ones that ignore it, and yes, you do need other
means of restricting them.

--
Dave
mail da ve@xxxxxxxxxxx (without the space)
http://www.llondel.org
So many gadgets, so little time
.

Relevant Pages

  • Re: robots.txt: Good, Bad, Ugly?
    ... debate a friend was suggesting that robots.txt should not be used ... anymore since there are other means of authorizing/restricting access ... to a web site. ... As was said in the other response, "good" crawlers still honor it; ...
    (comp.os.linux.security)
  • Re: POTM: Re: The ID ONLY Hypothesis
    ... web site be added to the talk.origins web site. ... The problem with Sean is that his shtick is all evasion. ... Just because he will engage in debate, doesn't mean that the debate is ... that tries to defend the creationists propaganda line. ...
    (talk.origins)
  • Re: Id like a better understanding of the debate
    ... and I became interested in the Creation/ID/Evolution ... understanding of the debate. ... The web site associated with this newsgroup has ... go through the "Posts of the Month" and see what ...
    (talk.origins)
  • Re: To the writers of the Talk.Origins website
    ... then I would challenge you to an open debate on the subject. ... I have something up on the TO web site. ... What articles are suggesting what you claim? ... Support your alternative. ...
    (talk.origins)