Web Crawlers

I am looking for a newsgroup or forum that discusses best practices in
using methods to influence how web crawlers record a website.

Information will be appreciated,
Stan Hilliard
.

Relevant Pages

  • Re: Web Crawlers
    ... Stan Hilliard wrote in ... using methods to influence how web crawlers record a website. ...
    (comp.security.misc)
  • Re: Web Crawlers
    ... using methods to influence how web crawlers record a website. ... I have placed a robots.txt on my website. ...
    (comp.security.misc)
  • Re: Anyone speak Efe or Baka?
    ... and/or influence to make such research possible? ... Under About the Andaman Association on the Table of Contents page ... been paying him to produce his website: it's clearly a labour of love, ... but he's not exactly ignorant or "amateurish" in the current ...
    (sci.lang)
  • Re: Anybody willing to decipher Forth program
    ... Maybe it was Steele's influence. ... My website: http://p-cos.net ... Common Lisp Document Repository: http://cdr.eurolisp.org ...
    (comp.lang.lisp)
  • Grand Central Arena Sequel!
    ... I have the contracts in front of me for a sequel to _Grand Central Arena_, tentatively titled _Spheres of Influence_! ... Sea Wasp ... Website: http://www.grandcentralarena.com Blog: http://seawasp.livejournal.com ...
    (rec.arts.sf.written)