In article <mtuelygqkh7.fsf@all-night-tool.mit.edu>,
Nathan J. Williams <nathanw@MIT.EDU> wrote:
> <huizing@cpsc.ucalgary.ca> (Erik Huizing) writes:
> 
> > I noticed yesterday that someone (www.webtop.com) is indexing my server. I
> > was wondering if there's any way to put a stop to this. I don't have any
> > sensitive information on my server, but its more of an annoyance that this
> > is happeneing (all they're really getting is a copy of my apache
> > documentation). 
> 
> Have you tried putting a robots.txt file in place to prohibit
> webcrawlers? That's the high-level solution to try first.

And it should get all of them, without you having to know IPs or
anything.

See:

http://info.webcrawler.com/mak/projects/robots/norobots.html