Subject: Re: blocking bots
To: Erik Huizing <huizing@cpsc.ucalgary.ca>
From: Nathan J. Williams <nathanw@MIT.EDU>
List: netbsd-users
Date: 01/06/2001 19:22:44
<huizing@cpsc.ucalgary.ca> (Erik Huizing) writes:

> I noticed yesterday that someone (www.webtop.com) is indexing my server. I
> was wondering if there's any way to put a stop to this. I don't have any
> sensitive information on my server, but its more of an annoyance that this
> is happeneing (all they're really getting is a copy of my apache
> documentation). 

Have you tried putting a robots.txt file in place to prohibit
webcrawlers? That's the high-level solution to try first.

        - Nathan