Subject: Re: blocking bots
To: Erik Huizing <>
From: Nathan J. Williams <nathanw@MIT.EDU>
List: netbsd-users
Date: 01/06/2001 19:22:44
<> (Erik Huizing) writes:

> I noticed yesterday that someone ( is indexing my server. I
> was wondering if there's any way to put a stop to this. I don't have any
> sensitive information on my server, but its more of an annoyance that this
> is happeneing (all they're really getting is a copy of my apache
> documentation). 

Have you tried putting a robots.txt file in place to prohibit
webcrawlers? That's the high-level solution to try first.

        - Nathan