Subject: Re: blocking bots
To: None <netbsd-users@NetBSD.ORG>
From: Ty Sarna <>
List: netbsd-users
Date: 01/07/2001 17:03:17
In article <>,
Nathan J. Williams <nathanw@MIT.EDU> wrote:
> <> (Erik Huizing) writes:
> > I noticed yesterday that someone ( is indexing my server. I
> > was wondering if there's any way to put a stop to this. I don't have any
> > sensitive information on my server, but its more of an annoyance that this
> > is happeneing (all they're really getting is a copy of my apache
> > documentation). 
> Have you tried putting a robots.txt file in place to prohibit
> webcrawlers? That's the high-level solution to try first.

And it should get all of them, without you having to know IPs or