Subject: Re: blocking bots
To: None <netbsd-users@NetBSD.ORG>
From: Ty Sarna <email@example.com>
Date: 01/07/2001 17:03:17
In article <firstname.lastname@example.org>,
Nathan J. Williams <nathanw@MIT.EDU> wrote:
> <email@example.com> (Erik Huizing) writes:
> > I noticed yesterday that someone (www.webtop.com) is indexing my server. I
> > was wondering if there's any way to put a stop to this. I don't have any
> > sensitive information on my server, but its more of an annoyance that this
> > is happeneing (all they're really getting is a copy of my apache
> > documentation).
> Have you tried putting a robots.txt file in place to prohibit
> webcrawlers? That's the high-level solution to try first.
And it should get all of them, without you having to know IPs or