Subject: Re: blocking bots
To: Ty Sarna <>
From: Alex van Denzel <>
List: netbsd-users
Date: 01/24/2001 23:52:33
Ty Sarna wrote:
> > Have you tried putting a robots.txt file in place to prohibit
> > webcrawlers? That's the high-level solution to try first.
> And it should get all of them, without you having to know IPs or
> anything.

Yeah. For benign robots. Malicious robots ignore the file and happily
create server load, scan for e-mail addresses to spam, etc.

Greetings (normally not so pessimistic) 

Replace bitbucket with anything else.