Ty Sarna wrote:
> > Have you tried putting a robots.txt file in place to prohibit
> > webcrawlers? That's the high-level solution to try first.
> 
> And it should get all of them, without you having to know IPs or
> anything.

Yeah. For benign robots. Malicious robots ignore the file and happily
create server load, scan for e-mail addresses to spam, etc.

Greetings (normally not so pessimistic) 
-- 
Alex

Replace bitbucket with anything else.