On 12/5/06, Gilbert Fernandes <gilb@nerim.net> wrote:
> insert this into a robots.txt file :
>
> User-agent: *
> Disallow:/
>
> upload to the root folder of your web server.
>
> most will comply but some wont. you can either insert rules
> into your firewall to only allow your friends (if they have
> static ip for example) or require authentification.

Ok, I did this already just to see who complies and who doesn't :)

But the part of my question that I really want an answer to is, when I
find some bot hitting my web server, is it best to block it by raw IP,
fqdn, or just the domain? How do I make this decision? Seems like raw
IP or fqdn could change, because what I see is a whole list of
hostnames that all have the same domain name, and I assume these could
change at any time. If I block the entire domain, I don't anticipate
these guys being back, but I'm blocking things pretty broadly at that
point.

Advice?

Thanks.

Andy