Subject: Re: blocking bots
To: None <netbsd-users@netbsd.org>
From: Wolfgang Rupprecht <wolfgang@wsrcc.com>
List: netbsd-users
Date: 01/06/2001 17:28:20
huizing@cpsc.ucalgary.ca (Erik Huizing) writes:
> I noticed yesterday that someone (www.webtop.com) is indexing my server. I
> was wondering if there's any way to put a stop to this. I don't have any
> sensitive information on my server, but its more of an annoyance that this
> is happeneing (all they're really getting is a copy of my apache
> documentation). 
> 
> I've tried blocking them in my ipf.conf:
> block in quick on ep0 from 212.11.41.1/24 to any
> (this is the first rule in the file)
> I've also tried specifically blocking port 80, and tcp protocol, and the
> full /32 address, all to no avail. 

    $ host6 212.11.41.1
    212.11.41.1: IPv4 212.11.41.1 '' 'adsl-rg-2-1.pops.easynet.fr'

Are you sure you have that IP address right?  You might be being
indexed by a spammer on an adsl line.

I have this in my ipf.conf and can verify that it still works as of an
two hours ago.

        block in log quick from 24.0.0.203/32 to any

        Jan  6 15:24:40 capsicum ipmon[130]: 15:24:39.930863             tlp0 @0:9 b authorized-scan1.security.home.net,45216 -> c460058-a.frmt1.sfba.home.com,nntp PR tcp len 20 10240 -R IN 

On the other hand, I tend to treat web-crawling indexers as friendlies
as long as they wait a few seconds between requesting new pages.  Its
only the isp's scanners that I bother to block.  (I no longer know if
the clueful people are still running the show at my ISP or if they
have all vested and left.)

-wolfgang
-- 
       Wolfgang Rupprecht <wolfgang+gnus@dailyplanet.wsrcc.com>
		    http://www.wsrcc.com/wolfgang/
Coming soon: GPS mapping tools for Open Systems. http://www.gnomad-mapping.com/