greylisting multiple mail servers, greylisting with SPF, challenge response

["Jeremy C. Reed" <jreed%isc.org@localhost>]

I am curious if any of you still use greylisting?

I have been using spamd for around a decade. Using greylisting helps me 
block around 94.5% of spam senders. But over the past few months it has 
become too difficult to manage. The main reason is that a lot of mail is 
being retried by too many mail servers. For example, from many servers 
under outbound.protection.outlook.com, bullet.mail.*.yahoo.com, 
mail-*.google.com, etc.  Greylisting just is no working because the 
tuplet is never (rarely) reused (i.e. different sending IP).

I workaround these by adding individual IPs or blocks to my pf rules to 
bypass the spamd (so goes direct to mail server). Some I gathered 
manually from parsing spamdb database and others from DNS SPF records. I 
also script getting some known servers also via SPF and add to a pf 
whitelist (to bypass spamd and go direct to mail server). I 
can automate updating the pf whitelist table from DNS SPF records, but 
that doesn't help with unknown senders.

I could try to make some script to attempt to look at spamdb greylist 
database to see if there is any others I should whitelist. An example of 
that is Yahoo. It doesn't have ranges defined in SPF but uses SPF's PTR.

I could use a different greylister than has SPF checks builtin. I 
understand that this is not the purpose of SPF, especially since 
spammers can use correct SPF and then bypass my greylisting.
I could do SPF check and still greylist first time to stop or punish 
some spammers (and legitimate mailers) at least one time by making them 
try again later.

Does anyone know of any research about what percentage of spammers use 
their own domains that have good SPF? (Maybe I can analyze my own 
collection.)

Or maybe I can extend or use a greylister that uses the network for the 
tuplet instead specific IP (but network would just be a guess). Or maybe 
the greylister uses the networks/IPs from the SPF (including its "ptr" 
support) for greylisting.

Now a problem I have with the many IPs and networks I already whitelist 
is that I get spam from them too. (For example I get spam from 
outbound.protection.outlook.com.)

In addition, I tarpit/blackhole IPs that send mail direct to some of my 
spamtrap email addresses.  This ends up tarpitting the same IPs that I 
receive legitimate email from. (Yes spam coming from legitimate 
servers!)

I also trapped IPs for trying last MX first but maybe that is bad idea 
and maybe I end up blocking legitimate senders. My research had shown 
this blocks approximately 59% of unknown senders. 

Currently my tarpit database has 1.14 times more IPs than my whitelist. 
(For a long time, it was only around 6 to 12% the size, but now more and 
more are tarpitted.)

Do you use greylisting? Spamtraps? SPF to create whitelists?

I still want to enable a challenge response system, but we need 
protocols to be created/extended so mail senders can understand that 
they are being challenged and require a response (so they can provide a 
friendly and understandable method for senders to verify, which may be 
like a sender using a micropayment, etc.).  Any of you using 
challenge-response to limit spam?

  Jeremy C. Reed

p.s. I noticed my spamd greylist database has 698631 entries in it. It 
doesn't seem to be cleaning up very quickly.