70,000 TLB shootdown IPIs per second

[Thor Lancelot Simon <tls%panix.com@localhost>]

I have been doing some testing on a fileserver recently donated to TNF.

The system has an Areca 1280 controller (arcmsr driver) with a single
RAID6 volume configured on 12 disks; 32GB of RAM; two quad-core Xeon L5420
CPUs.

I have tested under NetBSD-6 and NetBSD-current as well as the tls-maxphys
branch.  The test is a 'dd bs=2m of=/test count=65536".  Write throughput,
while acceptable (300-350MB/sec; 350-400MB/sec with tls-maxphys) is about
2/3 what I get on the same hardware under Linux.  Read throughput is less
good, 250-300MB/sec.

The filesystem is FFSv2, 32k block/4k frag, with WAPBL.

Watching systat while I do the dd tests, I see up to 70,000 TLB shootdown
IPIs per second.  Is this really right?  I am not sure I know how to count
these under Linux but I don't see any evidence of them.  Is there a pmap
problem?  I'm running port-amd64.

I have also seen some other odd things I'll detail elsewhere, but just for
a start, can anyone explain to me whether this number of TLB shootdowns
should be expected, whether each should really generate its own IPI, and
what the performance impact may be?

-- 
 Thor Lancelot Simon                                          
tls%panix.com@localhost

        It's very complicated.  It's very cumbersome.  There's a
        lot of numbers involved with it.