wm(4) interrupt params [was: CVS commit: src/sys/dev/pci]

[Simon Burge <simonb%NetBSD.org@localhost>]

[ Not sure if tech-kern or tech-net is better, going for the latter
  for now... ]

Thor Lancelot Simon wrote:

> Module Name:  src
> Committed By: tls
> Date:         Fri Dec 14 00:23:49 UTC 2007
> 
> Modified Files:
> 
>       src/sys/dev/pci: if_wm.c
> 
> Log Message:
> 
> Adjust interrupt moderation for newer chips to be saner according to the
> 82571 manual and Intel Application Note 450.  Previously, we were setting
> RADV and TIDV/TADV values that didn't make any sense given the enormous
> ITR value we were setting (well outside the range recommended by Intel
> and quite possibly rejected silently by the chip as junk) and setting
> RADV without setting RDTR, which is explicitly documented as having no
> effect.
> 
> A considerable performance improvement is achieved for TCP and UDP at
> gigabit speed.  I need to revisit this to deal with the timer ticks
> being 4X as long when the chip's in 100mbit mode, and to set values
> for the older chips' interrupt timers that are more like what the
> appnote recommends.  This should help for 82543 and newer, though.

This part of this change:

-               CSR_WRITE(sc, WMREG_RDTR, 0 | RDTR_FPD);
-               CSR_WRITE(sc, WMREG_RADV, 128);
+               CSR_WRITE(sc, WMREG_RDTR, 375 | RDTR_FPD);      /* ITR/4 */
+               CSR_WRITE(sc, WMREG_RADV, 375);         /* MUST be same */

results in a wm(4) on an ARM IQ80321 dropping performance reasonably
badly.  Using "nttcp -t -l 131072 -U 5 ..." with RDTR set to 375 gets
between 210-220 Mbit/sec and with RDTR set to 0 gets between 280-320
Mbit/sec.

Here's some quick figures for a couple of different buffer lengths with
the current ITR/TADV/TIDV/RADV/RDTR values, the same but with RDTR set
to 0 ("mod new") and the old values prior to your change:

                        new           mod new           old
                    -----------     -----------     -----------
                     rx      tx      rx      tx      rx      tx
    buffer size
        32           43      26      42      25      40      24
       256          114      41     111      41     107      40
      1k            127      85     143      84     137      82
      4k            138     126     161     106     149      87
     16k            159     180     152     255     149     252
    128k            155     221     150     322     149     319

The 4k TX case is a bit different - that's reasonably faster with RDTR
non-zero ("new" column).

I also found this in the Linux driver:

    /* Receive Interrupt Delay in units of 1.024 microseconds
     *   hardware will likely hang if you set this to anything but zero.

and this in the FreeBSD driver:

     *   CAUTION: When setting EM_RDTR to a value other than 0, adapters
     *            may hang (stop transmitting) under certain network conditions.
     *            If this occurs a WATCHDOG message is logged in the system
     *            event log. In addition, the controller is automatically reset,
     *            restoring the network connection. To eliminate the potential
     *            for the hang ensure that EM_RDTR is set to 0.

But I don't see hangs, just a drop off in performance...

Comments or ideas on what we should do?

Simon.