Very bad interactive response, and awful amount of system time

[Rhialto <rhialto%falu.nl@localhost>]

I am currently rebuilding all my packages for NetBSD 5.0, with pkg_comp,
using the GENERIC kernel.  This keeps my single-CPU system rather busy.
(Asus A8V DeLuxe)

I noted that the interactive response of other tasks suffered
progressively more and more under the load. At one point, it had gotten
so bad that getting an echo in an xterm took seconds! (Of course,
complex programs like Firefox suffered even worse)

This is really unusable. Previous versions of NetBSD seemed to be much
better in this regard. In desperation I niced the whole pkg_chk process
group to the max, which fortunately helped.

I wonder if this is related to the enormous amount of system time that
top (still) shows:

load averages:  1.91,  2.03,  2.26;               up 1+01:34:52        16:51:14
126 processes: 3 runnable, 122 sleeping, 1 on CPU
CPU states:  4.2% user, 32.3% nice, 63.1% system,  0.0% interrupt,  0.4% idle
Memory: 1568M Act, 801M Inact, 15M Wired, 86M Exec, 1796M File, 188M Free
Swap: 6124M Total, 6124M Free

systat vmstat shows values like

   11 users    Load  1.62  1.78  1.96                  Sat Jun  6 17:04:08

Proc:r  d  s  w     Csw    Trp    Sys   Int   Sof    Flt      PAGING   SWAPPING
     3    47        979  23193   8460     8   385  23234      in  out   in  out
                                                        ops
  82.2% Sy   4.2% Us  13.6% Ni   0.0% In   0.0% Id    pages
|    |    |    |    |    |    |    |    |    |    |
=========================================>>-------                    188 forks
                                                                       64 fkppw
           memory totals (in kB)               8 Interrupts            64 fksvm
          real  virtual     free                 ioapic0 pin 1            pwait
Active 1638276  1638276   159672                 ioapic0 pin 12           relck
All    2876668  2876668  6430952                 ioapic0 pin 17           rlkok
                                               4 ioapic0 pin 18           noram
Namei         Sys-cache     Proc-cache         4 ioapic0 pin 20      2891 ndcpy
    Calls     hits    %     hits     %           ioapic0 pin 14      1527 fltcp
    24072    23740   99                          ioapic0 pin 15     11867 zfod
                                                 ioapic0 pin 22       978 cow
Disks: seeks xfers bytes %busy                                        256 fmin
   fd0                                                                341 ftarg
   wd0           5   19K                                                  itarg
   wd1                                                               3786 wired
   cd0                                                                    pdfre
   cd1                                                                    pdscn
   md0
  nfs0

Where is the clock interrupt, if there are only 8 interrupts/second?
There is no shortage of memory, and the amount of disk I/O isn't that
big either. So the problem must be one of CPU scheduling; that idea is
reinforced by the fact that nice helps very effectively. But I should
not *need* to nice this sort of workload!

-Olaf.
-- 
___ Olaf 'Rhialto' Seibert    -- You author it, and I'll reader it.
\X/ rhialto/at/xs4all.nl      -- Cetero censeo "authored" delendum esse.