Subject: kern/33040: kernel resource shortage + paging activity -> freeze
To: None <kern-bug-people@netbsd.org, gnats-admin@netbsd.org,>
From: None <soda@NetBSD.org>
List: netbsd-bugs
Date: 03/09/2006 13:20:00
>Number:         33040
>Category:       kern
>Synopsis:       kernel resource shortage + paging activity -> freeze
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    kern-bug-people
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Thu Mar 09 13:20:00 +0000 2006
>Originator:     SODA Noriyuki
>Release:        NetBSD 3.0
>Organization:
The NetBSD project
>Environment:
System: NetBSD ftp.netbsd.org 3.0 NetBSD 3.0 (NBFTP) #1: Tue Dec 27 07:15:13 UTC 2005 root@:/usr/obj/sys/arch/i386/compile.i386/NBFTP i386
Architecture: i386
Machine: i386
>Description:

1. daily "supscan" process on ftp.netbsd.org stopped working
  with the following message:
	sbrk: grow 4096 failed, error = 12
	Segmentation fault

  (This segmentation fault is caused by the fact that
   supscan doesn't check the return value of malloc().)

  The sbrk(2) failure happens, when the process size grows to 58MB,
  and 58MB is far smaller than the datasize limit of the process.

  This "sbrk: grow 4096 failed, error = 12" message is printed by
  sys_obreak() in the kernel. So, it seems that some resource shortage
  occurs there.

2. I tried simple test program which did malloc(60MB), and touched
  every page of the malloc'ed region.
  The test program just failed at first with similar sbrk message above.
  (It succeded, if the size of malloc() was 50MB instead of 60MB.)
  But after running the test program several times, suddenly it started
  to working....
  But still, "supscan" itself didn't work.
  I thought I could make supscan work again, if I could let supscan
  allocate large size memory at once, instead of thousands times
  small size malloc().

3. I tried to workaround the problem, by using an alternate (and bogus)
  malloc() library, which allocates 256MB memory region at once by mmap(2)
  and use the region for the memory that malloc(3) returns.

  256MB mmap(2) works, and the process was using more than half of the
  allocated area, and then, the system freezed. No userlevel process was
  working. But kernel itself was working. e.g. The machine replied ping(8).
  And according to DDB, some VM activity was going on.

4. After reboot, "supscan" becomes to work again, without the bogus malloc
  library.

DDB log:

db> bt 
cpu_Debugger(647dd400,72103bec,f420,1b,1) at netbsd:cpu_Debugger+0x1

wskbd_translate(603324c0,2,1,0,ffffffff) at netbsd:wskbd_translate+0x74
wskbd_input(647dd400,2,1,72103c34,6032e900) at netbsd:wskbd_input+0x133
pckbd_input(63c54400,1,246,1,60340e60) at netbsd:pckbd_input+0x72
pckbportintr(60332600,0,1,0,1) at netbsd:pckbportintr+0x4f
pckbcintr(63c54480,0,7d130010,30,10) at netbsd:pckbcintr+0xad
Xintr_legacy1() at netbsd:Xintr_legacy1+0xa9
--- interrupt ---
Xspllower(0,0,1000,0,ffffffff) at netbsd:Xspllower+0xe
malloc(40,60315540,1,88db9c08,59610000) at netbsd:malloc+0x20a
amap_alloc1(10,0,1,0,1) at netbsd:amap_alloc1+0xc4
amap_alloc(10000,0,1,6025bf50,81ce30e8) at netbsd:amap_alloc+0x28
amap_copy(81ce30e8,88db9c08,1,1,59610000) at netbsd:amap_copy+0x83
uvmfault_amapcopy(72103ed4,6,0,0,0) at netbsd:uvmfault_amapcopy+0xca
uvm_fault(81ce30e8,59610000,0,2,be41a2b) at netbsd:uvm_fault+0x116
trap() at netbsd:trap+0x348
--- trap (number 6) ---

db> cont
... wait for a while

db> bt 
cpu_Debugger(647dd400,70e41e5c,f420,1b,10000) at netbsd:cpu_Debugger+0x1

wskbd_translate(603324c0,2,1,d692e136,9e3f) at netbsd:wskbd_translate+0x74
wskbd_input(647dd400,2,1,70e41ea4,30) at netbsd:wskbd_input+0x133
pckbd_input(63c54400,1,246,1,60340e60) at netbsd:pckbd_input+0x72
pckbportintr(60332600,0,1,603154a0,1) at netbsd:pckbportintr+0x4f
pckbcintr(63c54480,0,10,60340030,70e40010) at netbsd:pckbcintr+0xad
Xintr_legacy1() at netbsd:Xintr_legacy1+0xa9
--- interrupt ---
Xspllower(0,204,602e034e,0,603472b4) at netbsd:Xspllower+0xe

db>  ps/w
 PID          COMMAND     EMUL  PRI UTIME STIME WAIT-MSG    WAIT-CHANNEL

 13871          sleep   netbsd   32   0.0   0.0
 23811           ftpd   netbsd   24   0.0   0.0
 13722           ftpd   netbsd   24   0.0   0.0
 20761           ftpd   netbsd   24   0.0   0.0
 24028        supscan   netbsd    4   3.3 191.2
 3366            ftpd   netbsd   24   0.0   0.0
 26682           ftpd   netbsd   24   0.0   0.0
 24345           ftpd   netbsd   24   0.0   0.0
 19530          sleep   netbsd   32   0.0   0.0
 15756           ftpd   netbsd   24   0.0   0.0
 15569           ftpd   netbsd   24   0.0   0.0
 20168           ftpd   netbsd   24   0.0   0.0
 27105           ftpd   netbsd   24   0.0   0.0
 12090           ftpd   netbsd   24   0.0   0.0
 28110          sleep   netbsd   32   0.0   0.0
 13331           ftpd   netbsd   24   0.0   0.0
 8239            ftpd   netbsd   24   0.0   0.0
 25181           ftpd   netbsd   24   0.0   0.1
 10199             sh   netbsd   32   0.0   0.0 wait        0x81e74198
 29844           ftpd   netbsd   24   0.0   0.0
--db_more--      sshd   netbsd   24   0.0   0.0 select      netbsd:selwait
db6735           sshd   netbsd   24   0.0   0.0 select      netbsd:selwait
 24920           sshd   netbsd   24   0.0   0.0 netio       0x69f9b868EL
 16410           ftpd   netbsd   24   0.0   0.0
 18112           ftpd   netbsd   24   0.0   0.0 netio       0x69b317ec
 13884           ftpd   netbsd   24   0.0   0.0
 19663           ftpd   netbsd   24   0.0   0.0
 1762            ftpd   netbsd   24   0.0   0.0
 7077            ftpd   netbsd   24   0.0   0.0
 15907           ftpd   netbsd   24   0.0   0.0
 28932           ftpd   netbsd   24   0.0   0.1
 18396           ftpd   netbsd   24   0.0   0.0
 12032           ftpd   netbsd   24   0.0   0.0 pipewr      0x78fa66c8
 6105            ftpd   netbsd   24   0.0   0.0
 18649           ftpd   netbsd   24   0.0   0.0
 21957           ftpd   netbsd   24   0.0   0.0
 20115     supfilesrv   netbsd   24   0.8   0.1
 2434            ftpd   netbsd   24   0.8   1.6
 419             ftpd   netbsd   24   0.0   0.0
 11569           ftpd   netbsd   24   0.0   0.0
 22743           ftpd   netbsd   24   0.0   0.0
 11971           ftpd   netbsd   24   0.0   0.0
 18427           ftpd   netbsd   24   0.0   0.0
 28946           ftpd   netbsd   24   0.0   0.0
--db_more-- 
 237             ftpd   netbsd   24   0.0   0.0

 2686            ftpd   netbsd   24   0.0   0.0
 11854           ftpd   netbsd   24   0.1   5.6
 11888           ftpd   netbsd   24   0.7   1.0
 17646           ftpd   netbsd   24   0.0   0.2
 22195           ftpd   netbsd   24   1.2   3.0
 4001            ftpd   netbsd   24   0.1   5.2
 4913            ftpd   netbsd   24   0.1   5.2
 27866           ftpd   netbsd   24   0.1   5.3
 28570           sshd   netbsd   24   0.0   0.0
 4752            sshd   netbsd   24   0.0   0.0 netio       0x6510a24c
 29835           ftpd   netbsd   24   0.0   0.6
 10820           ftpd   netbsd   24   0.0   0.6
 7977            sshd   netbsd   24   0.0   0.0 select      netbsd:selwait
 13094           sshd   netbsd   24   0.0   0.0 netio       0x65884998
 9344            ftpd   netbsd   24   0.0   4.1
 25222           tcsh   netbsd   25   0.0   0.0 ttyin       0x705e1e7c
 16080         pickup   netbsd   24   0.0   0.0
 23676           ftpd   netbsd   24   0.1   0.6
 2611            ftpd   netbsd   24   0.0   0.1
 27517           ftpd   netbsd   24   0.0   0.6
 1214            ftpd   netbsd   24   8.9   7.8
--db_more--      ftpd   netbsd   24   0.0   0.7
 13489           ftpd   netbsd   24   8.5   8.0

 87              ftpd   netbsd   24   0.0   0.2
 26121     screen-4.0.2   netbsd   40   0.0   0.0
 29695           tcsh   netbsd   40   0.0   0.0 pause       0x89d1b33c
 18105           sshd   netbsd   24   0.2   0.2 select      netbsd:selwait
 2379            sshd   netbsd   24   0.0   0.0 netio       0x65066798
 2779            tcsh   netbsd   50   0.0   0.0
 1506            tcsh   netbsd   40   0.0   0.0 pause       0x7344b020
 14842           ftpd   netbsd   24   0.0   0.0
 14687           ftpd   netbsd   24   0.0   0.0
 13883           ftpd   netbsd   24   0.0   2.0
 129             ftpd   netbsd   24   0.0   0.1
 6934            ftpd   netbsd   24   0.0   0.0
 16812           ftpd   netbsd   24   0.0   0.0
 16705           ftpd   netbsd   24   0.0   2.0
 28064         vmstat   netbsd   32   0.0   0.6
 1696            tcsh   netbsd   40   0.0   0.0 pause       0x9f004514
 13029           sshd   netbsd   24   1.3   1.0
 16225           sshd   netbsd   24   0.0   0.0 netio       0x65ddfa4c
 28516           more   netbsd   25   0.0   0.0 ttyin       0x71d40c78
 7266              sh   netbsd   32   0.0   0.0 wait        0x739cc000
 10572            man   netbsd   32   0.0   0.0 wait        0x8e501004
--db_more--      ftpd   netbsd   24   0.0   1.4
 5907           rsync   netbsd   24 212.8 101.8

 14623           tcsh   netbsd   40   0.0   0.0 pause       0x81a99248
 29665           sshd   netbsd   24   0.0   0.0 select      netbsd:selwait
 21317           sshd   netbsd   24   0.0   0.0 netio       0x68130174
 2192      screen-4.0.2   netbsd   40   0.0   0.0
 28938          rsync   netbsd   24   1.0   2.2
 18317     screen-4.0.2   netbsd   40   0.0   0.0
 10344           tcsh   netbsd   40   0.0   0.0 pause       0x7c904238
 14593           sshd   netbsd   24   0.3   0.2 select      netbsd:selwait
 2362            sshd   netbsd   24   0.0   0.0 netio       0x6a0382c0
 14920     screen-4.0.2   netbsd   40   0.0   0.1
 17145            ksh   netbsd   40   0.0   0.0 pause       0x8a02659c
 25397           sshd   netbsd   24   1.3   1.2 select      netbsd:selwait
 1808            sshd   netbsd   24   0.0   0.0 netio       0x68a3e2dc
 7360            ftpd   netbsd   24 117.2  51.2
 2046            tcsh   netbsd   25   0.0   0.0 ttyin       0x70e9f014
 15003           sshd   netbsd   24   0.5   0.6 select      netbsd:selwait
 18210           sshd   netbsd   24   0.0   0.0 netio       0x6504d264
 24808           ftpd   netbsd   24  49.81007.1
 12853           bash   netbsd   25   0.0   0.0 ttyin       0x70e9fd7c
 14533           ftpd   netbsd   24 336.0 102.6
 11874     screen-4.0.2   netbsd   40   0.1   0.1
--db_more--      bash   netbsd   32   0.0   0.0 wait        0x8149c1a8
 13385           sshd   netbsd   24   1.1   1.2 select      netbsd:selwait

 24748           sshd   netbsd   24   0.0   0.0 netio       0x682971dc
 7091            bash   netbsd   32   0.0   0.0 wait        0x8db23808
 587             sshd   netbsd   24   0.0   0.0 select      netbsd:selwait
 7407            sshd   netbsd   24   0.0   0.0 netio       0x6653f6e8
 9123            tcsh   netbsd   25   0.0   0.0 ttyin       0x70e9f224
 6318            tcsh   netbsd   25   0.0   0.0 ttyin       0x705e163c
 22408          rsync   netbsd   24 495.6 244.8
 24703            icb   netbsd   24   0.0   0.0 select      netbsd:selwait
 21028             sh   netbsd   32   0.0   0.0 wait        0x88b08b2c
 9811      screen-4.0.2   netbsd   24   0.0   0.1 select      netbsd:selwait
 17978     screen-4.0.2   netbsd   40   0.2   0.2
 11857           bash   netbsd   32   0.0   0.0 wait        0x7215c1a4
 616             sshd   netbsd   24   2.4   2.7 select      netbsd:selwait
 6909            sshd   netbsd   24   0.0   0.0 netio       0x679976b0
 10609     screen-4.0.2   netbsd   40   0.0   0.6
 26720            csh   netbsd   40   0.0   0.0 pause       0x77d5c184
 9416            sshd   netbsd   24   1.3   1.6 select      netbsd:selwait
 2018            sshd   netbsd   24   0.0   0.0 netio       0x6911d37c
 14174          rsync   netbsd   24 294.5 493.6
 18511      python2.4   netbsd   5014537.81458.1 *
 17320      python2.4   netbsd   507883.4 520.9 *
--db_more-- python2.4   netbsd   248956.74155.3
 17854             sh   netbsd   32   0.0   0.0 wait        0x729424cc

 1065            ftpd   netbsd   24 116.21045.8
 23507     screen-4.0.2   netbsd   40   0.1   0.1
 25833           bash   netbsd   32   0.0   0.0 wait        0x87402664
 12071           sshd   netbsd   24   2.8   2.5 select      netbsd:selwait
 7336            sshd   netbsd   24   0.0   0.0 netio       0x65eb51f4
 2130      screen-4.0.2   netbsd   40   0.2   0.4
 5440            bash   netbsd   32   0.0   0.0 wait        0x8a5f9808
 20312           sshd   netbsd   24   6.6   4.3
 29683           sshd   netbsd   24   0.0   0.0 netio       0x65c2dc98
 28488     irc-20040820   netbsd   24   5.0   4.1
 12508     screen-4.0.2   netbsd   24   0.8   2.1 select      netbsd:selwait
 22969           tcsh   netbsd   25   0.0   0.0 ttyin       0x70e9e640
 16327           tcsh   netbsd   25   0.0   0.0 ttyin       0x70e9f53c
 5405            tcsh   netbsd   25   0.0   0.0 ttyin       0x70e9f434
 22039           sshd   netbsd   24   0.0   0.1 select      netbsd:selwait
 28332           sshd   netbsd   24   0.0   0.0 netio       0x65936268
 18924     irc-20040820   netbsd   24   6.6   4.9
 1336      screen-4.0.2   netbsd   40   0.4   0.4
 7125             ksh   netbsd   40   0.0   0.0 pause       0x7ee12014
 23300           sshd   netbsd   24   7.0   5.8 select      netbsd:selwait
 25224           sshd   netbsd   24   0.0   0.0 netio       0x6538760c
--db_more--       ksh   netbsd   25   0.0   0.0 ttyin       0x70e9fb6c
 11394            ksh   netbsd   40   0.0   0.0 pause       0x89d9e10c

 23899     screen-4.0.2   netbsd   24   2.5   3.8 select      netbsd:selwait
 16580     irc-20040820   netbsd   24   9.9   6.4
 22200            ksh   netbsd   40   0.0   0.0 pause       0x8dea7114
 11766     screen-4.0.2   netbsd   24   1.9   4.2 select      netbsd:selwait
 5698             icb   netbsd   24   1.4   3.3 select      netbsd:selwait
 24289             sh   netbsd   32   0.0   0.0 wait        0x8c24d4d8
 29845           bash   netbsd   32   0.0   0.0 wait        0x88b08004
 24844     screen-4.0.2   netbsd   24   1.5   3.1 select      netbsd:selwait
 29482            icb   netbsd   24   0.1   0.2 select      netbsd:selwait
 25152     irc-20040820   netbsd   24   8.9   5.0
 3521      screen-4.0.2   netbsd   24   2.6   3.8 select      netbsd:selwait
 27742            csh   netbsd   25   0.0   0.0 ttyin       0x70e9fa64
 23503           tcsh   netbsd   25   0.1   0.0 ttyin       0x705e1954
 19590           sshd   netbsd   24   0.2   0.1 select      netbsd:selwait
 13184           sshd   netbsd   24   0.0   0.0 netio       0x6614a2e0
 10309            icb   netbsd   25   0.3   0.5 ttyin       0x71d40018
 27786     screen-4.0.2   netbsd   24   1.3   1.1 select      netbsd:selwait
 25294            csh   netbsd   25   0.0   0.0 ttyin       0x70e9fe84
 13756           bash   netbsd   25   0.0   0.0 ttyin       0x70e9fc74
 20615           sshd   netbsd   24   2.0   1.4 select      netbsd:selwait
 9038            sshd   netbsd   24   0.0   0.0 netio       0x6693eaf8
--db_more-- rc-20040820   netbsd   24   7.0   5.6
 12880           bash   netbsd   32   0.0   0.0 wait        0x8db23b38

 11106     screen-4.0.2   netbsd   24   1.6   3.9 select      netbsd:selwait
 17132             sh   netbsd   25   0.0   0.0 ttyin       0x70e9f644
 4204            sshd   netbsd   24   0.9   2.2 select      netbsd:selwait
 26029           sshd   netbsd   24   0.0   0.0 netio       0x64ec1e70
 8525            tcsh   netbsd   40   0.0   0.0 pause       0x9f56a0b0
 244             sshd   netbsd   24   0.2   0.2 select      netbsd:selwait
 17124           sshd   netbsd   24   0.0   0.0 netio       0x6693e948
 16055            cat   netbsd   24   0.0   0.0 piperd      0x72c923f4
 3063            sshd   netbsd   24   0.9   2.8 select      netbsd:selwait
 5974            sshd   netbsd   24   0.0   0.0 netio       0x66a85c0c
 6174      irc-20040820   netbsd   24   8.8   5.6
 13132           bash   netbsd   32   0.0   0.0 wait        0x81e74e58
 1364      screen-4.0.2   netbsd   24   3.3   4.6 select      netbsd:selwait
 1550      irc-20040820   netbsd   24   7.3   4.3
 2132             csh   netbsd   40   0.0   0.0 pause       0x7d4b3738
 20742     screen-4.0.2   netbsd   24   1.8   4.0 select      netbsd:selwait
 29127     irc-20040820   netbsd   24   8.8   4.8
 8202             csh   netbsd   40   0.0   0.0 pause       0x7e20a540
 28400     screen-4.0.2   netbsd   24   2.8   5.3 select      netbsd:selwait
 26589            icb   netbsd   24   1.6   2.8 select      netbsd:selwait
 10102             sh   netbsd   32   0.0   0.0 wait        0x722ed4c8
--db_more--      tcsh   netbsd   40   0.0   0.0 pause       0x9f4ef090
19383           tcsh   netbsd   40   0.0   0.0 pause       0x9f4ef090
  4927           tcsh   netbsd   40   0.0   0.0 pause       0x7c11308
 22750     screen-4.0.2   netbsd   24   2.4   4.0 select      netbsd:selwait
 7197      screen-4.0.2   netbsd   40   0.0   1.1
 7541             ksh   netbsd   40   0.0   0.0 pause       0x890a0398
 2890            sshd   netbsd   24   8.0   5.5 select      netbsd:selwait
 21572           sshd   netbsd   24   0.0   0.0 netio       0x6618c690
 22043     irc-20040820   netbsd   24   8.7   5.6
 4809      screen-4.0.2   netbsd   24   2.5   4.2 select      netbsd:selwait
 2659             zsh   netbsd   25   0.0   0.0 ttyin       0x70e9e850
 25567           sshd   netbsd   24   0.0   0.0 select      netbsd:selwait
 1001            sshd   netbsd   24   0.0   0.0 netio       0x65223ea8
 13934            icb   netbsd   24   1.2   2.4 select      netbsd:selwait
 24529           bash   netbsd   32   0.0   0.0 wait        0x7740e1a0
 24335           sshd   netbsd   24   5.2   3.8 select      netbsd:selwait
 19248           sshd   netbsd   24   0.0   0.0 netio       0x64bca5dc
 8163            tcsh   netbsd   40   0.0   0.0 pause       0x81e69234
 6241            tcsh   netbsd   25   0.0   0.0 ttyin       0x70e9e538
 21641     irc-20040820   netbsd   24  10.0   6.9
 13792     screen-4.0.2   netbsd   24   2.5   4.2 select      netbsd:selwait
 8746      screen-4.0.2   netbsd   40   0.0   1.3
 6831            tcsh   netbsd   40   0.0   0.0 pause       0x950cb25c
 14535           sshd   netbsd   24   7.2   4.9 select      netbsd:selwait
--db_more-- 
 13294            icb   netbsd   24   0.5   0.4 select      netbsd:selwait

 15343     screen-4.0.2   netbsd   24   0.6   1.0 select      netbsd:selwait
 5045      irc-20040820   netbsd   24   7.5   4.4
 25723           bash   netbsd   32   0.0   0.0 wait        0x7740e998
 20668     screen-4.0.2   netbsd   24   1.8   2.9 select      netbsd:selwait
 8108             icb   netbsd   25   0.1   0.1 ttyin       0x705e1c6c
 6695            tcsh   netbsd   40   0.0   0.0 pause       0x739d317c
 6244      screen-4.0.2   netbsd   24   0.1   0.2 select      netbsd:selwait
 6644      screen-4.0.2   netbsd   40   0.2   0.9
 7354            tcsh   netbsd   40   0.0   0.0 pause       0x727c70c4
 6577            sshd   netbsd   24   0.2   0.2 select      netbsd:selwait
 5241            sshd   netbsd   24   0.0   0.0 netio       0x64bc5bc0
 4959      irc-20040820   netbsd   24   9.8   6.1
 3417             ksh   netbsd   40   0.0   0.0 pause       0x735a214c
 3936      screen-4.0.2   netbsd   24   2.9   4.5 select      netbsd:selwait
 1195             csh   netbsd   25   0.0   0.0 ttyin       0x705e184c
 1770            sshd   netbsd   24   1.7   1.6 select      netbsd:selwait
 468             sshd   netbsd   24   0.0   0.0 netio       0x64dceae4
 464       irc-20040820   netbsd   24  16.3   9.5
 399       screen-4.0.2   netbsd   24   4.3   6.0 select      netbsd:selwait
 910       screen-4.0.2   netbsd   40   1.0   0.0
 973             tcsh   netbsd   40   0.0   0.0 pause       0x722e30a4
--db_more--      sshd   netbsd   24  16.8   8.4
 1081           emacs   netbsd   24  60.4  10.2 select      netbsd:selwait

 1464      screen-4.0.2   netbsd   24   4.5   6.8 select      netbsd:selwait
 1435            sshd   netbsd   24   0.0   0.0 netio       0x64ce8990
 289            getty   netbsd   25   0.0   0.0 ttyin       0x71d45758
 841            getty   netbsd   25   0.0   0.0 ttyin       0x71d45650
 1438           getty   netbsd   25   0.0   0.0 ttyin       0x71d45548
 742            getty   netbsd   25   0.0   0.0
 1532            cron   netbsd   32   1.2   2.9
 881            inetd   netbsd   24   3.3  33.5
 1131            qmgr   netbsd   24   0.4   2.5
 831           master   netbsd   24   1.3   4.7
 438             ntpd   netbsd   40   0.2   2.6
 766             sshd   netbsd   24   1.3  12.0
 664           rtsold   netbsd   24  12.5  10.2
 381             ntpd   netbsd   40  17.1  15.4
 599               sh   netbsd   32   4.8  13.1 wait        0x71fba668
 617               sh   netbsd   32   0.9   2.7 wait        0x71fba998
 306             icbd   netbsd   24  78.2  93.5
 276       supfilesrv   netbsd   24   0.0   0.0 netcon      0x648a92c2
 275       supfilesrv   netbsd   24   0.0   0.1 netcon      0x648a9112
 278       supfilesrv   netbsd   24   0.0   0.0 netcon      0x63c7dcde
 272       supfilesrv   netbsd   24   0.0   0.5 netcon      0x63c7de8e
--db_more--     named   netbsd   24 153.3  50.9 select      netbsd:selwait
 438             ntpd   netbsd   40   0.2   2.6
 766             sshd   netbsd   24   1.3  12.0
 664           rtsold   netbsd   24  12.5  10.2
 381             ntpd   netbsd   40  17.1  15.4
 599               sh   netbsd   32   4.8  13.1 wait        0x71fba668
 617               sh   netbsd   32   0.9   2.7 wait        0x71fba998
 306             icbd   netbsd   24  78.2  93.5
 276       supfilesrv   netbsd   24   0.0   0.0 netcon      0x648a92c2
 275       supfilesrv   netbsd   24   0.0   0.1 netcon      0x648a9112
 278       supfilesrv   netbsd   24   0.0   0.0 netcon      0x63c7dcde
 272       supfilesrv   netbsd   24   0.0   0.5 netcon      0x63c7de8e
 154           racoon   netbsd   24  20.3   2.3 select      netbsd:selwait

 145          syslogd   netbsd   24  59.8 192.9
 7           aiodoned   netbsd    4   0.0 477.2 aiodoned    netbsd:uvm+0x38
 6            ioflush   netbsd   40   0.08156.5
>5         pagedaemon   netbsd    4   0.01531.6
 4          atapibus0   netbsd   16   0.0   0.0 sccomp      0x64752b78
 3            atabus1   netbsd   16   0.0   0.0 atath       0x64752b9c
 2            atabus0   netbsd   16   0.0   0.0 atath       0x64752a30
 1               init   netbsd   32   0.8   1.5 wait        0x705e2000
 0            swapper   netbsd   17   0.0  38.1
 18824           ftpd   netbsd   -1   0.0   0.0 *
db>  8           ftpd   netbsd   -1   0.0   0.0 *

db> trace/t 0t24028 

ltsleep(60347110,204,602dffa5,0,603472b4) at netbsd:ltsleep+0x355

uvm_wait(602dffa5,6,0,1,59610000) at netbsd:uvm_wait+0x69
uvmfault_amapcopy(72103ed4,6,0,0,0) at netbsd:uvmfault_amapcopy+0x49
uvm_fault(81ce30e8,59610000,0,2,be41a2b) at netbsd:uvm_fault+0x116
trap() at netbsd:trap+0x348
--- trap (number 6) ---

db> show uvmexp
  pagesize=4096 (0x1000), pagemask=0xfff, pageshift=12

  902487 VM pages: 459679 active, 229949 inactive, 1643 wired, 1189 free
  min  10% (25) anon, 30% (76) file, 5% (12) exec
  max  40% (102) anon, 70% (179) file, 30% (76) exec
  pages  192679 anon, 595105 file, 4190 exec
  freemin=64, free-target=94, inactive-target=229876, wired-max=300829
  faults=1329879033, traps=1330789198, intrs=-2079543085, ctxswitch=1449975612
  softint=513044429, syscalls=301245284, swapins=1098539, swapouts=1098705
  fault counts:
    noram=1478, noanon=0, pgwait=215, pgrele=0
    ok relocks(total)=141062(141285), anget(retrys)=1514816847(65698), amapcopy=
312497213
    neighbor anon/obj pg=466582697/-794624859, gets(lock/unlock)=944876720/75542

    cases: anon=1004541618, anoncow=226892341, obj=794877961, prcopy=149998548, 
p
rzero=-1067770903
  daemon and swap counts:
    woke=127502690, revs=1286459, scans=843409523, obscans=353105170, anscans=30
8365
    busy=436, freed=353413535, reactivate=398737573, deactivate=865174792
--db_more-- =23185, pending=0, nswget=65847
    nswapdev=2, nanon=1370300, nanonneeded=1370300 nfreeanon=1251185

db>  wpages=524286, swpginuse=35472, swpgonly=28450 paging=0

db> cont
... wait for a while
Stopped in pid 24028.1 (supscan) at     netbsd:cpu_Debugger+0x1:        r

db> trace 
cpu_Debugger(647dd400,72103c7c,f420,1b,63b33000) at netbsd:cpu_Debugger+0x1

wskbd_translate(603324c0,2,1,dab9337c,295a1) at netbsd:wskbd_translate+0x74
wskbd_input(647dd400,2,1,72103cc4,72103cdc) at netbsd:wskbd_input+0x133
pckbd_input(63c54400,1,246,1,60340e60) at netbsd:pckbd_input+0x72
pckbportintr(60332600,0,1,60346e60,1) at netbsd:pckbportintr+0x4f
pckbcintr(63c54480,6,60250010,60340030,7d130010) at netbsd:pckbcintr+0xad
Xintr_legacy1() at netbsd:Xintr_legacy1+0xa9
--- interrupt ---
Xspllower(6,59610000,72103ee8,0,1) at netbsd:Xspllower+0xe
uvm_wait(602dffa5,6,0,1,59610000) at netbsd:uvm_wait+0x45
uvmfault_amapcopy(72103ed4,6,0,0,0) at netbsd:uvmfault_amapcopy+0x49
uvm_fault(81ce30e8,59610000,0,2,be41a2b) at netbsd:uvm_fault+0x116
trap() at netbsd:trap+0x348
--- trap (number 6) ---

db> show uvmexp 

  pagesize=4096 (0x1000), pagemask=0xfff, pageshift=12

  902487 VM pages: 459679 active, 229949 inactive, 1643 wired, 1190 free
  min  10% (25) anon, 30% (76) file, 5% (12) exec
  max  40% (102) anon, 70% (179) file, 30% (76) exec
  pages  192679 anon, 595105 file, 4190 exec
  freemin=64, free-target=94, inactive-target=229876, wired-max=300829
  faults=1329879033, traps=1330789199, intrs=-2079541254, ctxswitch=1466636147
  softint=513047860, syscalls=301245284, swapins=1098539, swapouts=1098705
  fault counts:
    noram=1478, noanon=0, pgwait=215, pgrele=0
    ok relocks(total)=141062(141285), anget(retrys)=1514816847(65698), amapcopy=
312497213
    neighbor anon/obj pg=466582697/-794624859, gets(lock/unlock)=944876720/75542

    cases: anon=1004541618, anoncow=226892341, obj=794877961, prcopy=149998548, 
p
rzero=-1067770903
  daemon and swap counts:
    woke=135832957, revs=1286459, scans=843409523, obscans=353105170, anscans=30
8365
    busy=436, freed=353413535, reactivate=398737573, deactivate=865174792
    nswapdev=2, nanon=1370300, nanonneeded=1370300 nfreeanon=1251185

db>  wpages=524286, swpginuse=35472, swpgonly=28450 paging=0

db> cont
... wait for a while
Stopped in pid 5.1 (pagedaemon) at      netbsd:cpu_Debugger+0x1:        ret

db> trace 
cpu_Debugger(647dd400,70e41dc8,f420,1b,70e41e2a) at netbsd:cpu_Debugger+0x1

wskbd_translate(603324c0,2,1,fa48,f103) at netbsd:wskbd_translate+0x74
wskbd_input(647dd400,2,1,70e41e10,70e41e28) at netbsd:wskbd_input+0x133
pckbd_input(63c54400,1,246,1,60340e60) at netbsd:pckbd_input+0x72
pckbportintr(60332600,0,1,3f1b5,1) at netbsd:pckbportintr+0x4f
pckbcintr(63c54480,0,10,30,3fd40010) at netbsd:pckbcintr+0xad
Xintr_legacy1() at netbsd:Xintr_legacy1+0xa9
--- interrupt ---
Xspllower(0,705e2660,6a9,0,601fc70a) at netbsd:Xspllower+0xe
mpidle(705eb210,0,949aa6d0,601fc22a,949aa6d0) at netbsd:mpidle+0xd1
ltsleep(603472ac,204,602e034e,0,603472b4) at netbsd:ltsleep+0x355

db> show uvmexp
Current UVM status:
  pagesize=4096 (0x1000), pagemask=0xfff, pageshift=12
  902487 VM pages: 459679 active, 229949 inactive, 1643 wired, 1201 free
  min  10% (25) anon, 30% (76) file, 5% (12) exec
  max  40% (102) anon, 70% (179) file, 30% (76) exec
  pages  192679 anon, 595105 file, 4190 exec
  freemin=64, free-target=94, inactive-target=229876, wired-max=300829
  faults=1329879033, traps=1330789200, intrs=-2079536620, ctxswitch=1509903848
  softint=513056680, syscalls=301245284, swapins=1098539, swapouts=1098705
  fault counts:7dd400,70e41dc8,f420,1b,70e41e2a) at netbsd:cpu_Debugger+0x1
    noram=1478, noanon=0, pgwait=215, pgrele=0
    ok relocks(total)=141062(141285), anget(retrys)=1514816847(65698), amapcopy=
312497213
    neighbor anon/obj pg=466582697/-794624859, gets(lock/unlock)=944876720/75542

    cases: anon=1004541618, anoncow=226892341, obj=794877961, prcopy=149998548, 
p
rzero=-1067770903
  daemon and swap counts:
    woke=157466807, revs=1286459, scans=843409523, obscans=353105170, anscans=30
8365
    busy=436, freed=353413535, reactivate=398737573, deactivate=865174792
    nswapdev=2, nanon=1370300, nanonneeded=1370300 nfreeanon=1251185
    swpages=524286, swpginuse=35472, swpgonly=28450 paging=0

db> cont
... wait for a while
Stopped in pid 5.1 (pagedaemon) at      netbsd:cpu_Debugger+0x1:        ret

db> trace 
cpu_Debugger(647dd400,70e41e1c,f420,1b,e) at netbsd:cpu_Debugger+0x1

wskbd_translate(603324c0,2,1,0,b6cc0000) at netbsd:wskbd_translate+0x74
wskbd_input(647dd400,2,1,70e41e64,0) at netbsd:wskbd_input+0x133
pckbd_input(63c54400,1,246,1,60340e60) at netbsd:pckbd_input+0x72
pckbportintr(60332600,0,1,6ea6d000,1) at netbsd:pckbportintr+0x4f
pckbcintr(63c54480,0,10,30,705e0010) at netbsd:pckbcintr+0xad
Xintr_legacy1() at netbsd:Xintr_legacy1+0xa9
--- interrupt ---
Xspllower(0,0,949aa6d0,601fc22a,949aa6d0) at netbsd:Xspllower+0xe
ltsleep(603472ac,204,602e034e,0,603472b4) at netbsd:ltsleep+0x2f7
db>  ageout(705eb210,388000,38f000,0,60100321) at netbsd:uvm_pageout+0x5c

db> show uvmexp 
  pagesize=4096 (0x1000), pagemask=0xfff, pageshift=12

  902487 VM pages: 459695 active, 229949 inactive, 1643 wired, 1181 free
  min  10% (25) anon, 30% (76) file, 5% (12) exec
  max  40% (102) anon, 70% (179) file, 30% (76) exec
  pages  192695 anon, 595105 file, 4190 exec
  freemin=64, free-target=94, inactive-target=229881, wired-max=300829
  faults=1329879049, traps=1330789218, intrs=-2079534729, ctxswitch=1528184380
  softint=513060246, syscalls=301245616, swapins=1098539, swapouts=1098705
  fault counts:
    noram=1478, noanon=0, pgwait=215, pgrele=0
    ok relocks(total)=141062(141285), anget(retrys)=1514816847(65698), amapcopy=
312497214
    neighbor anon/obj pg=466582697/-794624859, gets(lock/unlock)=944876720/75542

    cases: anon=1004541618, anoncow=226892341, obj=794877961, prcopy=149998548, 
p
rzero=-1067770887
  daemon and swap counts:
    woke=166607073, revs=1286459, scans=843409523, obscans=353105170, anscans=30
8365
    busy=436, freed=353413535, reactivate=398737573, deactivate=865174792
--db_more-- =23185, pending=0, nswget=65847
    nswapdev=2, nanon=1370300, nanonneeded=1370300 nfreeanon=1251169
    swpages=524286, swpginuse=35472, swpgonly=28450 paging=0

db> trace/t 0t24028  

ltsleep(60347110,204,602dffa5,0,603472b4) at netbsd:ltsleep+0x355

uvm_wait(602dffa5,6,0,1,59620000) at netbsd:uvm_wait+0x69
uvmfault_amapcopy(72103ed4,6,0,0,0) at netbsd:uvmfault_amapcopy+0x49
uvm_fault(81ce30e8,59620000,0,2,be41a2b) at netbsd:uvm_fault+0x116
trap() at netbsd:trap+0x348
--- trap (number 6) ---


>How-To-Repeat:

not known.

>Fix:

N/A