Subject: kern/12183: uvm hangs on a process, then takes out the system
To: None <gnats-bugs@gnats.netbsd.org>
From: John Hawkinson <jhawk@mit.edu>
List: netbsd-bugs
Date: 02/11/2001 06:54:31
>Number: 12183
>Category: kern
>Synopsis: uvm hangs on a process, then takes out the system
>Confidential: no
>Severity: serious
>Priority: low
>Responsible: kern-bug-people
>State: open
>Class: sw-bug
>Submitter-Id: net
>Arrival-Date: Sun Feb 11 06:57:00 PST 2001
>Closed-Date:
>Last-Modified:
>Originator: John Hawkinson
>Release: -current of 7 Feb 2000
>Organization:
MIT
>Environment:
System: NetBSD zorkmid.mit.edu 1.5R NetBSD 1.5R (ZORKMID-$Revision: 1.6 $) #89: Wed Feb 7 23:37:46 EST 2001 jhawk@zorkmid.mit.edu:/usr/local/netbsd-current/src/sys/arch/i386/compile/ZORKMID i386
>Description:
While building a library in the bowels of bsd.lib.mk, an "nm -go"
process got stuck. It ran for 10 minutes before I noticed it (it should
have been nigh instantaneous), and was swapped out.
Investigation showed it was tsleeping at uvn_fp2. I began
some investigation with mycroft's assistance and basically got as far
as obtaining a stack trace, displaying the page it was blocked on (from the
WCHAN data), and getting a ps of some other processes, before going to sleep.
After waking up, chs and I started to look at it, but the story had
changed due to nightly cron jobs running (a find was stuck in vnlock),
and shortly afterwards the system pseudohung[*], and I was unable to get
more work done.
This is not trivially reproducible (suck). Various data follow.
[*] pseudohung: I tried to start emacs an emacs and it hung forever,
and I switched out of X to ddb to look at things, and then switched
back, and X was hung. Mouse would not move. The machine still pinged.
It would accept connections, but not send data over them, except for
inetd internal servicers (chargen and echo were the two I tried).
"finger", for instance, would hang. I gave up and rebooted the machine
the hard way.
Thanks to Charles and Chuq for their assistance.
>How-To-Repeat:
02:42am: ps listing from ddb:
db> PID COMMAND EMUL PRI UTIME STIME WAIT-MSG WAIT-CHANNEL
4659 nm netbsd 4 0.0 0.0 uvn_fp2 0xc073eab0
4656 tsort netbsd 24 0.0 0.0 netio 0xc086b944
4655 sh netbsd 32 0.0 0.0 wait 0xc6eb13ac
4654 sh netbsd 32 0.0 0.0 wait 0xc6eb1acc
4653 sh netbsd 24 0.0 0.0 netio 0xc086bd1c
4353 make netbsd 32 0.0 0.0 wait 0xc6eb101c
4352 sh netbsd 32 0.0 0.0 wait 0xc6eb11e4
4323 make netbsd 32 0.0 0.0 wait 0xc6eb1c94
4172 less netbsd 50 0.0 0.0
3943 ssh netbsd 24 1.4 1.8 select selwait
3941 dhclient netbsd 24 0.0 0.0 select selwait
3767 ssh netbsd 24 0.1 0.0 netio 0xc08d43ac
3766 less netbsd 50 0.0 0.0
3765 cvs netbsd 24 0.0 0.0 netio 0xc08d4828
3760 ssh netbsd 24 0.0 0.0
3759 less netbsd 50 0.0 0.0
3758 cvs netbsd 24 0.0 0.0 netio 0xc086b77c
3744 tcsh netbsd 50 0.0 0.0
3705 vi netbsd 50 0.0 0.0
2670 netscape linux 24 0.0 0.0 select selwait
2669 netscape linux 24 230.8 47.8 select selwait
Attempts to get a trace of pid 4659 fail bascuae it's swapped out.
!wakeup()ing the wait channel succeeds. Stack trace:
db> trace: pid 4659 at 0xc6f02b1c
bpendtsleep(c073eab0,204,c045b0e4,0,c6d7c038) at bpendtsleep
uvn_findpage(c6d7c038,0,0,c6f02ca4,0) at uvn_findpage+0x47
uvn_findpages(c6d7c038,0,0,c6f02c98,c6f02ca4,0) at uvn_findpages+0x40
genfs_getpages(c6f02cfc,c6d7c038,c6f02dc8,0,c0439fc0) at genfs_getpages+0x462
VOP_GETPAGES(c6d7c038,0,0,c6f02d64,c6f02dbc) at VOP_GETPAGES+0x58
ufs_balloc_range(c6d7c038,13f,0,1ec1,0) at ufs_balloc_range+0x188
ffs_write(c6f02e80,1,c04396a0,c6d7c038,c6f02f0c) at ffs_write+0x1fd
VOP_WRITE(c6d7c038,c6f02f0c,1,c0871d00,c6d7c038) at VOP_WRITE+0x38
vn_write(c6c64190,c6c641ac,c6f02f0c,c0871d00,1) at vn_write+0x9e
dofilewrite(c6ef41f0,1,c6c64190,8057000,2000) at dofilewrite+0x94
sys_write(c6ef41f0,c6f02f88,c6f02f80) at sys_write+0x67
syscall_plain(1f,1f,8057000,804cd38,bfbfd0c0) at syscall_plain+0x98
2:52am ps:
db> PID COMMAND EMUL PRI UTIME STIME WAIT-MSG WAIT-CHANNEL
4792 less netbsd 25 0.0 0.0 ttyin 0xc63f47cc
4785 less netbsd 50 0.0 0.0
4780 less netbsd 50 0.0 0.0
4775 tcsh netbsd 40 0.0 0.0 pause 0xc6eba984
4774 xterm netbsd 24 0.1 0.0 select selwait
4772 more netbsd 50 0.0 0.0
4771 sh netbsd 32 0.0 0.0
4770 man netbsd 32 0.0 0.0
4769 less netbsd 25 0.0 0.0 ttyin 0xc63f46b0
4767 less netbsd 50 0.1 0.0
4766 objdump netbsd 24 0.0 0.0 netio 0xc08d41c0
4723 less netbsd 50 0.0 0.0
4659 nm netbsd 4 0.0 0.0 uvn_fp2 0xc073eab0
4656 tsort netbsd 24 0.0 0.0 netio 0xc086b944
4655 sh netbsd 32 0.0 0.0 wait 0xc6eb13ac
4654 sh netbsd 32 0.0 0.0 wait 0xc6eb1acc
4653 sh netbsd 24 0.0 0.0 netio 0xc086bd1c
4353 make netbsd 32 0.0 0.0 wait 0xc6eb101c
4352 sh netbsd 32 0.0 0.0 wait 0xc6eb11e4
4323 make netbsd 32 0.0 0.0 wait 0xc6eb1c94
4172 less netbsd 50 0.0 0.0
2:58am ps:
db> PID COMMAND EMUL PRI UTIME STIME WAIT-MSG WAIT-CHANNEL
4794 less netbsd 50 0.0 0.0
4792 less netbsd 50 0.0 0.0
4785 less netbsd 50 0.0 0.0
4780 less netbsd 25 0.0 0.0 ttyin 0xc63f47cc
4775 tcsh netbsd 40 0.0 0.0 pause 0xc6eba984
4774 xterm netbsd 24 0.2 0.0 select selwait
4769 less netbsd 25 0.0 0.0 ttyin 0xc63f46b0
4767 less netbsd 50 0.1 0.0
4766 objdump netbsd 24 0.0 0.0 netio 0xc08d41c0
4723 less netbsd 50 0.0 0.0
4659 nm netbsd 4 0.0 0.0 uvn_fp2 0xc073eab0
4656 tsort netbsd 24 0.0 0.0 netio 0xc086b944
4655 sh netbsd 32 0.0 0.0 wait 0xc6eb13ac
4654 sh netbsd 32 0.0 0.0 wait 0xc6eb1acc
4653 sh netbsd 24 0.0 0.0 netio 0xc086bd1c
4353 make netbsd 32 0.0 0.0 wait 0xc6eb101c
4352 sh netbsd 32 0.0 0.0 wait 0xc6eb11e4
4323 make netbsd 32 0.0 0.0 wait 0xc6eb1c94
4172 less netbsd 50 0.0 0.0
3943 ssh netbsd 24 1.4 1.8 select selwait
3941 dhclient netbsd 24 0.0 0.0 select selwait
'show page' of the wait channel:
db> PAGE 0xc073eab0:
flags=7<BUSY,WANTED,TABLED>, pqflags=4<ACTIVE>, vers=46, wire_count=0, pa=0x3543000
uobject=0xc6d7c038, uanon=0x0, offset=0x0 loan_count=0
[page ownership tracking disabled]
Another ps listing, 3:09am:
db> PID COMMAND EMUL PRI UTIME STIME WAIT-MSG WAIT-CHANNEL
4818 less netbsd 50 0.0 0.0
4802 less netbsd 50 0.0 0.0
4794 less netbsd 50 0.0 0.0
4792 less netbsd 50 0.0 0.0
4785 less netbsd 50 0.0 0.0
4780 less netbsd 50 0.0 0.0
4775 tcsh netbsd 25 0.0 0.0 ttyin 0xc63f47cc
4774 xterm netbsd 24 0.4 0.1 select selwait
4769 less netbsd 25 0.0 0.0 ttyin 0xc63f46b0
4767 less netbsd 50 0.1 0.0
4766 objdump netbsd 24 0.0 0.0 netio 0xc08d41c0
4723 less netbsd 50 0.0 0.0
4659 nm netbsd 4 0.0 0.0 uvn_fp2 0xc073eab0
4656 tsort netbsd 24 0.0 0.0 netio 0xc086b944
4655 sh netbsd 32 0.0 0.0 wait 0xc6eb13ac
4654 sh netbsd 32 0.0 0.0 wait 0xc6eb1acc
4653 sh netbsd 24 0.0 0.0 netio 0xc086bd1c
4353 make netbsd 32 0.0 0.0 wait 0xc6eb101c
4352 sh netbsd 32 0.0 0.0 wait 0xc6eb11e4
4323 make netbsd 32 0.0 0.0 wait 0xc6eb1c94
4172 less netbsd 50 0.0 0.0
3943 ssh netbsd 24 1.4 1.8 select selwait
3941 dhclient netbsd 24 0.0 0.0 select selwait
3767 ssh netbsd 24 0.1 0.0 netio 0xc08d43ac
3766 less netbsd 50 0.0 0.0
3765 cvs netbsd 24 0.0 0.0 netio 0xc08d4828
3760 ssh netbsd 24 0.0 0.0
3759 less netbsd 50 0.0 0.0
3758 cvs netbsd 24 0.0 0.0 netio 0xc086b77c
3744 tcsh netbsd 50 0.0 0.0
3705 vi netbsd 50 0.0 0.0
2670 netscape linux 24 0.0 0.0 select selwait
2669 netscape linux 24 233.6 48.1 select selwait
1523 sshd1 netbsd 24 1.9 0.0 select selwait
398 tcsh netbsd 40 0.2 0.1 pause 0xc6e75c84
397 xterm netbsd 24 5.6 2.5 select selwait
374 zhm netbsd 24 0.0 0.2 select selwait
351 tcsh netbsd 50 0.0 0.0
334 tcsh netbsd 40 0.0 0.0 pause 0xc6e75014
333 tcsh netbsd 40 0.0 0.0 pause 0xc6e5cc80
332 tcsh netbsd 25 0.0 0.2 ttyin 0xc63f4478
331 xterm netbsd 24 6.8 2.8 select selwait
330 xterm netbsd 24 2.4 1.4 select selwait
329 xconsole netbsd 24 0.0 0.0 select selwait
328 xclock netbsd 24 0.1 0.1 select selwait
327 twm netbsd 24 1.5 1.4 select selwait
321 xterm netbsd 24 4.2 2.5 select selwait
318 XF86_SVGA netbsd 24 216.2 36.5 select selwait
317 xinit netbsd 32 0.0 0.0 wait 0xc6dfd008
316 sh netbsd 32 0.0 0.0 wait 0xc6c55c7c
314 ssh-agent netbsd 24 1.6 0.0 select selwait
220 getty netbsd 25 0.0 0.0 ttyopn 0xc63f4008
219 getty netbsd 25 0.0 0.0 ttyin 0xc6d778f0
218 getty netbsd 25 0.0 0.0 ttyin 0xc6d777d4
217 tcsh netbsd 25 0.0 0.0 ttyin 0xc6d776b8
216 tcsh netbsd 40 0.0 0.0 pause 0xc6c3a31c
214 cron netbsd 32 0.0 0.4 nanosleep nanowait.110
210 inetd netbsd 24 0.0 0.0 select selwait
185 apmd netbsd 24 0.0 3.4 select selwait
114 rpcbind netbsd 24 0.0 0.0 select selwait
110 named netbsd 24 1.0 0.4 select selwait
102 syslogd netbsd 24 0.0 0.4 select selwait
8 aiodoned netbsd 4 0.0 0.2 aiodoned uvm+0x54
7 ioflush netbsd 40 0.0 32.4 syncer rushjob
6 reaper netbsd 4 0.0 0.1 reaper deadproc
5 pagedaemon netbsd 4 0.0 0.5 pgdaemon uvm+0x48
4 apm0 netbsd 32 0.053305.5 apmev 0xc07c6400
3 cardslot0 netbsd 32 0.0 3.1 cardslotev 0xc07b3344
2 usb0 netbsd 32 0.0 0.0 usbevt 0xc07b38c0
1 init netbsd 32 0.0 0.0 wait 0xc6c36000
0 swapper netbsd 4 0.0 0.1 scheduler proc0
db>
A pair of ps listings from 9:02am. Notice that find is stuck in vnlock:
db> PID PPID PGRP UID S FLAGS COMMAND WAIT
4893 4886 4885 0 3 0x4004 find vnlock
4888 4885 4885 0 3 0x4184 sendmail netio
4887 4885 4885 0 3 0x4084 tee netio
4886 4885 4885 0 3 0x4084 sh wait
4885 4884 4885 0 3 0x4084 sh wait
4884 214 214 0 3 0x84 cron netio
4818 4775 4818 10786 4 0x5002 less
4802 4775 4802 10786 4 0x5002 less
4794 4775 4794 10786 4 0x5002 less
4792 4775 4792 10786 4 0x5002 less
4785 4775 4785 10786 4 0x5002 less
4780 4775 4780 10786 4 0x5002 less
4775 4774 4775 10786 3 0x4086 tcsh ttyin
4774 1 321 10786 3 0x4186 xterm select
4769 398 4769 10786 3 0x5082 less ttyin
4767 398 4766 10786 4 0x5002 less
4766 398 4766 10786 4 0x5082 objdump netio
4723 332 4722 10786 4 0x5002 less
4659 1 4323 10786 3 0x4002 nm uvn_fp2
4656 4654 4323 10786 3 0x4082 tsort netio
4655 4654 4323 10786 3 0x4082 sh wait
4654 4653 4323 10786 3 0x82 sh wait
4653 4353 4323 10786 3 0x4082 sh netio
4353 4352 4323 10786 3 0x4082 make wait
4352 4323 4323 10786 3 0x4082 sh wait
4323 334 4323 10786 3 0x5082 make wait
4172 334 4171 10786 4 0x5002 less
3943 333 3943 10786 3 0x4186 ssh select
3941 1 3941 0 3 0x84 dhclient select
3767 3765 3765 10786 4 0x4182 ssh netio
3766 398 3765 10786 4 0x5002 less
3765 398 3765 10786 4 0x5082 cvs netio
3760 3758 3758 10786 4 0x4182 ssh
3759 398 3758 10786 4 0x5002 less
3758 398 3758 10786 4 0x5082 cvs netio
3744 398 3744 0 4 0x5002 tcsh
3705 398 3705 10786 4 0x5002 vi
2670 2669 2669 10786 3 0x4082 netscape select
2669 332 2669 10786 3 0x4086 netscape select
1523 1 1523 0 3 0x84 sshd1 select
398 397 398 10786 3 0x4082 tcsh pause
397 1 321 10786 3 0x4182 xterm select
374 1 374 10786 3 0x80 zhm select
351 332 351 0 4 0x5002 tcsh
334 331 334 10786 3 0x4082 tcsh pause
333 321 333 10786 3 0x4082 tcsh pause
332 330 332 10786 3 0x4082 tcsh ttyin
331 321 321 10786 3 0x4186 xterm select
330 321 321 10786 3 0x4182 xterm select
329 321 321 10786 3 0x4082 xconsole select
328 321 321 10786 3 0x4086 xclock select
327 321 321 10786 3 0x4086 twm select
321 317 321 10786 3 0x4186 xterm select
318 317 318 10786 3 0x4184 XF86_SVGA select
317 316 316 10786 3 0x4082 xinit wait
316 216 316 10786 3 0x5082 sh wait
314 1 314 10786 3 0x80 ssh-agent select
220 1 1 0 3 0x4080 getty ttyopn
219 1 219 0 3 0x4082 getty ttyin
218 1 218 0 3 0x4082 getty ttyin
217 1 217 10786 3 0x4082 tcsh ttyin
216 1 216 10786 3 0x4082 tcsh pause
214 1 214 0 3 0x84 cron nanosle
210 1 210 0 3 0x80 inetd select
185 1 185 0 3 0x84 apmd select
114 1 114 0 3 0x84 rpcbind select
110 1 110 0 3 0x84 named select
102 1 102 0 3 0x84 syslogd select
8 0 0 0 3 0x20204 aiodoned aiodone
7 0 0 0 3 0x20204 ioflush syncer
6 0 0 0 3 0x20204 reaper reaper
5 0 0 0 3 0x20204 pagedaemon pgdaemo
4 0 0 0 3 0x20204 apm0 apmev
3 0 0 0 3 0x20204 cardslot0 cardslo
2 0 0 0 3 0x20204 usb0 usbevt
1 0 1 0 3 0x4080 init wait
0 -1 0 0 3 0x20204 swapper schedul
db> PID COMMAND EMUL PRI UTIME STIME WAIT-MSG WAIT-CHANNEL
4893 find netbsd 20 3.1 14.5 vnlock 0xc6d7c0bc
4888 sendmail netbsd 24 0.0 0.0 netio 0xc08d3250
4887 tee netbsd 24 0.0 0.0 netio 0xc08d40f8
4886 sh netbsd 32 0.0 0.0 wait 0xc6f17588
4885 sh netbsd 32 0.0 0.0 wait 0xc6f17ae0
4884 cron netbsd 24 0.0 0.0 netio 0xc08d4574
4818 less netbsd 50 0.0 0.0
4802 less netbsd 50 0.0 0.0
4794 less netbsd 50 0.0 0.0
4792 less netbsd 50 0.0 0.0
4785 less netbsd 50 0.0 0.0
4780 less netbsd 50 0.0 0.0
4775 tcsh netbsd 25 0.0 0.0 ttyin 0xc63f47cc
4774 xterm netbsd 24 0.4 0.1 select selwait
4769 less netbsd 25 0.0 0.0 ttyin 0xc63f46b0
4767 less netbsd 50 0.1 0.0
4766 objdump netbsd 24 0.0 0.0 netio 0xc08d41c0
4723 less netbsd 50 0.0 0.0
4659 nm netbsd 4 0.0 0.0 uvn_fp2 0xc073eab0
4656 tsort netbsd 24 0.0 0.0 netio 0xc086b944
4655 sh netbsd 32 0.0 0.0 wait 0xc6eb13ac
4654 sh netbsd 32 0.0 0.0 wait 0xc6eb1acc
4653 sh netbsd 24 0.0 0.0 netio 0xc086bd1c
4353 make netbsd 32 0.0 0.0 wait 0xc6eb101c
4352 sh netbsd 32 0.0 0.0 wait 0xc6eb11e4
4323 make netbsd 32 0.0 0.0 wait 0xc6eb1c94
4172 less netbsd 50 0.0 0.0
3943 ssh netbsd 24 1.4 1.8 select selwait
3941 dhclient netbsd 24 0.0 0.0 select selwait
3767 ssh netbsd 24 0.1 0.0 netio 0xc08d43ac
3766 less netbsd 50 0.0 0.0
3765 cvs netbsd 24 0.0 0.0 netio 0xc08d4828
3760 ssh netbsd 24 0.0 0.0
3759 less netbsd 50 0.0 0.0
3758 cvs netbsd 24 0.0 0.0 netio 0xc086b77c
3744 tcsh netbsd 50 0.0 0.0
3705 vi netbsd 50 0.0 0.0
2670 netscape linux 24 0.0 0.0 select selwait
2669 netscape linux 24 267.4 51.6 select selwait
1523 sshd1 netbsd 24 1.9 0.0 select selwait
398 tcsh netbsd 40 0.2 0.1 pause 0xc6e75c84
397 xterm netbsd 24 5.6 2.5 select selwait
374 zhm netbsd 24 0.0 0.2 select selwait
351 tcsh netbsd 50 0.0 0.0
334 tcsh netbsd 40 0.0 0.0 pause 0xc6e75014
333 tcsh netbsd 40 0.0 0.0 pause 0xc6e5cc80
332 tcsh netbsd 25 0.0 0.2 ttyin 0xc63f4478
331 xterm netbsd 24 6.8 2.8 select selwait
330 xterm netbsd 24 2.4 1.4 select selwait
329 xconsole netbsd 24 0.0 0.0 select selwait
328 xclock netbsd 24 0.2 0.2 select selwait
327 twm netbsd 24 1.6 1.4 select selwait
321 xterm netbsd 24 4.2 2.5 select selwait
318 XF86_SVGA netbsd 24 219.2 37.0 select selwait
317 xinit netbsd 32 0.0 0.0 wait 0xc6dfd008
316 sh netbsd 32 0.0 0.0 wait 0xc6c55c7c
314 ssh-agent netbsd 24 1.6 0.0 select selwait
220 getty netbsd 25 0.0 0.0 ttyopn 0xc63f4008
219 getty netbsd 25 0.0 0.0 ttyin 0xc6d778f0
218 getty netbsd 25 0.0 0.0 ttyin 0xc6d777d4
217 tcsh netbsd 25 0.0 0.0 ttyin 0xc6d776b8
216 tcsh netbsd 40 0.0 0.0 pause 0xc6c3a31c
214 cron netbsd 32 0.0 0.4 nanosleep nanowait.110
210 inetd netbsd 24 0.0 0.0 select selwait
185 apmd netbsd 24 0.0 3.8 select selwait
114 rpcbind netbsd 24 0.0 0.0 select selwait
110 named netbsd 24 1.0 0.5 select selwait
102 syslogd netbsd 24 0.0 0.4 select selwait
8 aiodoned netbsd 4 0.0 0.2 aiodoned uvm+0x54
7 ioflush netbsd 40 0.0 36.6 syncer rushjob
6 reaper netbsd 4 0.0 0.1 reaper deadproc
5 pagedaemon netbsd 4 0.0 0.5 pgdaemon uvm+0x48
4 apm0 netbsd 32 0.053305.7 apmev 0xc07c6400
3 cardslot0 netbsd 32 0.0 3.1 cardslotev 0xc07b3344
2 usb0 netbsd 32 0.0 0.0 usbevt 0xc07b38c0
1 init netbsd 32 0.0 0.0 wait 0xc6c36000
0 swapper netbsd 4 0.0 0.1 scheduler proc0
db>
Stopped at cpu_Debugger+0x4: leave
Overnight, from 3am-9am, there were no exceptional logs. cron did the following:
Feb 11 03:00:00 zorkmid /USR/SBIN/CRON[4812]: (root) CMD (/usr/bin/newsyslog)
Feb 11 03:00:00 zorkmid /USR/SBIN/CRON[4813]: (root) CMD (/usr/libexec/atrun)
Feb 11 03:10:00 zorkmid /USR/SBIN/CRON[4829]: (root) CMD (/usr/libexec/atrun)
Feb 11 03:20:00 zorkmid /USR/SBIN/CRON[4835]: (root) CMD (/usr/libexec/atrun)
Feb 11 03:30:00 zorkmid /USR/SBIN/CRON[4838]: (root) CMD (/usr/libexec/atrun)
Feb 11 03:36:00 zorkmid /USR/SBIN/CRON[4841]: (root) CMD (/usr/sbin/sendmail -q)
Feb 11 03:40:00 zorkmid /USR/SBIN/CRON[4844]: (root) CMD (/usr/libexec/atrun)
Feb 11 03:50:00 zorkmid /USR/SBIN/CRON[4847]: (root) CMD (/usr/libexec/atrun)
Feb 11 04:00:00 zorkmid /USR/SBIN/CRON[4851]: (root) CMD (/usr/bin/newsyslog)
Feb 11 04:00:00 zorkmid /USR/SBIN/CRON[4852]: (root) CMD (/usr/libexec/atrun)
Feb 11 04:10:00 zorkmid /USR/SBIN/CRON[4856]: (root) CMD (/usr/libexec/atrun)
Feb 11 04:20:00 zorkmid /USR/SBIN/CRON[4859]: (root) CMD (/usr/libexec/atrun)
Feb 11 04:30:00 zorkmid /USR/SBIN/CRON[4862]: (root) CMD (/usr/libexec/atrun)
Feb 11 04:36:00 zorkmid /USR/SBIN/CRON[4865]: (root) CMD (/usr/sbin/sendmail -q)
Feb 11 04:40:00 zorkmid /USR/SBIN/CRON[4868]: (root) CMD (/usr/libexec/atrun)
Feb 11 04:50:00 zorkmid /USR/SBIN/CRON[4873]: (root) CMD (/usr/libexec/atrun)
Feb 11 05:00:00 zorkmid /USR/SBIN/CRON[4877]: (root) CMD (/usr/bin/newsyslog)
Feb 11 05:00:00 zorkmid /USR/SBIN/CRON[4878]: (root) CMD (/usr/libexec/atrun)
Feb 11 05:10:00 zorkmid /USR/SBIN/CRON[4882]: (root) CMD (/usr/libexec/atrun)
Feb 11 05:15:00 zorkmid /USR/SBIN/CRON[4885]: (root) CMD (/bin/sh /etc/daily 2>&1 | tee /var/log/daily.out | sendmail -t)
Feb 11 05:20:00 zorkmid /USR/SBIN/CRON[4895]: (root) CMD (/usr/libexec/atrun)
Feb 11 05:30:00 zorkmid /USR/SBIN/CRON[4898]: (root) CMD (/usr/libexec/atrun)
Feb 11 05:36:00 zorkmid /USR/SBIN/CRON[4901]: (root) CMD (/usr/sbin/sendmail -q)
Feb 11 05:40:00 zorkmid /USR/SBIN/CRON[4904]: (root) CMD (/usr/libexec/atrun)
Feb 11 05:50:00 zorkmid /USR/SBIN/CRON[4907]: (root) CMD (/usr/libexec/atrun)
Feb 11 06:00:00 zorkmid /USR/SBIN/CRON[4911]: (root) CMD (/usr/bin/newsyslog)
Feb 11 06:00:00 zorkmid /USR/SBIN/CRON[4912]: (root) CMD (/usr/libexec/atrun)
Feb 11 06:10:00 zorkmid /USR/SBIN/CRON[4916]: (root) CMD (/usr/libexec/atrun)
Feb 11 06:20:00 zorkmid /USR/SBIN/CRON[4919]: (root) CMD (/usr/libexec/atrun)
Feb 11 06:30:00 zorkmid /USR/SBIN/CRON[4922]: (root) CMD (/usr/libexec/atrun)
Feb 11 06:36:00 zorkmid /USR/SBIN/CRON[4925]: (root) CMD (/usr/sbin/sendmail -q)
Feb 11 06:40:00 zorkmid /USR/SBIN/CRON[4928]: (root) CMD (/usr/libexec/atrun)
Feb 11 06:50:00 zorkmid /USR/SBIN/CRON[4934]: (root) CMD (/usr/libexec/atrun)
Feb 11 07:00:00 zorkmid /USR/SBIN/CRON[4938]: (root) CMD (/usr/bin/newsyslog)
Feb 11 07:00:00 zorkmid /USR/SBIN/CRON[4939]: (root) CMD (/usr/libexec/atrun)
Feb 11 07:10:00 zorkmid /USR/SBIN/CRON[4943]: (root) CMD (/usr/libexec/atrun)
Feb 11 07:20:00 zorkmid /USR/SBIN/CRON[4946]: (root) CMD (/usr/libexec/atrun)
Feb 11 07:30:00 zorkmid /USR/SBIN/CRON[4949]: (root) CMD (/usr/libexec/atrun)
Feb 11 07:36:00 zorkmid /USR/SBIN/CRON[4952]: (root) CMD (/usr/sbin/sendmail -q)
Feb 11 07:40:00 zorkmid /USR/SBIN/CRON[4955]: (root) CMD (/usr/libexec/atrun)
Feb 11 07:50:00 zorkmid /USR/SBIN/CRON[4958]: (root) CMD (/usr/libexec/atrun)
Feb 11 08:00:00 zorkmid /USR/SBIN/CRON[4962]: (root) CMD (/usr/bin/newsyslog)
Feb 11 08:00:00 zorkmid /USR/SBIN/CRON[4963]: (root) CMD (/usr/libexec/atrun)
Feb 11 08:10:00 zorkmid /USR/SBIN/CRON[4967]: (root) CMD (/usr/libexec/atrun)
Feb 11 08:20:00 zorkmid /USR/SBIN/CRON[4972]: (root) CMD (/usr/libexec/atrun)
Feb 11 08:30:00 zorkmid /USR/SBIN/CRON[4975]: (root) CMD (/usr/libexec/atrun)
Feb 11 08:36:00 zorkmid /USR/SBIN/CRON[4978]: (root) CMD (/usr/sbin/sendmail -q)
Feb 11 08:40:00 zorkmid /USR/SBIN/CRON[4981]: (root) CMD (/usr/libexec/atrun)
Feb 11 08:50:00 zorkmid /USR/SBIN/CRON[4984]: (root) CMD (/usr/libexec/atrun)
Feb 11 09:00:00 zorkmid /USR/SBIN/CRON[4988]: (root) CMD (/usr/bin/newsyslog)
Feb 11 09:00:00 zorkmid /USR/SBIN/CRON[4989]: (root) CMD (/usr/libexec/atrun)
Clearly /etc/daily didn't finish running:
zorkmid% ls -ld daily.out
-rw-r--r-- 1 root wheel 157 Feb 11 05:15 daily.out
zorkmid% cat daily.out
To: root
Subject: zorkmid.mit.edu daily output for Sun Feb 11 05:15:01 EST 2001
Uptime: 5:15AM up 2 days, 10:38, 7 users, load averages: 0.10, 0.09, 0.08
zorkmid%
>Fix:
None known at this time. PR is for tracking purposes.
>Release-Note:
>Audit-Trail:
>Unformatted: