Subject: Re: System running out of vnodes
To: None <tech-kern@NetBSD.org>
From: der Mouse <mouse@Rodents.Montreal.QC.CA>
List: tech-kern
Date: 03/25/2005 15:51:52
> There is some kind of serious bug here.  Things should run slower if
> you fill up the vnode table, but they should still run -- the table
> should be managed by LRU.

I recently saw a failure mode of a 2.0 machine that may be related.
This is (part of) the console log of a SPARCclassic running 2.0 (with
serial console).  It had been up about six days when the problem
started early this morning.

login: Mar 19 04:54:56 NetBSD-2-0 su: mouse to root on /dev/ttyp1
Mar 25 00:07:43 NetBSD-2-0 ntpd[210]: can't open /var/db/ntp.drift.TEMP: Input/output error
Mar 25 00:30:00 NetBSD-2-0 syslogd: /var/log/cron: Input/output error
Mar 25 01:07:43 NetBSD-2-0 ntpd[210]: can't open /var/db/ntp.drift.TEMP: Input/output error
Mar 25 02:07:43 NetBSD-2-0 /netbsd: uid 0, pid 210, command ntpd, on /: out of inodes
Mar 25 02:07:43 NetBSD-2-0 ntpd[210]: can't open /var/db/ntp.drift.TEMP: No space left on device
Mar 25 03:07:43 NetBSD-2-0 /netbsd: uid 0, pid 210, command ntpd, on /: out of inodes
Mar 25 03:07:44 NetBSD-2-0 syslogd: /var/log/messages: No space left on device
Mar 25 03:07:44 NetBSD-2-0 syslogd: /var/log/messages: No space left on device
Mar 25 03:07:44 NetBSD-2-0 /netbsd: uid 0, pid 113, command syslogd, on /: file system full
Mar 25 03:07:44 NetBSD-2-0 /netbsd: uid 0, pid 113, command syslogd, on /: file system full
Mar 25 03:07:43 NetBSD-2-0 ntpd[210]: can't open /var/db/ntp.drift.TEMP: No space left on device
Mar 25 03:08:40 NetBSD-2-0 su: mouse to root on /dev/ttyp1

I had a window open on the machine (which is where the last su line
came from).  Some commands (ls, su, df, everything involved in my
.cshrc) worked; some (halt) reported I/O errors.

Note the lack of any disk error messages.  The "out of inodes" error
seems to have been spurious; df -i reported lots of inodes free.  I had
to halt the machine ungracefully; on reboot, fsck was happy with the
filesystem - it wasn't clean, of course, but fsck didn't report any
problems:

Starting file system checks:
/dev/rsd0e: 206752 files, 1994977 used, 3786509 free (164173 frags, 452792 blocks, 2.8% fragmentation)
/dev/rsd0e: MARKING FILE SYSTEM CLEAN
/dev/rsd0a: 13 files, 38472 used, 24999 free (7 frags, 3124 blocks, 0.0% fragmentation)
/dev/rsd0a: MARKING FILE SYSTEM CLEAN

I don't know whether it actually is related, but the spurious "out of
inodes" sounds possibly related to vnode exhaustion.

/~\ The ASCII				der Mouse
\ / Ribbon Campaign
 X  Against HTML	       mouse@rodents.montreal.qc.ca
/ \ Email!	     7D C8 61 52 5D E7 2D 39  4E F1 31 3E E8 B3 27 4B