Re: Unallocated inode

[Paul Ripke <stix%stix.id.au@localhost>]

On Wed, Nov 26, 2014 at 09:37:22PM +1100, Paul Ripke wrote:
> On Tue, Nov 25, 2014 at 08:42:29AM -0500, Christos Zoulas wrote:
> > On Nov 25,  1:22pm, stix%stix.id.au@localhost (Paul Ripke) wrote:
> > -- Subject: Re: Unallocated inode
> > 
> > | On Mon, Nov 24, 2014 at 08:01:57PM -0500, Christos Zoulas wrote:
> > | > On Nov 25, 10:59am, stix%stix.id.au@localhost (Paul Ripke) wrote:
> > | > -- Subject: Re: Unallocated inode
> > | > 
> > | > | On Tue, Sep 02, 2014 at 06:15:49PM -0400, Christos Zoulas wrote:
> > | > | > On Sep 2,  2:27pm, stix%stix.id.au@localhost (Paul Ripke) wrote:
> > | > | > -- Subject: Re: Unallocated inode
> > | > | > 
> > | > | > | Yeah, that's what scares me - it was the daily rsync and security cron
> > | > | > | jobs starting to generate errors that alerted me - those inodes must've
> > | > | > | been marked partially allocated only recently. Makes me wish I dumped
> > | > | > | out the contents of those two inodes before running the fsck (maybe
> > | > | > | fsdb should be able to do that?).
> > | > | > 
> > | > | > Yes, I think fsdb can do that...
> > | > | 
> > | > | Ok, this happened again. And with a tweaked fsdb, I get:
> > | > | 
> > | > | fsdb (inum: 125770625)> print
> > | > | command `print
> > | > | '
> > | > | current inode 125770625: unallocated inode
> > | > | current inode: unallocated
> > | > | I=125770625 MODE=0 SIZE=0
> > | > |         MTIME=Jan  1 10:00:00 1970 [0 nsec]
> > | > |         CTIME=Nov 25 10:34:43 2014 [282996506 nsec]
> > | > |         ATIME=Jan  1 10:00:00 1970 [0 nsec]
> > | > | OWNER=root GRP=wheel LINKCNT=0 FLAGS=0x0 BLKCNT=0x0 GEN=0x0
> > | > | 
> > | > | Nothing was being touched in that part of the filesystem at the time...
> 
> s/touched/written to/
> 
> > | > | I think the CTIME may be related to me mv'ing the file elsewhere, the
> > | > | time somewhat matches. I find this somewhat scary. For reference:
> > | > 
> > | > Is that inode accessible from the filesystem? Can you reproduce this
> > | > running find on that subtree?
> > | 
> > | Nope, it's inaccessible - again, it was the daily security reports
> > | that alerted me:
> > | 
> > | slave:ksh$ find /home/tmp > /dev/null
> > | find: /home/tmp/badfile: Bad file descriptor
> > | 
> > | (I renamed the inaccessible file into a convenient location as I
> > | couldn't figure out in 10 seconds how to cd thru directories with
> > | spaces in the names in fsdb...)
> > 
> > Looks like a filesystem race/bug, but i am not sure what your usage
> > pattern is that triggers it.
> 
> That's my hunch... while it's just a home server with a lot of junk
> running in that filesystem, as yet all files I've seen with corrupted
> inodes would've either last been touched by mediatomb doing its regular
> filesystem scan for changed files, and/or possibly simultaneously with
> rsync update of netbsd src repo. All the corrupted inodes would have been
> either cold, with no updates apart from perhaps atime, or being updated
> by rsync.

It's been a while, but here we are again:

slave:ksh$ uname -a
NetBSD slave 7.0_RC2 NetBSD 7.0_RC2 (SLAVE) #5: Sun Aug  9 16:17:05 EST 2015  stix@slave:/h
slave:ksh$ find /home/stix/src/compilers/llvm/llvm/.svn/pristine -ls > /dev/null
find: /home/stix/src/compilers/llvm/llvm/.svn/pristine/85: Bad file descriptor
find: /home/stix/src/compilers/llvm/llvm/.svn/pristine/b4: Bad file descriptor
ome/netbsd/netbsd-7/obj.amd64/home/netbsd/netbsd-7/src/sys/arch/amd64/compile/SLAVE amd64

This time, they're directories, not files. And they're cold, apart
from find(1) and dump(8). Perhaps there's a one in a billion race with
st_atime updates?

Also:

fsdb (inum: 62772027)> cd 85
command `cd 85
'
component `85': current inode 71225600: unallocated inode
current inode: unallocated
I=71225600 MODE=20 SIZE=110740547584
        MTIME=Jun  2 17:06:40 1970 [-1992935936 nsec]
Memory fault (core dumped) 

yay :/

-- 
Paul Ripke
"Great minds discuss ideas, average minds discuss events, small minds
 discuss people."
-- Disputed: Often attributed to Eleanor Roosevelt. 1948.