Re: Finding out where biowait is stuck

[buhrow%lothlorien.nfbcal.org@localhost (Brian Buhrow)]

        Hello.  Given that additional data, I'd say that the scsi  layer is
issuing a command the raid devices don't like, and then somehow never
timing out and re-issuing the command to wake up the raid devices again.
I'll bet if you put each raid device on a separate channel of a
dual-channel scsi card, you don't see the problem either, right?
Once the problem manifests, can you get things going again with a scsictl
/dev/sdx reset?
If so, then I'd say that our scsi stack is doing something to foobar the
raid devices, or the raid devices themselves are not complying with the
scsi standard somewhere.  In other words, I'd characterize this is a badly
behaved device/disk.
-Brian
On Feb 25,  9:22am, Stuart Brooks wrote:
} Subject: Re: Finding out where biowait is stuck
} } Subject: Re: Finding out where biowait is stuck
} } 
} } > On Mon, Feb 23, 2009 at 09:42:31PM +0000, Andrew Doran wrote:
} } >   
} } >> On Mon, Feb 23, 2009 at 09:47:42AM -0500, Allen Briggs wrote:
} } >>     
} } >>> The process is in src/sys/vfs_bio.c:biowait(),
} } >>> but the question is why isn't it getting woken up--or if it's
} } >>> getting woken up, why aren't B_DONE or B_DELWRI set?
} } >>>       
} } >> I closed a PR with a similar symptom last year. We had sloppy 
manipluation
} } >> of buf::b_flag. Updates were made without any protection against disk
} } >> interrupts.
} } >>
} } >> It was exacerbated by the advent of RISC (load, modify, store) and 
softdep.
} } >> A large chunk of the softdep code runs in interrupt context, which broke
} } >> some of the undocumented assumptions that the pre-5.0 buffer code made 
about
} } >> updates to b_flag.
} } >>
} } >> The good news is that it is fixed in 5.0. The bad news is that there is 
not
} } >> yet any remediation for this issue in earlier releases (that I know of).
} } >>     
} } >
} } > Although I should note, it could be a very different problem in this
} } > instance. The one I note is a good candidate, though.
} } >
} } >
} } >   
} } I'm guessing this is PR/38761 - didn't pick this up this in my search. 
} } Having had a quick look it appears this code (or at least the filenames) 
} } has changed quite a bit between NetBSD3/4 and 5 so it appears it's not 
} } just a cut and paste to backport the fix if this is the problem. How 
} } much work do you think it would be to put it into NetBSD 4, do the same 
} } principles hold in the biowait code?
} } 
} } We run quite a few NetBSD systems and the only ones we have seen this on 
} } are running easyRaid storage devices through a scsi card - the one is an 
} } Adaptec, the other LSI :
} } 
} } mpt0 at pci3 dev 10 function 0: LSI Logic 53c1030 Ultra320 SCSI
} } mpt0: interrupting at irq 15
} } 
} } Most of the other systems have SATA drives. So the external RAID seems to 
be the difference. And as I mentioned, the last few times have all coincided 
with the running of the daily script which would obviously work over the 
disks/filesystems.
} } 
} } Another thing which might be relevant is that these systems are receiving 
fairly heavy network traffic of the order of 30-40Mbps.
} 
} >     Hello.  I'm running a number of NetBSD-3 and NetBSD-4 machines with
} > the mpt(4) driver in play, all be it without the raid parts.  I've seen a
} > couple of hangs in the last couple of years, usually due to a disk
} > responding badly to some command.  I don't think the mpt driver in NetBSD 3
} > is particularly robust when it comes to error handling, and I believe there
} > may be some latent timing issues in that code which cause it to fall over
} > into bad cases under some circumstances.  However, it generally runs fine,
} > assuming the disks are well behaved, and it does this under load.
} >     All this is to say, I don't think  it's a buffer handling issue, so I
} > don't think you need to worry about back porting a bunch of complicated
} > buffer handling code to NetBSD-3.  What I'd do, if I wre you, is look at
} > the fixes to the mpt(4) driver which have been put in since your version was
} > released, and see if any relate to timing/error handling.  I'd also look at
} > the Linux and FreeBSD/OpenBSD driver histories for this card and see if
} > they've got anything about timing/error handling fixes as well which
} > haven't been pulled into NetBSD.  I'd be willing to bet that it's a driver
} > issue, and that the problem can be fixed with some changes in the driver
} > code.  Either you're missing interrupts, which are causing the driver not
} > to wake up when it should and return status to the biowait routines, or the
} > driver is mis-handling certain commands, and failing to complete, causing
} > the same effect.  turning on debug in the mpt(4) code might reveal some of
} > what is going on.
} >
} >
} >   
} The thing is that this isn't only happening on this card, we've also 
} seen it on the following
} 
} ahc1 at pci3 dev 10 function 0: Adaptec 29160 Ultra160 SCSI adapter
} ahc1: interrupting at irq 15
} ahc1: aic7892: Ultra160 Wide Channel A, SCSI Id=15, 32/253 SCBs
} 
} with the same raid devices. And we've only seen it when there are 2 raid 
} devices on the card, when one is removed the problem seems to go away. 
} That said, it only manifests every 3 or 4 days which makes it quite hard 
} to analyse easily.
} 
} I'll have a look at the respective drivers and see if there are any 
} changes which look like they might solve this,
} 
} Many thanks
}  Stuart
} 
>-- End of excerpt from Stuart Brooks