Re: Finding out where biowait is stuck

[Stuart Brooks <stuartb%cat.co.za@localhost>]

} Subject: Re: Finding out where biowait is stuck
} 
} > On Mon, Feb 23, 2009 at 09:42:31PM +0000, Andrew Doran wrote:
} >   
} >> On Mon, Feb 23, 2009 at 09:47:42AM -0500, Allen Briggs wrote:
} >>     
} >>> The process is in src/sys/vfs_bio.c:biowait(),
} >>> but the question is why isn't it getting woken up--or if it's
} >>> getting woken up, why aren't B_DONE or B_DELWRI set?
} >>>       
} >> I closed a PR with a similar symptom last year. We had sloppy manipluation
} >> of buf::b_flag. Updates were made without any protection against disk
} >> interrupts.
} >>
} >> It was exacerbated by the advent of RISC (load, modify, store) and softdep.
} >> A large chunk of the softdep code runs in interrupt context, which broke
} >> some of the undocumented assumptions that the pre-5.0 buffer code made 
about
} >> updates to b_flag.
} >>
} >> The good news is that it is fixed in 5.0. The bad news is that there is not
} >> yet any remediation for this issue in earlier releases (that I know of).
} >>     
} >
} > Although I should note, it could be a very different problem in this
} > instance. The one I note is a good candidate, though.
} >
} >
} >   
} I'm guessing this is PR/38761 - didn't pick this up this in my search. 
} Having had a quick look it appears this code (or at least the filenames) 
} has changed quite a bit between NetBSD3/4 and 5 so it appears it's not 
} just a cut and paste to backport the fix if this is the problem. How 
} much work do you think it would be to put it into NetBSD 4, do the same 
} principles hold in the biowait code?
} 
} We run quite a few NetBSD systems and the only ones we have seen this on 
} are running easyRaid storage devices through a scsi card - the one is an 
} Adaptec, the other LSI :
} 
} mpt0 at pci3 dev 10 function 0: LSI Logic 53c1030 Ultra320 SCSI
} mpt0: interrupting at irq 15
} 
} Most of the other systems have SATA drives. So the external RAID seems to be the difference. And as I mentioned, the last few times have all coincided with the running of the daily script which would obviously work over the disks/filesystems.
} 
} Another thing which might be relevant is that these systems are receiving fairly heavy network traffic of the order of 30-40Mbps.

>         Hello.  I'm running a number of NetBSD-3 and NetBSD-4 machines with
> the mpt(4) driver in play, all be it without the raid parts.  I've seen a
> couple of hangs in the last couple of years, usually due to a disk
> responding badly to some command.  I don't think the mpt driver in NetBSD 3
> is particularly robust when it comes to error handling, and I believe there
> may be some latent timing issues in that code which cause it to fall over
> into bad cases under some circumstances.  However, it generally runs fine,
> assuming the disks are well behaved, and it does this under load.
>         All this is to say, I don't think  it's a buffer handling issue, so I
> don't think you need to worry about back porting a bunch of complicated
> buffer handling code to NetBSD-3.  What I'd do, if I wre you, is look at
> the fixes to the mpt(4) driver which have been put in since your version was
> released, and see if any relate to timing/error handling.  I'd also look at
> the Linux and FreeBSD/OpenBSD driver histories for this card and see if
> they've got anything about timing/error handling fixes as well which
> haven't been pulled into NetBSD.  I'd be willing to bet that it's a driver
> issue, and that the problem can be fixed with some changes in the driver
> code.  Either you're missing interrupts, which are causing the driver not
> to wake up when it should and return status to the biowait routines, or the
> driver is mis-handling certain commands, and failing to complete, causing
> the same effect.  turning on debug in the mpt(4) code might reveal some of
> what is going on.
>
>
>   
The thing is that this isn't only happening on this card, we've also 
seen it on the following

ahc1 at pci3 dev 10 function 0: Adaptec 29160 Ultra160 SCSI adapter
ahc1: interrupting at irq 15
ahc1: aic7892: Ultra160 Wide Channel A, SCSI Id=15, 32/253 SCBs

with the same raid devices. And we've only seen it when there are 2 raid 
devices on the card, when one is removed the problem seems to go away. 
That said, it only manifests every 3 or 4 days which makes it quite hard 
to analyse easily.

I'll have a look at the respective drivers and see if there are any 
changes which look like they might solve this,

Many thanks
Stuart