tech-kern archive

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index][Old Index]

Re: Finding out where biowait is stuck



} Subject: Re: Finding out where biowait is stuck
} } > On Mon, Feb 23, 2009 at 09:42:31PM +0000, Andrew Doran wrote: } > } >> On Mon, Feb 23, 2009 at 09:47:42AM -0500, Allen Briggs wrote: } >> } >>> The process is in src/sys/vfs_bio.c:biowait(),
} >>> but the question is why isn't it getting woken up--or if it's
} >>> getting woken up, why aren't B_DONE or B_DELWRI set?
} >>> } >> I closed a PR with a similar symptom last year. We had sloppy manipluation
} >> of buf::b_flag. Updates were made without any protection against disk
} >> interrupts.
} >>
} >> It was exacerbated by the advent of RISC (load, modify, store) and softdep.
} >> A large chunk of the softdep code runs in interrupt context, which broke
} >> some of the undocumented assumptions that the pre-5.0 buffer code made 
about
} >> updates to b_flag.
} >>
} >> The good news is that it is fixed in 5.0. The bad news is that there is not
} >> yet any remediation for this issue in earlier releases (that I know of).
} >> } >
} > Although I should note, it could be a very different problem in this
} > instance. The one I note is a good candidate, though.
} >
} >
} > } I'm guessing this is PR/38761 - didn't pick this up this in my search. } Having had a quick look it appears this code (or at least the filenames) } has changed quite a bit between NetBSD3/4 and 5 so it appears it's not } just a cut and paste to backport the fix if this is the problem. How } much work do you think it would be to put it into NetBSD 4, do the same } principles hold in the biowait code? } } We run quite a few NetBSD systems and the only ones we have seen this on } are running easyRaid storage devices through a scsi card - the one is an } Adaptec, the other LSI : } } mpt0 at pci3 dev 10 function 0: LSI Logic 53c1030 Ultra320 SCSI
} mpt0: interrupting at irq 15
} } Most of the other systems have SATA drives. So the external RAID seems to be the difference. And as I mentioned, the last few times have all coincided with the running of the daily script which would obviously work over the disks/filesystems. } } Another thing which might be relevant is that these systems are receiving fairly heavy network traffic of the order of 30-40Mbps.

        Hello.  I'm running a number of NetBSD-3 and NetBSD-4 machines with
the mpt(4) driver in play, all be it without the raid parts.  I've seen a
couple of hangs in the last couple of years, usually due to a disk
responding badly to some command.  I don't think the mpt driver in NetBSD 3
is particularly robust when it comes to error handling, and I believe there
may be some latent timing issues in that code which cause it to fall over
into bad cases under some circumstances.  However, it generally runs fine,
assuming the disks are well behaved, and it does this under load.
        All this is to say, I don't think  it's a buffer handling issue, so I
don't think you need to worry about back porting a bunch of complicated
buffer handling code to NetBSD-3.  What I'd do, if I wre you, is look at
the fixes to the mpt(4) driver which have been put in since your version was
released, and see if any relate to timing/error handling.  I'd also look at
the Linux and FreeBSD/OpenBSD driver histories for this card and see if
they've got anything about timing/error handling fixes as well which
haven't been pulled into NetBSD.  I'd be willing to bet that it's a driver
issue, and that the problem can be fixed with some changes in the driver
code.  Either you're missing interrupts, which are causing the driver not
to wake up when it should and return status to the biowait routines, or the
driver is mis-handling certain commands, and failing to complete, causing
the same effect.  turning on debug in the mpt(4) code might reveal some of
what is going on.


The thing is that this isn't only happening on this card, we've also seen it on the following

ahc1 at pci3 dev 10 function 0: Adaptec 29160 Ultra160 SCSI adapter
ahc1: interrupting at irq 15
ahc1: aic7892: Ultra160 Wide Channel A, SCSI Id=15, 32/253 SCBs

with the same raid devices. And we've only seen it when there are 2 raid devices on the card, when one is removed the problem seems to go away. That said, it only manifests every 3 or 4 days which makes it quite hard to analyse easily.

I'll have a look at the respective drivers and see if there are any changes which look like they might solve this,

Many thanks
Stuart



Home | Main Index | Thread Index | Old Index