Re: kernel_lock, splbio() and SMP_SAFE

[Manuel Bouyer <bouyer%antioche.eu.org@localhost>]

On Mon, Aug 03, 2009 at 04:37:11PM +0100, Mindaugas Rasiukevicius wrote:
> Hello,
> 
> Manuel Bouyer <bouyer%antioche.eu.org@localhost> wrote:
> > given a low-level subsystem (say, a driver) which is not SMP-safe
> > and is using the old spl() locking scheme, how is this subsystem
> > protected from SMP-save upper-level subsystems calling one of its
> > entry point ? I would expect kernel_lock() to be taken at some
> > point, but I can't find where it's happening in this case.
> >
> > ...
> > cpu_dumpconf(0,0,14,0,0,c03098a0,0,0,c07090b8,0) at netbsd:cpu_dumpconf
> > +0x37 main(0,c01002a7,0,0,0,0,0,0,0,0) at netbsd:main+0x28b
> > ...
> > 
> > I can't believe we have such a big synchronisation issue in our kernel
> > that would have been unnoticed for so long. Can someone explain me how this
> > is supposed to work ? 
> > 
> 
> Why big? It seems to be a very specific and rare case i.e. cpu_dumpconf() is
> called very early and calls RAIDframe in unusual way.

Yes, this is what I found later

> 
> Kernel-lock is handled in upper-level subsystems (regarding I/O, please see
> biodone2() in vfs_bio.c, intr_biglock_wrapper() for interrupt handlers, etc).

that's what I understood. This is also why I wondered why wdstart()
would be called without biglock.

FWIW I'm tracking down a hard hang of netbsd-5 on a NFS server here.
I'm starting to wonder if the hardware could be at fault. When this happens
ddb's 'show event' doens't report any move, so I suspect either the
kernel is stuck at IPL_VM or some interrupts stay masked at the ioapic
level. interrupts are not completely masked because I can enter DDB.
I couldn't get usefull backtraces from the i386 kernel.
ps /l always show softbio/0 and softser/0 as SLONPROC in addition with
some other thread (could be idle/0 or some userland process) which looks odd.
I couldn't get usefull backtraces from the i386 kernel.
From a amd64 kernel I got, for the 3 threads marked SLONPROC on proc0:
db{0}> tr/a ffff80004546cbc0 /* this is softser/0 */
trace: pid 0 lid 6 at 0xffff800046778a00
breakpoint() at netbsd:breakpoint+0x5
comintr() at netbsd:comintr+0x53a
Xintr_ioapic_edge17() at netbsd:Xintr_ioapic_edge17+0xef
--- interrupt ---
Xspllower() at netbsd:Xspllower+0xe
intr_biglock_wrapper() at netbsd:intr_biglock_wrapper+0x16
Xintr_ioapic_level14() at netbsd:Xintr_ioapic_level14+0xf7
--- interrupt ---
Xspllower() at netbsd:Xspllower+0xe 
DDB lost frame for netbsd:Xsoftintr+0x50, trying 0xffff800046778d70
Xsoftintr() at netbsd:Xsoftintr+0x50
--- interrupt ---
0:
db{0}> tr/a ffff80004546a3e0 /* softbio/0 */
trace: pid 0 lid 4 at 0xffff80004676fd60
Xsoftintr() at netbsd:Xsoftintr+0x50
--- interrupt ---
0:
db{0}> tr/a ffff80004546aba0 /* idle/0 */
trace: pid 0 lid 2 at 0xffff800046765c40
idle_loop() at netbsd:idle_loop+0xda

gdb shows:
0xffffffff80115970 <Xspllower+0>:       push   %rbx
0xffffffff80115971 <Xspllower+1>:       push   %r13
0xffffffff80115973 <Xspllower+3>:       push   %r12
0xffffffff80115975 <Xspllower+5>:       mov    %edi,%ebx
0xffffffff80115977 <Xspllower+7>:       lea    0(%rip),%r13        # 
0xffffffff8011597e <Xspllower+14>
0xffffffff8011597e <Xspllower+14>:      mov    %ebx,%eax
0xffffffff80115980 <Xspllower+16>:      mov    %gs:0x364(,%rax,4),%eax
0xffffffff80115988 <Xspllower+24>:      cli    
0xffffffff80115989 <Xspllower+25>:      and    %gs:0x328,%eax
[...]
0xffffffff80115804 <Xsoftintr+68>:      sti    
0xffffffff80115805 <Xsoftintr+69>:      mov    %r15,%rdi
0xffffffff80115808 <Xsoftintr+72>:      mov    0x0(%rax),%esi
0xffffffff8011580b <Xsoftintr+75>:      callq  0xffffffff803adb10 
<softint_dispatch>
0xffffffff80115810 <Xsoftintr+80>:      cli    


Any idea on how to debug this further is welcome. Unfortunably I can only
reproduce this on the production server so debugging sessions are limited.


> 
> Objective, of course, to fully remove it (which could mean pushing deeper
> to lower layers). Talking about ATA system, I hope you can help on this. :)

Sure, but lets kill existing bugs before introducing new ones :)

-- 
Manuel Bouyer, LIP6, Universite Paris VI.           
Manuel.Bouyer%lip6.fr@localhost
     NetBSD: 26 ans d'experience feront toujours la difference
--