Re: Panic: vrelel: bad ref count (9.99.54)

[Yorick Hardy <yorickhardy%gmail.com@localhost>]

Dear Andrew,

On 2020-05-05, Andrew Doran wrote:
> On Mon, May 04, 2020 at 03:54:57PM +0200, Leonardo Taccari wrote:
> > Hello Yorick and Andrew,
> > 
> > Yorick Hardy writes:
> > > > > > [...]
> > > > > > 
> > > > > >   Crash version 9.99.55, image version 9.99.55.
> > > > > >   crash: _kvm_kvatop(0)
> > > > > >   Kernel compiled without options LOCKDEBUG.
> > > > > >   System panicked: vrelel: bad ref count
> > > > > >   Backtrace from time of crash is available.
> > > > > >   crash> bt
> > > > > >   _KERNEL_OPT_NAGR() at 0
> > > > > >   ?() at 7f7ff7ecf000
> > > > > >   sys_reboot() at sys_reboot
> > > > > >   vpanic() at vpanic+0x181
> > > > > >   vtryrele() at vtryrele
> > > > > >   vcache_dealloc() at vcache_dealloc
> > > > > >   uvm_unmap_detach() at uvm_unmap_detach+0x76
> > > > > >   uvm_unmap1() at uvm_unmap1+0x4e
> > > > > >   uvm_mremap() at uvm_mremap+0x36b
> > > > > >   sys_mremap() at sys_mremap+0x68
> > > > > >   syscall() at syscall+0x227
> > > > > >   --- syscall (number 411) ---
> > > > > >   797459842e9a:
> > > > > >   crash>
> > > > > 
> > > > > The same just happened on 9.99.56 while fetching (POP) mail using mail/fdm.
> > > > 
> > > > Could you file a PR please?  If this panics again could you please run the
> > > > "dmesg" command in crash and find out what it printed about the vnode?  That
> > > > would be very useful.
> > > > 
> > > > Thanks,
> > > > Andrew
> > >
> > > I will do so (... perhaps only this weekend).
> > > [...]
> > 
> > I was able to reproduce it too with a yesterday evening NetBSD/amd64
> > -current when using mail/fdm and I will try to prepare a minimal
> > reproducer using mail/fdm and file a PR if noone beat me.
> > 
> > In the meantime here the information from dmesg:
> > 
> > [ 6107.6380323] vnode 0xffffa95219747d40 flags 0x418<MAPPED,MPSAFE,WRMAP>
> > [ 6107.6380323]    tag VT_TMPFS(25) type VREG(1) mount 0xffffa951f6d89000 typedata 0xffffa95255e32c90
> > [ 6107.6380323]    usecount 1 writecount 1 holdcount 0
> > [ 6107.6380323]    size 18000 writesize 18000 numoutput 0
> > [ 6107.6380323]    data 0xffffa952583304a0 lock 0xffffa95219747f00
> > [ 6107.6380323]    state LOADED key(0xffffa951f6d89000 8) a0 04 33 58 52 a9 ff ff
> > [ 6107.6380323]    lrulisthd 0xffffffff816b5ed0
> > [ 6107.6380323]    tag VT_TMPFS, tmpfs_node 0xffffa952583304a0, flags 0x0, links 1
> > [ 6107.6380323]    mode 0600, owner 1000, group 0, size 98304
> > [ 6107.6380323] panic: vrelel: bad ref count
> > [ 6107.6380323] cpu0: Begin traceback...
> > [ 6107.6380323] vpanic() at netbsd:vpanic+0x178
> > [ 6107.6480364] vnpanic() at netbsd:vnpanic+0x49
> > [ 6107.6480364] vrelel() at netbsd:vrelel+0x5b6
> > [ 6107.6480364] uvm_unmap_detach() at netbsd:uvm_unmap_detach+0x8e
> > [ 6107.6480364] sys_munmap() at netbsd:sys_munmap+0x85
> > [ 6107.6480364] syscall() at netbsd:syscall+0x2a0
> > [ 6107.6480364] --- syscall (number 73) ---
> > [ 6107.6480364] 7c1e5d18414a:
> > [ 6107.6480364] cpu0: End traceback...
> > [ 6107.6480364] fatal breakpoint trap in supervisor mode
> > [ 6107.6480364] trap type 1 code 0 rip 0xffffffff802219fd cs 0x8 rflags 0x202 cr2 0x7f7ff7ee5000 ilevel 0 rsp 0xffffc100c227ae20
> > [ 6107.6480364] curlwp 0xffffa9521e1b1600 pid 20756.20756 lowest kstack 0xffffc100c22772c0
> > [ 6107.6480364] dumping to dev 0,1 (offset=276847, size=2062664):
> > [ 6107.6480364] dump
> > 
> > If any possible further information is needed do not hesitate to
> > contact me!
> > 
> > 
> > Thanks!
> 
> Thank you.  I opened PR 55237 to track so I don't forget.
> 
> Andrew

I am still trying to track this down, but I can only understand small
pieces of the code at the moment. While going through uvm_map_enter
in sys/uvm/uvm_map.c, it looks like there is an unhandled case (patch
below). Is this correct? It seems to happen quite often, but with or
without the patch the system seems equally (un)stable.

-- 
Kind regards,

Yorick Hardy

Index: uvm_map.c
===================================================================
RCS file: /cvsroot/src/sys/uvm/uvm_map.c,v
retrieving revision 1.385
diff -u -r1.385 uvm_map.c
--- uvm_map.c	9 Jul 2020 05:57:15 -0000	1.385
+++ uvm_map.c	19 Nov 2020 16:04:07 -0000
@@ -1477,6 +1477,13 @@
 				    amapwaitflag | AMAP_EXTEND_BACKWARDS))
 					goto nomerge;
 			}
+
+			/*
+			 * We could not extend either amap, just skip on.
+			 */
+			else {
+				goto nomerge;
+			}
 		} else {
 			/*
 			 * Pull the next entry's amap backwards to cover this