ixg(4) mutex problem (PR#49328)

[Masanobu SAITOH <msaitoh%execsw.org@localhost>]

 Hi.

　Uwe Toenjes found a bug in ixg(4). See:

	http://mail-index.netbsd.org/current-users/2014/10/11/msg025932.html

Then, PR#49328 was submitted by him.

	http://gnats.netbsd.org/49328

This problem is can be reproduced when

	LOCKDEBUG is defined
	ifconfig ixg0 up

ixg(4) take a spin mutex before accessing MAC core:

------ ixgbe.h -------
#define IXGBE_CORE_LOCK_INIT(_sc, _name) \
        mutex_init(&(_sc)->core_mtx, MUTEX_DEFAULT, IPL_NET)

------- ixgbe.c::ixbge_ioctl() ------
        case SIOCADDMULTI:
        case SIOCDELMULTI:
        case SIOCSIFFLAGS:
        case SIOCSIFMTU:
        default:
                if ((error = ether_ioctl(ifp, command, data)) != ENETRESET)
                        return error;
                if ((ifp->if_flags & IFF_RUNNING) == 0)
                        ;
                else if (command == SIOCSIFCAP || command == SIOCSIFMTU) {
                        IXGBE_CORE_LOCK(adapter);
                        ixgbe_init_locked(adapter);
                        IXGBE_CORE_UNLOCK(adapter);

In this case, IXGBE_CORE_LOCK() is used to prevent interrupt.

I borrowed an old Intel 10G card(82598 based) from my colleague and tested.
I got a panic like as follows:

--------------------------------------------------------
Mutex error: lockdebug_barrier: spin lock held

lock address : 0xffff80001d0eaa08 type     :               spin
initialized  : 0xffffffff805cb716
shared holds :                  0 exclusive:                  1
shares wanted:                  0 exclusive:                  0
current cpu  :                  7 last held:                  7
current lwp  : 0xfffffe840d2049c0 last held: 0xfffffe840d2049c0
last locked* : 0xffffffff805ca63e unlocked : 000000000000000000
owner field  : 0x0000000000010600 wait/spin:                0/1

panic: LOCKDEBUG: Mutex error: lockdebug_barrier: spin lock held
cpu7: Begin traceback...
vpanic() at netbsd:vpanic+0x13c
snprintf() at netbsd:snprintf
lockdebug_more() at netbsd:lockdebug_more
mutex_enter() at netbsd:mutex_enter+0x45e
pmap_growkernel() at netbsd:pmap_growkernel+0x2d
uvm_map_prepare() at netbsd:uvm_map_prepare+0x24d
uvm_map() at netbsd:uvm_map+0x97
uvm_km_alloc() at netbsd:uvm_km_alloc+0xec
_bus_dmamem_map.isra.7() at netbsd:_bus_dmamem_map.isra.7+0x69
ixgbe_jcl_reinit() at netbsd:ixgbe_jcl_reinit+0x180
ixgbe_init_locked() at netbsd:ixgbe_init_locked+0x510
ixgbe_init() at netbsd:ixgbe_init+0x22
ether_ioctl() at netbsd:ether_ioctl+0xc8
ixgbe_ioctl() at netbsd:ixgbe_ioctl+0x41
in6_update_ifa1() at netbsd:in6_update_ifa1+0x70e
in6_update_ifa() at netbsd:in6_update_ifa+0x36
in6_ifattach_linklocal() at netbsd:in6_ifattach_linklocal+0x2eb
in6_ifattach() at netbsd:in6_ifattach+0x15d
in6_if_up() at netbsd:in6_if_up+0xf
ifioctl_common() at netbsd:ifioctl_common+0x343
ether_ioctl() at netbsd:ether_ioctl+0x162
ixgbe_ioctl() at netbsd:ixgbe_ioctl+0x41
doifioctl() at netbsd:doifioctl+0x493
soo_ioctl() at netbsd:soo_ioctl+0x2af
sys_ioctl() at netbsd:sys_ioctl+0x17e
syscall() at netbsd:syscall+0x9a
--- syscall (number 54) ---
7f7ff6cced0a:
cpu7: End traceback...
--------------------------------------------------------------

 ifconfig ixg0 up
 -> ioctl()
   -> take spin mutex
   -> alloc a lot of dmamem and dmamap.
   -> it got a shortage of the kernel address space and called pmap_growkernel()
     -> tried to get adaptive mutex.
       -> panic

 My stack trace is different from Uew's traces, but the root of the
problem is the same.

 How should we fix this problem?

 Some of Ethernet drivers avoid using splnet()/splx() and use mutexes.
wm, bnx, ixg, powerpc/booke/dev/pq3etsec.c and others. I think it's better
to know the common solution and puse the way.

 I'm not so familiar with mutex and I have no experience using mutex...
Any patches are welcome. I'll try them if you send me :-)

 Thanks.

--
-----------------------------------------------
                SAITOH Masanobu (msaitoh%execsw.org@localhost
                                 msaitoh%netbsd.org@localhost)