re: kern/52526: Panic on "ifconfig athn0 down"

[matthew green <mrg%eterna.com.au@localhost>]

i've been seeing odd panics with usb ethernet lately, the same sort
of problem across several devices.  i saw the same problem on 
cdce(4), axen(4), and ure(4).  this one looks the same as well.

i suspect there is a new problem in if.c calls to if_start(), since
the exact same problem occurs on all 3 for me, and 4th for you.

inspired by mlelstv's changes to if_mue.c (providing a mutex around
the if_start call, effectively), which seems to help some, but i
still see crashes in cdce_start() fairly often, but the cause did
change from two cases two one case.

previously, it would crash with most likely this trace:

panic: kernel diagnostic assertion "xfer->ux_state == XFER_BUSY" failed: file "/usr/src/sys/dev/usb/usbdi.c", line 1039 xfer 0xffff0000f5cf7b80 state is 9e
fp ffffffc060998ab0 kern_assert() at ffffffc00058dc4c netbsd:kern_assert+0x5c
fp ffffffc060998b40 usb_insert_transfer() at ffffffc0000c6dd0 netbsd:usb_insert_transfer+0xe0
fp ffffffc060998b60 ehci_device_bulk_transfer() at ffffffc00016d54c netbsd:ehci_device_bulk_transfer+0x2c
fp ffffffc060998b90 usbd_transfer() at ffffffc0000c5a28 netbsd:usbd_transfer+0x98
fp ffffffc060998bc0 cdce_start() at ffffffc0000f5978 netbsd:cdce_start+0xa8
fp ffffffc060998bf0 cdce_txeof() at ffffffc0000f5b78 netbsd:cdce_txeof+0x118
fp ffffffc060998c40 usb_transfer_complete() at ffffffc0000c6aa8 netbsd:usb_transfer_complete+0x1c8
fp ffffffc060998c90 ehci_softintr() at ffffffc00016e7d4 netbsd:ehci_softintr+0x174

or less likely without the mutex, but *only* with the mutex:

panic: kernel diagnostic assertion "xfer->ux_state == XFER_BUSY" failed: file "/usr/src/sys/dev/usb/usbdi.c", line 1039 xfer 0xffff0000f5cf7b80 state is 9e

fp ffffffc06198f760 kern_assert() at ffffffc00058dccc netbsd:kern_assert+0x5c
fp ffffffc06198f7f0 usb_insert_transfer() at ffffffc0000c6dd0 netbsd:usb_insert_transfer+0xe0
fp ffffffc06198f810 ehci_device_bulk_transfer() at ffffffc00016d5cc netbsd:ehci_device_bulk_transfer+0x2c
fp ffffffc06198f840 usbd_transfer() at ffffffc0000c5a28 netbsd:usbd_transfer+0x98
fp ffffffc06198f870 cdce_start() at ffffffc0000f59a8 netbsd:cdce_start+0xb8
fp ffffffc06198f8b0 if_transmit() at ffffffc0004eabf0 netbsd:if_transmit+0x150
fp ffffffc06198f8f0 ether_output() at ffffffc0004f73e4 netbsd:ether_output+0x244
fp ffffffc06198f950 ip_if_output() at ffffffc00021e1bc netbsd:ip_if_output+0xac
fp ffffffc06198f990 ip_output() at ffffffc00021fa80 netbsd:ip_output+0x1160

the patch i'm using is below.


.mrg.

Index: if_cdce.c
===================================================================
RCS file: /cvsroot/src/sys/dev/usb/if_cdce.c,v
retrieving revision 1.47
diff -p -u -r1.47 if_cdce.c
--- if_cdce.c	5 May 2019 03:17:54 -0000	1.47
+++ if_cdce.c	14 Jun 2019 21:26:06 -0000
@@ -271,6 +271,8 @@ cdce_attach(device_t parent, device_t se
 		eaddr[5] = (uint8_t)(device_unit(sc->cdce_dev));
 	}
 
+	mutex_init(&sc->cdce_start_lock, MUTEX_DEFAULT, IPL_NONE);
+
 	s = splnet();
 
 	aprint_normal_dev(self, "address %s\n", ether_sprintf(eaddr));
@@ -326,6 +328,8 @@ cdce_detach(device_t self, int flags)
 	sc->cdce_attached = 0;
 	splx(s);
 
+	mutex_destroy(&sc->cdce_start_lock);
+
 	return 0;
 }
 
@@ -338,12 +342,17 @@ cdce_start(struct ifnet *ifp)
 	if (sc->cdce_dying || (ifp->if_flags & IFF_OACTIVE))
 		return;
 
+	mutex_enter(&sc->cdce_start_lock);
+
 	IFQ_POLL(&ifp->if_snd, m_head);
-	if (m_head == NULL)
+	if (m_head == NULL) {
+		mutex_exit(&sc->cdce_start_lock);
 		return;
+	}
 
 	if (cdce_encap(sc, m_head, 0)) {
 		ifp->if_flags |= IFF_OACTIVE;
+		mutex_exit(&sc->cdce_start_lock);
 		return;
 	}
 
@@ -353,6 +362,8 @@ cdce_start(struct ifnet *ifp)
 
 	ifp->if_flags |= IFF_OACTIVE;
 
+	mutex_exit(&sc->cdce_start_lock);
+
 	ifp->if_timer = 6;
 }
 
Index: if_cdcereg.h
===================================================================
RCS file: /cvsroot/src/sys/dev/usb/if_cdcereg.h,v
retrieving revision 1.9
diff -p -u -r1.9 if_cdcereg.h
--- if_cdcereg.h	23 Apr 2016 10:15:31 -0000	1.9
+++ if_cdcereg.h	14 Jun 2019 21:26:06 -0000
@@ -85,4 +85,6 @@ struct cdce_softc {
 	int			 cdce_rxeof_errors;
 	uint16_t		 cdce_flags;
 	char			 cdce_attached;
+
+	kmutex_t		 cdce_start_lock;
 };