Re: bnx(4) lockups?

[Masanobu SAITOH <msaitoh%execsw.org@localhost>]

Hi, Stephen.

On 2019/03/19 22:27, Stephen Borrill wrote:
> I have a reliable installation on one Lenovo server which has wm(4) NICs. It has connections to the LAN and directly to the open Internet. I tried to migrate it to a different server which has bnx(4) NICs. If not connected to the Internet, it runs reliably. If connected to the Internet and traffic is flowing, it will lock solid after a while. I have encountered this previously and tracked it down to IPFilter. My workaround on that occasion was to use "pass in all/pass out all" rules (I could not disable IPFilter as I needed NAT) as anything more complex caused the lock. However, on this occasion I switched to NPF and the lockups continue. This is on netbsd-7.
>
> The hardware has been running XenServer solidly under heavy load for a number of years. Doing internal data copies (to stress the HDDs and RAID controller) is also reliable.
>
> I really can't think of much else besides the network hardware (and I don't have any spare NICs to hand). Is bnx(4) known bad?
>
> bnx0 at pci1 dev 0 function 0: Broadcom NetXtreme II BCM5709 1000Base-T
> bnx0: Ethernet address 5c:f3:fc:e4:e6:78
> bnx0: interrupting at ioapic1 pin 4
> bnx0: ASIC BCM5709 C0 (0x57092003)
> bnx0: PCIe x2 5Gbps
> bnx0: Coal (RX:6,6,18,18; TX:20,20,80,80)
> bnx1 at pci1 dev 0 function 1: Broadcom NetXtreme II BCM5709 1000Base-T
> bnx1: Ethernet address 5c:f3:fc:e4:e6:7a
> bnx1: interrupting at ioapic1 pin 16
> bnx1: ASIC BCM5709 C0 (0x57092003)
> bnx1: PCIe x2 5Gbps
> bnx1: Coal (RX:6,6,18,18; TX:20,20,80,80)
> bnx2 at pci2 dev 0 function 0: Broadcom NetXtreme II BCM5709 1000Base-T
> bnx2: Ethernet address 5c:f3:fc:6b:c6:b4
> bnx2: interrupting at ioapic1 pin 5
> bnx2: ASIC BCM5709 C0 (0x57092003)
> bnx2: PCIe x2 5Gbps
> bnx2: Coal (RX:6,6,18,18; TX:20,20,80,80)
> bnx3 at pci2 dev 0 function 1: Broadcom NetXtreme II BCM5709 1000Base-T
> bnx3: Ethernet address 5c:f3:fc:6b:c6:b6
> bnx3: interrupting at ioapic1 pin 17
> bnx3: ASIC BCM5709 C0 (0x57092003)
> bnx3: PCIe x2 5Gbps
> bnx3: Coal (RX:6,6,18,18; TX:20,20,80,80)

Could you try the following patch? This is taken from OpenBSD rev. 1.93.
Just guess. Not tested yet.


Index: if_bnx.c
===================================================================
RCS file: /cvsroot/src/sys/dev/pci/if_bnx.c,v
retrieving revision 1.68
diff -u -p -r1.68 if_bnx.c
--- if_bnx.c	22 Jan 2019 03:42:27 -0000	1.68
+++ if_bnx.c	20 Mar 2019 07:37:55 -0000
@@ -2213,6 +2213,8 @@ bnx_dma_free(struct bnx_softc *sc)
 
 	/* Destroy the status block. */
 	if (sc->status_block != NULL && sc->status_map != NULL) {
+		bus_dmamap_sync(sc->bnx_dmatag, sc->status_map, 0,
+		    sc->status_map->dm_mapsize, BUS_DMASYNC_POSTREAD);
 		bus_dmamap_unload(sc->bnx_dmatag, sc->status_map);
 		bus_dmamem_unmap(sc->bnx_dmatag, (void *)sc->status_block,
 		    BNX_STATUS_BLK_SZ);
@@ -2355,6 +2357,9 @@ bnx_dma_alloc(struct bnx_softc *sc)
 		goto bnx_dma_alloc_exit;
 	}
 
+	bus_dmamap_sync(sc->bnx_dmatag, sc->status_map, 0,
+	    sc->status_map->dm_mapsize, BUS_DMASYNC_PREREAD);
+
 	sc->status_block_paddr = sc->status_map->dm_segs[0].ds_addr;
 	memset(sc->status_block, 0, BNX_STATUS_BLK_SZ);
 
@@ -5275,7 +5280,7 @@ bnx_intr(void *xsc)
 	DBRUNIF(1, sc->interrupts_generated++);
 
 	bus_dmamap_sync(sc->bnx_dmatag, sc->status_map, 0,
-	    sc->status_map->dm_mapsize, BUS_DMASYNC_POSTWRITE);
+	    sc->status_map->dm_mapsize, BUS_DMASYNC_POSTREAD);
 
 	/*
 	 * If the hardware status block index
@@ -5354,7 +5359,7 @@ bnx_intr(void *xsc)
 	}
 
 	bus_dmamap_sync(sc->bnx_dmatag, sc->status_map, 0,
-	    sc->status_map->dm_mapsize, BUS_DMASYNC_PREWRITE);
+	    sc->status_map->dm_mapsize, BUS_DMASYNC_PREREAD);
 
 	/* Re-enable interrupts. */
 	REG_WR(sc, BNX_PCICFG_INT_ACK_CMD,



The same diff is at:

	http://www.netbsd.org/~msaitoh/bnx-20190320-0.dif

--
-----------------------------------------------
                SAITOH Masanobu (msaitoh%execsw.org@localhost
                                 msaitoh%netbsd.org@localhost)