[src/trunk]: src/sys/dev/pci/ixgbe Don't call ixgbe_rearm_queues() in ixgbe_l...

[msaitoh <msaitoh%NetBSD.org@localhost>]

details:   https://anonhg.NetBSD.org/src/rev/079fa5c055e2
branches:  trunk
changeset: 832990:079fa5c055e2
user:      msaitoh <msaitoh%NetBSD.org@localhost>
date:      Mon Jun 04 02:42:23 2018 +0000

description:
Don't call ixgbe_rearm_queues() in ixgbe_local_timer1().

   ixgbe_enable_queue() and ixgbe_disable_queue() try to enable/disable queue
  interrupt safely. It has the internal counter. When a queue's MSI-X is
  received, ixgbe_msix_que() is called (IPL_NET). This function disable the
  queue's interrupt by ixgbe_disable_queue() and issues an softint.
  ixgbe_handle() queue is called by the softint (IPL_SOFTNET), process TX, RX
  and call ixgbe_enable_queue() at the end.

   ixgbe_local_timer1() is a callout and run always on CPU 0 (IPL_SOFTCLOCK).
  When ixgbe_rearm_queues() called, an MSI-X interrupt is issued for a specific
  queue. It may not CPU 0. If this interrupt's ixgbe_msix_que() is called and
  sofint_schedule() is called before the last sofint's softint_execute() is not
  called, the softint_schedule() fails because of SOFTINT_PENDING. It result
  in breaking ixgbe_{enable,disable}_queue()'s internal counter.

   ixgbe_local_timer1() is written not to call ixgbe_rearm_queues() if
  the interrupt is disabled, but it's called because of unknown bug or a race.
  One solution to avoid this problem is to not to use the internal counter,
  but it's little difficult. Another solution is stop using
  ixgbe_rearm_queues() at all. Essentially, ixgbe_rearm_queues() is not
  required (it was added in ixgbe.c rev. 1.43 (2016/12/01)).
  ixgbe_rearm_queues() helps for lost interrupt problem but I've never seen it
  other than ixgbe_rearm_queues() problem.

XXX pullup-8.

diffstat:

 sys/dev/pci/ixgbe/ixgbe.c |  6 ++++--
 sys/dev/pci/ixgbe/ixv.c   |  4 +++-
 2 files changed, 7 insertions(+), 3 deletions(-)

diffs (55 lines):

diff -r 72eb505ef8f4 -r 079fa5c055e2 sys/dev/pci/ixgbe/ixgbe.c
--- a/sys/dev/pci/ixgbe/ixgbe.c Mon Jun 04 02:29:53 2018 +0000
+++ b/sys/dev/pci/ixgbe/ixgbe.c Mon Jun 04 02:42:23 2018 +0000
@@ -1,4 +1,4 @@
-/* $NetBSD: ixgbe.c,v 1.159 2018/06/03 10:24:24 maxv Exp $ */
+/* $NetBSD: ixgbe.c,v 1.160 2018/06/04 02:42:23 msaitoh Exp $ */
 
 /******************************************************************************
 
@@ -4411,6 +4411,7 @@
        /* Only truely watchdog if all queues show hung */
        if (hung == adapter->num_queues)
                goto watchdog;
+#if 0 /* XXX Avoid unexpectedly disabling interrupt forever (PR#53294) */
        else if (queues != 0) { /* Force an IRQ on queues with work */
                que = adapter->queues;
                for (i = 0; i < adapter->num_queues; i++, que++) {
@@ -4421,6 +4422,7 @@
                        mutex_exit(&que->dc_mtx);
                }
        }
+#endif
 
 out:
        callout_reset(&adapter->timer, hz, ixgbe_local_timer, adapter);
@@ -6643,7 +6645,7 @@
 /************************************************************************
  * ixgbe_rearm_queues
  ************************************************************************/
-static void
+static __inline void
 ixgbe_rearm_queues(struct adapter *adapter, u64 queues)
 {
        u32 mask;
diff -r 72eb505ef8f4 -r 079fa5c055e2 sys/dev/pci/ixgbe/ixv.c
--- a/sys/dev/pci/ixgbe/ixv.c   Mon Jun 04 02:29:53 2018 +0000
+++ b/sys/dev/pci/ixgbe/ixv.c   Mon Jun 04 02:42:23 2018 +0000
@@ -1,4 +1,4 @@
-/*$NetBSD: ixv.c,v 1.103 2018/06/03 10:24:24 maxv Exp $*/
+/*$NetBSD: ixv.c,v 1.104 2018/06/04 02:42:23 msaitoh Exp $*/
 
 /******************************************************************************
 
@@ -1266,9 +1266,11 @@
        /* Only truly watchdog if all queues show hung */
        if (hung == adapter->num_queues)
                goto watchdog;
+#if 0
        else if (queues != 0) { /* Force an IRQ on queues with work */
                ixv_rearm_queues(adapter, queues);
        }
+#endif
 
        callout_reset(&adapter->timer, hz, ixv_local_timer, adapter);