Source-Changes-HG archive

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index][Old Index]

[src/trunk]: src Modify some parameters to reduce packet dropping.



details:   https://anonhg.NetBSD.org/src/rev/85a31e4d7650
branches:  trunk
changeset: 960144:85a31e4d7650
user:      msaitoh <msaitoh%NetBSD.org@localhost>
date:      Tue Mar 09 10:03:18 2021 +0000

description:
Modify some parameters to reduce packet dropping.

 - Background: ixgbe doesn't use common MCLGET() interface and use the
   driver specific cluster allocation mechanism (jcl). The cluster is
   pre-allocated with a fixed number and the current number per queue
   is num_rx_desc * 2 (2048*2=4096). It's too small. It also has a problem
   that the max length of the pcq which is used in the TX path is big
   (4096). Example:

    100M <----- [ixg0                                      ixg1] <----- 1G
                2048 TX descs <--- 4096 pcqs <---- 2048 RX descs

   If a machine forwards a traffic from 1G interface to 100M interface,
   It would require 2048+4096+2048=8192 descriptors, but the current number
   is 2048*2=4096. It's too small. Even if the both interface's link speed
   is the same and only small number of packet is queued in the pcq, 4096
   jcl is small because 2048(RX)+TX(2048)=4096. If jcl is exhausted, not only
   forwarding from ixg1 to ixg0 is dropped, but also another forwarding path
   from ixg1 to another interface(e.g. wm0) is also dropped. Sockets also
   queue packets, so if a lot of sockets are used and/or a socket buffer
   size is changed to bigger one, it'll also become a problem. If the jcl
   is exhausted, evcnt(9) counter "ixgX qY Rx no jumbo mbuf" is incremented.
   Example:
     vmstat -ev | grep ixg1 | grep "no jumbo"
     ixg1 q0 Rx no jumbo mbuf                           0     0 misc
     ixg1 q1 Rx no jumbo mbuf                           0     0 misc
     ixg1 q2 Rx no jumbo mbuf                      141326     0 misc
     ixg1 q3 Rx no jumbo mbuf                           0     0 misc


 - To solve this problem:
   - Add new config parameter IXGBE_JCLNUM_MULTI and set the default to 3
     (2048 * 3). The minimum number is 2. The total number of jcl per queue
     is available with hw.ixgN.num_jcl_per_queue sysctl.
   - Reduce the max length of the pcq() which is used in the TX path from
     4096 to 2048.

 - Reviewed by knakahara@ and ozaki-r@.

 - TODO: Use MCLGET().

diffstat:

 share/man/man4/ixg.4             |  22 ++++++++++++++++++++--
 share/man/man4/ixv.4             |  22 ++++++++++++++++++++--
 sys/dev/pci/files.pci            |   3 ++-
 sys/dev/pci/ixgbe/ix_txrx.c      |   4 ++--
 sys/dev/pci/ixgbe/ixgbe.c        |  12 +++++++++++-
 sys/dev/pci/ixgbe/ixgbe.h        |   5 +++--
 sys/dev/pci/ixgbe/ixgbe_netbsd.h |  16 +++++++++++++++-
 sys/dev/pci/ixgbe/ixv.c          |  12 +++++++++++-
 8 files changed, 84 insertions(+), 12 deletions(-)

diffs (257 lines):

diff -r 464cff7af163 -r 85a31e4d7650 share/man/man4/ixg.4
--- a/share/man/man4/ixg.4      Tue Mar 09 01:17:37 2021 +0000
+++ b/share/man/man4/ixg.4      Tue Mar 09 10:03:18 2021 +0000
@@ -1,4 +1,4 @@
-.\" $NetBSD: ixg.4,v 1.12 2018/05/09 08:01:16 wiz Exp $
+.\" $NetBSD: ixg.4,v 1.13 2021/03/09 10:03:18 msaitoh Exp $
 .\"
 .\" Copyright (c) 2001-2008, Intel Corporation
 .\" All rights reserved.
@@ -33,7 +33,7 @@
 .\"
 .\" $FreeBSD: src/share/man/man4/ixgbe.4,v 1.3 2010/12/19 23:54:31 yongari Exp $
 .\"
-.Dd May 9, 2018
+.Dd March 9, 2021
 .Dt IXG 4
 .Os
 .Sh NAME
@@ -83,6 +83,24 @@
 .\" with a supported adapter, email the specific information related to the
 .\" issue to
 .\" .Aq freebsdnic%mailbox.intel.com@localhost .
+.Sh OPTIONS
+The
+.Nm
+driver doesn't use the common
+.Xr MCLGET 9
+interface and use the driver specific cluster allocation mechanism.
+If it's exhausted,
+.Xr evcnt 9
+counter "ixgX qY Rx no jumbo mbuf" is incremented. If it's observed,
+The number can be changed by the following config parameter:
+.Bl -tag -width IXGBE_JCLNUM_MULTI -offset 3n
+.It Dv IXGBE_JCLNUM_MULTI
+The number of RX jumbo buffer (cluster) per queue is calculated by
+.Dv IXGBE_JCLNUM_MULTI
+* (number of rx descriptor).
+The total number of clusters per queue is available with
+hw.ixgN.num_jcl_per_queue sysctl.
+.El
 .Sh SEE ALSO
 .Xr arp 4 ,
 .Xr ixv 4 ,
diff -r 464cff7af163 -r 85a31e4d7650 share/man/man4/ixv.4
--- a/share/man/man4/ixv.4      Tue Mar 09 01:17:37 2021 +0000
+++ b/share/man/man4/ixv.4      Tue Mar 09 10:03:18 2021 +0000
@@ -1,4 +1,4 @@
-.\"    $NetBSD: ixv.4,v 1.5 2019/09/05 10:01:30 msaitoh Exp $
+.\"    $NetBSD: ixv.4,v 1.6 2021/03/09 10:03:18 msaitoh Exp $
 .\"
 .\" Copyright (c) 2018 The NetBSD Foundation, Inc.
 .\" All rights reserved.
@@ -27,7 +27,7 @@
 .\" ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
 .\" POSSIBILITY OF SUCH DAMAGE.
 .\"
-.Dd September 5, 2019
+.Dd March 9, 2021
 .Dt IXV 4
 .Os
 .Sh NAME
@@ -43,6 +43,24 @@
 It can be used on a
 .Nx
 guest that the host supports SR-IOV.
+.Sh OPTIONS
+The
+.Nm
+driver doesn't use the common
+.Xr MCLGET 9
+interface and use the driver specific cluster allocation mechanism.
+If it's exhausted,
+.Xr evcnt 9
+counter "ixgX qY Rx no jumbo mbuf" is incremented. If it's observed,
+The number can be changed by the following config parameter:
+.Bl -tag -width IXGBE_JCLNUM_MULTI -offset 3n
+.It Dv IXGBE_JCLNUM_MULTI
+The number of RX jumbo buffer (cluster) per queue is calculated by
+.Dv IXGBE_JCLNUM_MULTI
+* (number of rx descriptor).
+The total number of clusters per queue is available with
+hw.ixgN.num_jcl_per_queue sysctl.
+.El
 .Sh SEE ALSO
 .Xr arp 4 ,
 .Xr ixg 4 ,
diff -r 464cff7af163 -r 85a31e4d7650 sys/dev/pci/files.pci
--- a/sys/dev/pci/files.pci     Tue Mar 09 01:17:37 2021 +0000
+++ b/sys/dev/pci/files.pci     Tue Mar 09 10:03:18 2021 +0000
@@ -1,4 +1,4 @@
-#      $NetBSD: files.pci,v 1.435 2021/03/01 17:48:52 jakllsch Exp $
+#      $NetBSD: files.pci,v 1.436 2021/03/09 10:03:18 msaitoh Exp $
 #
 # Config file and device description for machine-independent PCI code.
 # Included by ports that need it.  Requires that the SCSI files be
@@ -664,6 +664,7 @@
 file   dev/pci/ixgbe/ixgbe_vf.c        ixg | ixv
 file   dev/pci/ixgbe/if_bypass.c       ixg | ixv
 file   dev/pci/ixgbe/if_fdir.c         ixg | ixv
+defparam opt_ixgbe.h   IXGBE_JCLNUM_MULTI
 
 # This appears to be the driver for virtual instances of i82599.
 device ixv: ether, ifnet, arp, mii, mii_phy
diff -r 464cff7af163 -r 85a31e4d7650 sys/dev/pci/ixgbe/ix_txrx.c
--- a/sys/dev/pci/ixgbe/ix_txrx.c       Tue Mar 09 01:17:37 2021 +0000
+++ b/sys/dev/pci/ixgbe/ix_txrx.c       Tue Mar 09 10:03:18 2021 +0000
@@ -1,4 +1,4 @@
-/* $NetBSD: ix_txrx.c,v 1.66 2021/03/08 07:10:45 msaitoh Exp $ */
+/* $NetBSD: ix_txrx.c,v 1.67 2021/03/09 10:03:18 msaitoh Exp $ */
 
 /******************************************************************************
 
@@ -1507,7 +1507,7 @@
         * Assume all of rxr->ptag are the same.
         */
        ixgbe_jcl_reinit(adapter, rxr->ptag->dt_dmat, rxr,
-           (2 * adapter->num_rx_desc), adapter->rx_mbuf_sz);
+           adapter->num_jcl, adapter->rx_mbuf_sz);
 
        IXGBE_RX_LOCK(rxr);
 
diff -r 464cff7af163 -r 85a31e4d7650 sys/dev/pci/ixgbe/ixgbe.c
--- a/sys/dev/pci/ixgbe/ixgbe.c Tue Mar 09 01:17:37 2021 +0000
+++ b/sys/dev/pci/ixgbe/ixgbe.c Tue Mar 09 10:03:18 2021 +0000
@@ -1,4 +1,4 @@
-/* $NetBSD: ixgbe.c,v 1.278 2021/01/14 05:47:35 msaitoh Exp $ */
+/* $NetBSD: ixgbe.c,v 1.279 2021/03/09 10:03:18 msaitoh Exp $ */
 
 /******************************************************************************
 
@@ -67,6 +67,7 @@
 #include "opt_inet.h"
 #include "opt_inet6.h"
 #include "opt_net_mpsafe.h"
+#include "opt_ixgbe.h"
 #endif
 
 #include "ixgbe.h"
@@ -982,6 +983,8 @@
        } else
                adapter->num_rx_desc = ixgbe_rxd;
 
+       adapter->num_jcl = adapter->num_rx_desc * IXGBE_JCLNUM_MULTI;
+
        /* Allocate our TX/RX Queues */
        if (ixgbe_allocate_queues(adapter)) {
                error = ENOMEM;
@@ -3366,6 +3369,13 @@
                aprint_error_dev(dev, "could not create sysctl\n");
 
        if (sysctl_createv(log, 0, &rnode, &cnode,
+           CTLFLAG_READONLY, CTLTYPE_INT, "num_jcl_per_queue",
+           SYSCTL_DESCR("Number of jumbo buffers per queue"),
+           NULL, 0, &adapter->num_jcl, 0, CTL_CREATE,
+           CTL_EOL) != 0)
+               aprint_error_dev(dev, "could not create sysctl\n");
+
+       if (sysctl_createv(log, 0, &rnode, &cnode,
            CTLFLAG_READONLY, CTLTYPE_INT,
            "num_queues", SYSCTL_DESCR("Number of queues"),
            NULL, 0, &adapter->num_queues, 0, CTL_CREATE, CTL_EOL) != 0)
diff -r 464cff7af163 -r 85a31e4d7650 sys/dev/pci/ixgbe/ixgbe.h
--- a/sys/dev/pci/ixgbe/ixgbe.h Tue Mar 09 01:17:37 2021 +0000
+++ b/sys/dev/pci/ixgbe/ixgbe.h Tue Mar 09 10:03:18 2021 +0000
@@ -1,4 +1,4 @@
-/* $NetBSD: ixgbe.h,v 1.74 2020/12/26 06:07:16 msaitoh Exp $ */
+/* $NetBSD: ixgbe.h,v 1.75 2021/03/09 10:03:18 msaitoh Exp $ */
 
 /******************************************************************************
   SPDX-License-Identifier: BSD-3-Clause
@@ -216,7 +216,7 @@
 #define IXGBE_TSO_SIZE                  262140
 #define IXGBE_RX_HDR                    128
 #define IXGBE_VFTA_SIZE                 128
-#define IXGBE_BR_SIZE                   4096
+#define IXGBE_BR_SIZE                   2048
 #define IXGBE_QUEUE_MIN_FREE            32
 #define IXGBE_MAX_TX_BUSY               10
 #define IXGBE_QUEUE_HUNG                0x80000000
@@ -568,6 +568,7 @@
        u64                     active_queues;
        u32                     num_rx_desc;
        u32                     rx_process_limit;
+       int                     num_jcl;
 
        /* Multicast array memory */
        struct ixgbe_mc_addr    *mta;
diff -r 464cff7af163 -r 85a31e4d7650 sys/dev/pci/ixgbe/ixgbe_netbsd.h
--- a/sys/dev/pci/ixgbe/ixgbe_netbsd.h  Tue Mar 09 01:17:37 2021 +0000
+++ b/sys/dev/pci/ixgbe/ixgbe_netbsd.h  Tue Mar 09 10:03:18 2021 +0000
@@ -1,4 +1,4 @@
-/*$NetBSD: ixgbe_netbsd.h,v 1.11 2019/03/05 08:25:02 msaitoh Exp $*/
+/*$NetBSD: ixgbe_netbsd.h,v 1.12 2021/03/09 10:03:18 msaitoh Exp $*/
 /*
  * Copyright (c) 2011 The NetBSD Foundation, Inc.
  * All rights reserved.
@@ -40,6 +40,20 @@
 #define        MJUM16BYTES     (16 * 1024)
 #define        MJUMPAGESIZE    PAGE_SIZE
 
+/*
+ * Number of jcl per queue is calculated by
+ * adapter->num_rx_desc * IXGBE_JCLNUM_MULTI. The lower limit is 2.
+ */
+#define        IXGBE_JCLNUM_MULTI_LOWLIM       2
+#define        IXGBE_JCLNUM_MULTI_DEFAULT      3
+#if !defined(IXGBE_JCLNUM_MULTI)
+# define IXGBE_JCLNUM_MULTI IXGBE_JCLNUM_MULTI_DEFAULT
+#else
+# if (IXGBE_JCLNUM_MULTI < IXGBE_JCLNUM_MULTI_LOWLIM)
+#  error IXGBE_JCLNUM_MULTI is too low.
+# endif
+#endif
+
 #define IFCAP_RXCSUM   \
        (IFCAP_CSUM_IPv4_Rx|IFCAP_CSUM_TCPv4_Rx|IFCAP_CSUM_UDPv4_Rx|\
        IFCAP_CSUM_TCPv6_Rx|IFCAP_CSUM_UDPv6_Rx)
diff -r 464cff7af163 -r 85a31e4d7650 sys/dev/pci/ixgbe/ixv.c
--- a/sys/dev/pci/ixgbe/ixv.c   Tue Mar 09 01:17:37 2021 +0000
+++ b/sys/dev/pci/ixgbe/ixv.c   Tue Mar 09 10:03:18 2021 +0000
@@ -1,4 +1,4 @@
-/*$NetBSD: ixv.c,v 1.154 2020/09/07 05:50:58 msaitoh Exp $*/
+/*$NetBSD: ixv.c,v 1.155 2021/03/09 10:03:18 msaitoh Exp $*/
 
 /******************************************************************************
 
@@ -38,6 +38,7 @@
 #include "opt_inet.h"
 #include "opt_inet6.h"
 #include "opt_net_mpsafe.h"
+#include "opt_ixgbe.h"
 #endif
 
 #include "ixgbe.h"
@@ -512,6 +513,8 @@
        } else
                adapter->num_rx_desc = ixv_rxd;
 
+       adapter->num_jcl = adapter->num_rx_desc * IXGBE_JCLNUM_MULTI;
+
        /* Setup MSI-X */
        error = ixv_configure_interrupts(adapter);
        if (error)
@@ -2558,6 +2561,13 @@
                aprint_error_dev(dev, "could not create sysctl\n");
 
        if (sysctl_createv(log, 0, &rnode, &cnode,
+           CTLFLAG_READONLY, CTLTYPE_INT, "num_jcl_per_queue",
+           SYSCTL_DESCR("Number of jumbo buffers per queue"),
+           NULL, 0, &adapter->num_jcl, 0, CTL_CREATE,
+           CTL_EOL) != 0)
+               aprint_error_dev(dev, "could not create sysctl\n");
+
+       if (sysctl_createv(log, 0, &rnode, &cnode,
            CTLFLAG_READWRITE, CTLTYPE_BOOL,
            "enable_aim", SYSCTL_DESCR("Interrupt Moderation"),
            NULL, 0, &adapter->enable_aim, 0, CTL_CREATE, CTL_EOL) != 0)



Home | Main Index | Thread Index | Old Index