Re: wm(4) and the maximum buffer length for TSO

[David Young <dyoung%pobox.com@localhost>]

On Thu, Sep 20, 2012 at 09:12:00AM +0200, Manuel Bouyer wrote:
> On Wed, Sep 19, 2012 at 06:34:15PM -0500, David Young wrote:
> > wm(4) sets up its Tx DMA maps like this,
> > 
> >                 if ((error = bus_dmamap_create(sc->sc_dmat, WM_MAXTXDMA,
> >                             WM_NTXSEGS, WTX_MAX_LEN, 0, 0,
> >                             &sc->sc_txsoft[i].txs_dmamap)) != 0) {
> > 
> > WM_MAXTXDMA is round_page(IP_MAXPACKET) == round_page(65535) ==
> > 65536.  Thus wm(4) will fail to map for Tx any mbuf whose m_pkthdr.len
> > > 65536.  That's ok if tcp_output() produces a buffer no longer
> > than 65536 bytes for the NIC to segment, but in practice it will
> > produce a longer buffer because first it clamps the length to
> > IP_MAXPACKET,
> > 
> >                 if (use_tso) {
> >                         /*
> >                          * Truncate TSO transfers to IP_MAXPACKET, and make
> >                          * sure that we send equal size transfers down the
> >                          * stack (rather than big-small-big-small-...).
> >                          */
> > #ifdef INET6
> >                         CTASSERT(IPV6_MAXPACKET == IP_MAXPACKET);
> > #endif
> >                         len = (min(len, IP_MAXPACKET) / txsegsize) * 
> > txsegsize;
> > 
> >     ...
> > 
> > and then it adds in the combined length of the IP and TCP headers:
> > 
> >         m->m_pkthdr.len = hdrlen + len;
> > 
> > In this way, wm(4) can see m->m_pkthdr.len greater than 65536 and fail
> > to map m.  It will send no feedback to TCP to stop trying to send such
> > long un-segmented buffers.  Also, it looks to me like it will retry
> > forever to map the same mbuf for DMA---that matches the misbehavior that
> > we're seeing at $DAYJOB, where the wm(4) ceases to transmit anything.
> 
> I's strange that I didn't run into this, I do use TSO with wm(4).
> I guess that the problem is dependant on the value of txsegsize:
> if its value is right, len will be rounded down and there is
> enough space for the header.

I rounded 65535 using several values of txsegsize that I thought were
likely, but none seemed to yield len that wasn't substantially less than
65535, so there should have been plenty of room for a header.  I'll have
to log the txsegsize sometime and see what is actually chosen.

> > and tcp_output() should be very careful not to send a buffer any
> > longer than what is supported.  What do you think?
> 
> I think we need something like that.

Ok.  It seems that tcp_output() should be rejiggered to compute the
hdrlen before finalizing len (if that is possible) and then to clamp len
at (IP_MAXPACKET - hdrlen).  I guess that we can prevail upon the driver
to create a DMA map whose size is IP_MAXPACKET + ethernet header length
+ other encapsulation overhead (e.g., VLAN tag), but I *think* I would
prefer that all of the encapsulation overhead between TCP and the wire
was conveyed to TCP.  Maybe in the TCP/IP Stack Overhaul of Our Dreams,
we can do something like that.

Dave

-- 
David Young
dyoung%pobox.com@localhost    Urbana, IL    (217) 721-9981