Re: IFQ_ENQUEUE argument refactor (was Re: RFC: ALTQ caller refactor)

[Kengo NAKAHARA <k-nakahara%iij.ad.jp@localhost>]

Hi,

On 2016/04/04 22:05, Thor Lancelot Simon wrote:
> On Mon, Apr 04, 2016 at 05:25:14PM +0900, Kengo NAKAHARA wrote:
>>
>> You are right. Hmm..., I'm sorry, I change my objective. I think it could
>> be better to use pool cache for m_tag itself.
> 
> The tags are a mess and I've seen them seriously impact performance in
> numerous profiles of the network stack.  In fact, at a former employer,
> it was sort of an expected thing: each time we'd experiment with some
> new feature of the stack (vlan tagging, altq, etc.) we'd see a
> performance degradation which profiling would revel traced back to the
> allocation, use, and deallocation of mbuf tags.

It's very interesting. Could you tell me in detail? (of course, if it
does not violate NDA).

I measured the ALTQ performance of 3 patterns, that is
    - NetBSD-current(using stack variable)
    - pattern 1(using malloc m_tag)
    - pattern 2(using pool cache m_tag)
As you indicated, pattern 1 much less performance than current.
However, pattern 2 is almost the same performance as current.
e.g. when the frame size is 128 byte, the throughput is about
147,800 fps, and the duration time is 20 sec, the number of dropped
packets are the following.

 run number | current | pattern 1 | pattern 2
------------+---------+-----------+-----------
          1 |      21 |   571,779 |       721
          2 |       0 |   568,329 |       766
          3 |     741 |   577,228 |        22
          4 |     650 |   561,325 |       774
          5 |     730 |   568,038 |         0

For this result, I think my ALTQ pool cache implementation (pattern 2)
does not degrade performance.

I think the changing from malloc m_tag to pool cache m_tag does not
degrade performance, either. As the latency of pool cache m_tag is
less than one of malloc m_tag.
    http://mail-index.netbsd.org/tech-net/2016/02/19/msg005633.html

> I would suggest that the only "proper" use of mbuf tags as currently
> implemented is experimentation.  Uses of the tags which have been in
> the tree for a decade or more should be subsumed into the mbuf
> datastructure itself, with some care taken to not make it too large,
> or perhaps a single, "standard" extension structure could be defined
> to capture the less-common cases.

Hmm, sorry, I would not understand well... I have two question.

(Q1) Is harmful m_tag performance increasing?
Even if it is used for experimentation, it is better to reduce known
performance issue, I think.

(Q2) How do I decide the data is too large or not?
e.g. ALTQ case, the data is struct altq_pktattr whose members are void *,
int, and void *.
    http://nxr.netbsd.org/xref/src/sys/altq/if_altq.h#86
Is this three member struct too large?
another PACKET_TAG_PF m_tag case, the data is struct pf_mtag
    http://nxr.netbsd.org/xref/src/sys/dist/pf/net/pf_mtag.h#46
How about this pattern?


Thanks,

-- 
//////////////////////////////////////////////////////////////////////
Internet Initiative Japan Inc.

Device Engineering Section,
IoT Platform Development Department,
Network Division,
Technology Unit

Kengo NAKAHARA <k-nakahara%iij.ad.jp@localhost>