Re: AF_UNIX socketpair dgram queue sizes

[Jan Schaumann <jschauma%netmeister.org@localhost>]

Jan Schaumann <jschauma%netmeister.org@localhost> wrote:
> I'm trying to wrap my head around the buffer sizes
> relevant to AF_UNIX/PF_LOCAL dgram socketpairs.

Still working on this, and I'm making some progress in
my understanding of mbufs here.

So:

An mbuf is 512 bytes in size, with an m_hdr consuming
56 bytes, and a packet header consuming 56 bytes.

If I write a small amount of data, then I only need
one mbuf of type MT_DATA, so can squeeze in 400 bytes.

If I write more than 856 bytes of data, then I get an
mbuf of type MT_DATA with flags M_EXT set, i.e., an
mbuf cluster.

For any amount of data >400 but <= 856, I get one mbuf
of type MT_DATA with flags M_PKTHDR set, capable of
holding 400 bytes plus a second mbuf of type MT_DATA
with up to 456 bytes.

After I allocate these chains, I then prepend an mbuf
of type MT_SONAME (and size 2 bytes for the socketpair
socket "name") to the beginning of the chain.

So for any dgram of size 1 byte to 400 bytes, I will
need two mbufs (one of type MT_SONAME plus one of type
MT_DATA); for a dgram of size 401 bytes up to 856
bytes, I will need three mbufs (one SONAME, two
DATA), but for a dgram of size >401 bytes, I only need
two mbufs (one SONAME, one DATA with M_EXT).

Given a socketpair with the read end having a
SO_RCVBUF set to 2048 bytes, I now observe the
following:

1) I can write 4 dgrams of size 1 up until 400 bytes
before my next attempt to write another 1 byte dgram
fails with ENOBUFS.  This uses 8 512 byte mbufs.

2) I can write 3 dgrams of size 401 to 856 bytes
before my next attempt to write another dgram fails
with ENOBUFS.  This uses 9 512 byte mbufs.

(If I write 856 bytes, the third datagram will only
carry 330 bytes for a total of 856 + 856 + 330 = 2042
bytes, which, when we add the two bytes from the
SONAME mbuf for each, adds up to 2048, i.e.,
SNDBUF.)

3) I can write 2 dgrams of size 857 bytes before my
next attempt yields ENOBUFS.  This uses 4 mbufs,
writing 1718 bytes in total.


My question at this point is: when does my mbuf
allocation fail? I.e., what is the limit here?

In (1), I allocated 8 mbufs = 4096 bytes = 2 * SO_RCVBUF.

In (2), I allocated 9 mbufs = 4608 bytes = 2 * SO_RCVBUF + 512.
But I ended up writing less data.

In (3), I allocate 4 mbufs = 2048 bytes = SO_RCVBUF.


These correlations don't make much sense to me.  What
am I missing?

-Jan