Subject: Re: sendto() and ENOBUFS question..
To: None <sudog@telus.net>
From: Jonathan Stone <jonathan@DSG.Stanford.EDU>
List: tech-net
Date: 05/15/2002 11:53:09
There are several things to bear in mind here.

First: suppose the bandwidth between your sending app, A, and your
receiving app, B, is BW.  Let the round-trip-time (RTT) between A and
B be RTT. There's little or no point having more than BW*RTT traffic
in flight between A and B.  To a first approximation (ignoring loss
and feedback effects), having more data than BW*RTT in flight just
increases queue depths without increasing actual goodput between
A and B. 


If your sender app can send faster than the pipe to your receiver app
can keep up, eventually *someone* will run out of buffers and start
dropping packets.  This is exactly what you're seeing.  (I'll skip
over some details since we don't know what kind of network you're
using, or what BW or RTT you expect).

From prior discussion it looks like you're exceeding the send-queue
limit on the socketbuffer on your sending host.  (See setsockopt
SO_SNDBUF). Once your app has queued more than this limit of data on
the socket, sosend() is going to start discarding data until the local
socket send queue drains.

The suggestion I offered before was: monitor for this condition; when
it occurs, whack the watermark up high, wait (via poll/select) for
the majority of the queued data to drain before sending more.


You can increase the size of your socket send queue, but soon you'll
find that a standing queue builds up on the outbound NIC send
queue. But NIC drivers check for "queue full condition": a NIC queue
is "full" at around 50 packets.  If you try to send more than that,
and your packets get dropped, but in the driver send queue rather
than from your app's socket send queue.

Bottom line is: the Internet drops packets. Your app should be aware
of that.  If you know upfront what kind of data rates and delay you
expect, you can precompute BW*RTT and adjust your app's sending rate
appropriately.