First off, I apologize in advance for beating a dead horse over.. and over.. 
and over. :)

On Wednesday 15 May 2002 11:53, Jonathan Stone wrote:
>
> First: suppose the bandwidth between your sending app, A, and your
> receiving app, B, is BW.  Let the round-trip-time (RTT) between A and
> B be RTT. There's little or no point having more than BW*RTT traffic
> in flight between A and B.  To a first approximation (ignoring loss
> and feedback effects), having more data than BW*RTT in flight just
> increases queue depths without increasing actual goodput between
> A and B. 

This I understand. Increasing queue depth is simply a resource hog which 
doesn't send packets any faster than the underlying mechanism can support. 
Reaching this theoretical maximum on a LAN is something I'm aiming to hit 
close to.

> If your sender app can send faster than the pipe to your receiver app
> can keep up, eventually *someone* will run out of buffers and start
> dropping packets.  This is exactly what you're seeing.  (I'll skip
> over some details since we don't know what kind of network you're
> using, or what BW or RTT you expect).

It's just a dinky little 10Mb hub right now. I would hope to see at least 1 
MB/sec in raw throughput, to match a similar TCP stream. This much I expect 
to be able to do. But if the mbufs are chewed up, it'd be nice to block 
somehow instead of regularly polling to check the writable status with a 
sendto(). 

> From prior discussion it looks like you're exceeding the send-queue
> limit on the socketbuffer on your sending host.  (See setsockopt
> SO_SNDBUF). Once your app has queued more than this limit of data on
> the socket, sosend() is going to start discarding data until the local
> socket send queue drains.
> 
> The suggestion I offered before was: monitor for this condition; when
> it occurs, whack the watermark up high, wait (via poll/select) for
> the majority of the queued data to drain before sending more.

So, grab the ENOBUFS, diddle with the watermark, and then send more. The 
problem I saw was that the watermark was by default 2048--larger than the 800 
byte packets I was sending, and neither select() nor poll() were blocking. :) 
I don't suppose you have another suggestion for me do you? =]

> You can increase the size of your socket send queue, but soon you'll
> find that a standing queue builds up on the outbound NIC send
> queue. But NIC drivers check for "queue full condition": a NIC queue
> is "full" at around 50 packets.  If you try to send more than that,
> and your packets get dropped, but in the driver send queue rather
> than from your app's socket send queue.

This is interesting--if the condition is detectable shouldn't the socket send 
queue stop trying to deliver packets? If not, then most of this is irrelevant 
since a working select() would just move the drops down lower and I'd be back 
where I started. I could definitely live with that. I mean if I have to 
interpret ENOBUFS I really don't mind. I just think that select()'ing for 
write on SOCK_DGRAMS feels broken. Purely my opinion of course. :)

> Bottom line is: the Internet drops packets. Your app should be aware
> of that.  If you know upfront what kind of data rates and delay you
> expect, you can precompute BW*RTT and adjust your app's sending rate
> appropriately.

This is precisely the kind of control I want to hand to the logic in 
userland, and (possibly) to the end user.

Thank you for your note,
Marc