Subject: Re: fixing send(2) semantics (kern/29750)
To: Emmanuel Dreyfus <manu@netbsd.org>
From: Jonathan Stone <jonathan@dsg.stanford.edu>
List: tech-kern
Date: 03/27/2005 01:05:24
In message <1gu2tib.1bjnrlifs45k2M%manu@netbsd.org>,
Emmanuel Dreyfus writes:

>Jonathan Stone <jonathan@dsg.stanford.edu> wrote:
>
>> Emmanuel, why don't you tell us what your actual problem is, and see
>> if we can help you with it, instead of suggesting ideas which are
>> well-known in the relevant field to be bad ideas, or incorrect (though
>> 100% sincere, I am sure) claims that these bad ideas are somehow
>> mandated by standards?
>
>See my other mail: I want to transmit UDP packets using full link
>bandwidth, and it cannot be acheived on NetBSD because:

Ooops. I just saw Mouse's reply quoting some of an earlier message; I
nver received that message. I guess maybe I deleted it accendally
(pilot error or multi-char-keystroke-parsing lossage) due to the
horrible packet loss I've had thi evening.

If so, I apologize for asking you to send something you had already
sent, but which I didn't (and still haven't) seen.

>- If I don't sleep between sends, the queue get full, I get ENOBUFS, I
>have to retry sending. And in the meantime the network interface
>consumed all the data on the queue. The queue is empty and the network
>is unused for some time.
>
>- If I sleep between send, or even if I do a single system call, the
>interface consume packets faster than I add them. The queue is empty and
>the network is unused for some time.  


That sounds rather, um, unusual. I have no absoulutely no atrouble
saturating gigabit links with a single ttcp TCP stream with what,
today, are fairly modest machines: 1.25GHz P3s.  Socket bufeers on the
order of 3*64Kibytes, bge with checksum offload (needs patches), bge
interrupt mitigation via sysctl; otherwise out-of-the-box.  Has been
since the 1.25Ghz machine was brand new and top-of-the-line.

But, on another note: the classis cmistake #1 of students who passed
an intro course in networking: to decide TCP has "too much" overhead
for their application, and to write their own lower-overhead protocol
on UDP. Like Plan 9 in the movie, this plan almost never works.

If our application really needs to resend, then perhaps you should be
using TCP instead?  Tuned with sufficiently large socket buffers,
interrupt mitigation, etc?  What is it that you need to do at such a
high rate, and why are you usind UDP if you need to recover from
packet loss?