Re: kern/54435: reading TCP urgent data with MSG_OOB doesn't clear poll(2) status

[Robert Elz <kre%munnari.OZ.AU@localhost>]

The following reply was made to PR kern/54435; it has been noted by GNATS.

From: Robert Elz <kre%munnari.OZ.AU@localhost>
To: Valery Ushakov <uwe%stderr.spb.ru@localhost>
Cc: gnats-bugs%netbsd.org@localhost
Subject: Re: kern/54435: reading TCP urgent data with MSG_OOB doesn't clear poll(2) status
Date: Thu, 08 Aug 2019 10:19:08 +0700

     Date:        Thu, 8 Aug 2019 04:21:46 +0300
     From:        Valery Ushakov <uwe%stderr.spb.ru@localhost>
     Message-ID:  <20190808012146.GC1102%pony.stderr.spb.ru@localhost>
 
   | See my previous reply to the PR.
 
 Yes, I saw that after I sent my reply (... not that it really
 matters, but just as an explanation - I was reading backwards
 through the messages ... that way I see what has been answered
 before I see the questions which means I don't need to think
 about whether I should attempt an answer in many cases) - your
 two messages were close together, with the same Subject (just
 like happened this time) - and I just assumed that (like happened
 this time) there was one copy sent via gnats, and another to either
 one of the lists, or direct to me, so I just skipped the earlier one 
 initially, but then noticed it later).
 
 Your point in it that it is entirely possible that no-one has ever
 written a poll/select application that deals with MSG_OOB with TCP
 without using SO_OOBINLINE is probably correct though.  Not that it
 should really matter, as no-one should be writing TCP applications
 that use OOB without setting SO_OOBINLINE in the receiver.  It simply
 is the wrong way.
 
   | and then calls poll with POLLIN|POLLPRI to decide what to do next, it
   | will end up in an busy-wait loop as poll will immediately return
   | POLLPRI (only), but recv(MSG_OOB) will fail, then the receiver calls
   | poll and the loop repeats.
 
 That looks like a bug somewhere - certainly in code that would do
 a recv(), get an EINVAL, and then repeat the same recv() again (regardless
 of an intervening poll()) - but it is possible the kernel also has a
 bug, that one I would need to look into more closely to see why that
 "no more data" is an issue.
 
   | Aside - I wonder if the notion of "at mark" made sense for those other
   | protocols that had real out of band data (of which I know nothing).
 
 Not really, as best I recall them.   I am fairly certain it would be
 meaningless for X.25, which had OOB data, but there was no way the
 receiver could know (from its transport protocol - or network layer
 protcol it would usually be called, there was just one - that is
 not considering anything that application protcol may have layered
 on top) how much regular data the sender had queued before it sent
 the OOB data.   OOB (just one packet permitted until that was ack'd)
 was sent outside all of the normal sequencing and flow control rules
 (which is why it is called "out of band").
 
 Whether OSI TP had any mechanism to mark where in the regular packet
 stream an OOB packet was sent I have no idea, I never looked at it in
 enough detail to pick up on that kind of nuance.
 
 This also indicates something of an oddity in the development of all
 of this in BSD ... the API was designed to cope with OSI (which at the
 time was a plausible mass use protocol - or its advocates thought so
 anyway) but the only procotol implemented (at the time) which could be
 used to test any of this was TCP, so while the API looks like it was
 designed for OOB data, it was really only ever (initially) used with
 urgent data.
 
   | It's also not entirely obvious to me what given POLLIN | POLLRDBAND
   | the right action is to read/drain the normal (in band) stream first.
 
 It is the nature of TCP.   Despite what the BSD API seems to promise,
 the urgent data cannot be guaranteed to be delivered until all previous
 data has been read.   What's more, and also despite the apparent API,
 an application could do
 
 	send("a"); send("b"); send("c", MSG_OOB);
 
 where the whole combination "abc" is intended to be urgent - it doesn't
 send the OOB flag until the end, as until then all the urgent data isn't
 available, it doesn't want the received to read to "b" and assume it
 has all of it, so the urgent pointer should never be there.
 
 Remember that all there is is data, and a "mark" that says, "somewhere
 before here there is data that the application protocol needs to see
 and process before it processes other data that is arriving earlier".
 There is also just one mark - if the application sends more urgent data
 the mark just moves further down the stream - the earlier urgent data
 is still there in the queue, but is no longer marked any way (the sender
 might know where it is, but has no (TCP) way to convey that to the
 recipient, the receiver cannot keep track of all the various urgent
 data pieces, as there's no guarantee anything will be sent to it at all
 in between 2 OOB writes from the sender.  That is, if my example had
 been
 
 	send("a", MSG_OOB); send("b"); send("c", MSG_OOB);
 
 (and no more MSG_OOB) we know the receiver will get the urgent
 pointer after "c", but the one after "a" might never have been
 sent to it - the sending TCP sees both, but after the "c" OOB
 has been queued, has no way to convey the earlier "a" mark to the
 receiver - there is just one urgent pointer and it is now after "c".
 
   | This is likely highly application and protocol dependent,
 
 Absolutely.   The application has to provide some design for
 what data should be sent as urgent, and how the receiver is to
 recognise that.   Further to do that, the application has to know
 exactly what its transport prococol provices.   For TCP transport
 telnet has all this done properly.  rlogin is simply a mess.
 
 There is simply no way to write a transport protocol agnostic
 application which uses any kind of out of band or urgent data.
 
   | but it would
   | seem natural to want to read real out of band data out of band,
 
 Yes, it would - but TCP has no such concept.  There is no out of band
 data.  If you don't understand that, you cannot sanely attempt to use
 MSG_OOB with a TCP transport.
 
   | i.e. calling recv(MSG_OOB) as soon as POLLRDBAND is reported.
 
 You can do that - and various hacks that are in the sockets TCP
 stack sometimes pretend to allow it to work.   But it is all nonsense,
 nothing in TCP makes any of the attempts to do that meaningful in any
 way at all, which is why not using SO_OOBINLINE with a TCP transport
 is simply crazy.
 
   | So I'm
   | somewhat worried that we are imposing the semantic of TCP not-really
   | OOB urgent data - which happens to be the only user of this interface
   | - on the way that interface behaves.
 
 It all depends upon what the transport procotol implements.   Since we
 currently have nothing but TCP (and SCTP, just in case that is different,
 though I doubt it - but as I recall it does keep record markers rather than
 just providing a byte stream, so it might be) it is all very hard to tell
 what might happen with a transport protocol that actually has OOB data
 (I never once used either X.25 or OSI procotocls on a BSD system - I had
 my own X.25 once, but it was all imolemented in a plug in board, the system
 just saw something not all that much different to a modem - can make calls,
 transport bytes, and hang up... which is just what applications like uucp
 wanted to be able to do.)
 
 But I would not be surprised if you are somewhat correct.   TCP is all that
 existed to be used for testing when all this was developed (the others came
 later) and is all that exists again now (the others have vanished again).
 Until we get another implemented transport protocol which supports genuine
 OOB data, we will probably never really know.   When do you expect one of
 those?
 
 kre