Re: fflush(NULL) and mem streams (fmemopen(), open_*memstream())

[Robert Elz <kre%munnari.OZ.AU@localhost>]

    Date:        Sat, 12 Jan 2019 17:42:18 +0100
    From:        Rhialto <rhialto%falu.nl@localhost>
    Message-ID:  <20190112164218.GH2330%falu.nl@localhost>

  | When I saw this, my first reaction was "What does it even *mean* to
  | flush a memory stream!? They don't have a buffer, they *are* the
  | buffer!"

That is an implementation choice, and is not required in general.
There is no reason a memory stream cannot also have a stdio buffer,
just like one created with fopen(), in which case the fflush() will
copy from the stream buffer into the memory buffer.

The wording is kind of precise to allow that to happen if desired, and
it might be useful if accessing the destination buffer is going to require
some kind of  memory barrier operation on each access, in order
to make sure that the data in it is consistent amongst threads running
on different processors.

  | The buffer is generally used to prevent doing too many expensive
  | operations, such as system calls.

Yes, or memory barriers, cache flushes, ... plus whatever mutex
operations are required.

  | Reflecting a bit more on the POSIX
  | description of open_memstream(), I came to the conclusion that the
  | expensive operation here may be the updating of *bufp and *sizep (which
  | may include reallocing the buffer), and the adding of a NUL (not null)
  | character just past the end. These are described to be only valid after
  | a call to fflush() or fclose().

Yes, they are - but also becaise there is no guarantee that the data
will actually appear in the (user supplied) buffer until a fflush() happens.

  | So it is unclear to me if you still need
  | to call fflush() on a memory stream to have *bufp and *sizep updated.

I believe that you do (or that is the intent.)   I suspect that the normal 
model for memory streams, is to write as much as you need for one
transaction (one or more calls to the output functions), and then call fflush()
to make sure that the output is available in the buffer correctly (with its
terminating \0 which might not have been added earlier) and that those
interface vars are updated.

I think it is more the observation that there is always (in practice) that
fflush(stream) when data needs to be made visible, that means that
doing an extra fflush operation during a fflush(NULL) will normally do
nothing to a memory stream, as the data has already been flushed
if it is stpposed to be available to the recipient.

Worse perhaps, if a thread is in the process of building a message for
another thread, and has done fomr fputc() or fprintf() or whatever, into
the memory stream, when it detects an error, and decides it must exit,
flushing the mem stream, so the partially completed data is available to
the other thread (which is presumably still running) might be exactly the
wrong thing to do.

  | There still needs to be such a list of non-memory streams anyway, so I
  | don't see the advantage there.

That one I think was just a throw away ... mostlty, but while the list
needs to exist for file backed streams, the operations to access it
in a thread safe manner, which means some kind of mutex) would not
be needed for memory streams if fflush(NULL) is not defined to
include those.

  | But for memory streams, fflush() (or fclose()) is also needed if you
  | want to look at the resulting memory buffer, so fflush() is probably
  | called more often. That makes this a weak argument as well.

Yes, it was, and yes, the fflush() is likely to be used more on mem
streams than processes typically use it on file backed streams, so
while the "avoid unnecessary work at exit time" isn't much an argument
it is still a valid one.

  | I think my reasoning above goes in the direction that fflush(NULL)
  | *should* also affect memory streams.

And while this isn't something I know much about, I think that after
being inspired to think more about it from your response (thanks for
that) I am slightly inclined the other way - perhaps even as much as
going the whole other way, and requiring that memory streams not
be flushed as part of fflush(NULL) (but I  also expect there is zero
chance that posix would do that far).

kre