Issue 8 POSIX changes coming

[Robert Elz <kre%munnari.OZ.AU@localhost>]

This contains a list of some of the non-trivial (such as formatting fixes,
corrections to the rationale, ...) that are going to be in POSIX 8 (and no,
no idea yet when that will appear).

When I know a change is irrelevant to NetBSD, I will omit mention of it
(that is, where I know we already comply with the new text) - but that I
am including something here does not mean that we do not already comply,
sometimes it does, but often it is just that I'm not sure one way or the
other, so someone who knows can speak up.

This is a wild mixture of unrelated issues - the URLS given relate to the
Austin group bug/defect report where all of these were processed.   If
my paraphrasing of the issue seems "weird" - please check the URL for
the actual change, for some of these I may have over simplified, or simply
misunderstand the change.   Corrections welcome.


On conforming filesystems readdir() must return the same pathname as
was used when the entry was created (no "normalisation" permitted),
except that a trailing "/" which is permitted in the sys calls that
make directories must be removed (no '/' in the directory - obviously,
and no '/' added by readdir()).   EINVAL is to be allowed when creating
files on non-conforming filesystems if the pathname supplied does not
conform to the requirements of the filesystem, and implementations are
suggested to use it, rather that transforming the final component of the
pathname (the new entry to go in the directory) into something which does
conform (on the other hand, when doing lookups, converting non-conforming
names to something conforming is permitted ... that is, if a filesystem
only supports upper case names, an attempt to create one in lower case should
fail (EINVAL) rather than converting to upper case, but lookups using lower
case are just fine.    Since this stuff only affects filesystems that are
non-conforming anyway (any byte sequence is valid for conforming filesystems,
and no transformations are permitted) the standard won't actually require
that EINVAL be returned for non-conforming names when creating files/dirs/...
URL: https://austingroupbugs.net/view.php?id=293 

sem_post() must return EOVERFLOW if the value to be placed in the semaphore
exceeds SAM_VALUE_MAX
URL: https://austingroupbugs.net/view.php?id=315 

::= += and ?= macro assignment operators added to make
URL: https://austingroupbugs.net/view.php?id=330 

-include added to make
https://austingroupbugs.net/view.php?id=333 
(nb: read this one in conjunction with a later change which affects
the same aread).

(this one is, partly anyway) mine sometime, perhaps ... a new form
of (built in) shell command, a "declaration utility" is being added
(and export readonly are made that) which has different parsing rules
for its args.   This one is a mess...
URL: https://austingroupbugs.net/view.php?id=351 

pthread_cond_timedwait() pthread_cond_wait() pthread_mutex_setprioceiling()
pthread_mutex_timedlock() can all return EAGAIN if a system limit on the
number of robust mutexes is exceeded.
URL: https://austingroupbugs.net/view.php?id=354 

Attempts to assign to readonly vars using the sh builtins export, read,
getopts, causes an error return (status > 1)
URL: https://austingroupbugs.net/view.php?id=367 

"Hidden" file descriptors in a whole bunch of library finctions are
required to have FD_CLOEXEC set (this means any fd left open by just
about any library routine, where the fd isn't returned to the user,
things like sethostent() for one examplke).
URL: https://austingroupbugs.net/view.php?id=368 

closing an already closed fd in posix_spawn is not an error.
URL: https://austingroupbugs.net/view.php?id=370 

make macro names can include '-'.
URL: https://austingroupbugs.net/view.php?id=373 

When semget() creates a set of semaphores, they're all init'd to 0
(this change also makes it clear that the S_* macros from <sys/stat.h>
can be used alternately to octal constants for setting permissions, but
that ought to have always been obvious).
URL: https://austingroupbugs.net/view.php?id=377 

free(3) (and several other functions that free data returned by other
library routines) must not alter errno if no error occurs (eg: for free(),
if the pointer passed is NULL, or was previously returned by malloc (et al)
and has not since been deallocated, then free() must not alter errno).
The rationale is to allow cleanup after an error without needing to save
errno around (known valid) free() (etc) calls.
URL: https://austingroupbugs.net/view.php?id=385 

dirfd(3) isn't allowed to return ENOTSUP
URL: https://austingroupbugs.net/view.php?id=391 

A whole bunch of *_r() functions shall fail with ERANGE if the supplied
buffer is not big enough (perviously was "may" fail).  This includes things
like getpwname_r() strerror_r() ...
URL: https://austingroupbugs.net/view.php?id=398 

In mailx treating '+' as a literal in the folders command happens only
when the folder is unset, not when it is null (this change is supposed to
match existing practice).
URL: https://austingroupbugs.net/view.php?id=1306 

A new iflags= keyword "fullblock" is added to dd (and seems unrelated to
any O_xxx options for open) which causes dd to continue reasing until its
input buffer is full (or EOF I assume) rather than returning short reads.
the count= parameter when combined with iflags=fullblock counts full blocks,
not the number of actual read requests as it usually does.   This is intended
for reasing from pipes, and behavious is unspecified if combined with
various other blocking options.
URL: https://austingroupbugs.net/view.php?id=406 

A whole bunch of mem*() str*() and w*() (wide char) functions will be
prohibited from changing errno when successful (which should match existing
practice).  For the list of affected functions see:
URL: https://austingroupbugs.net/view.php?id=448 

Binary mode ("b" in the mode string) is to be required for fmemopen
(but 'e' and 'x' which have been added to fopen() are not - perviously
fmemopen simply accepted the same modes as fopen()).
URL: https://austingroupbugs.net/view.php?id=456 

make's include (and -include) can include multiple files
URL: https://austingroupbugs.net/view.php?id=518 

Support for .PHONY added to make
URL: https://austingroupbugs.net/view.php?id=523 

Pattern matching substitutions in make var expansions added
( $(string1:[op]%[os]=[np][%][ns]) type thins).  NetBSD make
is (was - most of this happened in 2011) one of the examples of
a make that supported this (along with gnu make) - but there are
comments in the notes attatched to the bug about various defects
and incorrect behaviours (in at least one person's opinion).
How our make compares with what is to be required now (9 years
later) I have no idea...
URL:  https://austingroupbugs.net/view.php?id=519 

Changes to close(2) and addition of posix_close() ... the issue relates
to the state of the fd when close() is interrupted by a signal.  The
original proposal was to require that the fd remain open (but that only
another close() call would necessarily work) upon EINTR, and that (aside
from EBADF which is different) in all other cases the fd must be closed
(it was unspecified before after EINTR and EIO).   The linux cmmunity
objected to that, they always close the fd, regardless of any error (other
than EBADF obviously).   The resolution is more aligned with the linux
approach - close() must always close the fd, but it isn't required to do
it before the close() function returns if it was interrupted by a signal.
In that case it can return EINPROGRESS and indicates that the close()
operation is proceeding (the application has no way to tell when it has
completed).   posix_close() has 2 args, the fd, and a flag arg, for which
the only defined values are 0, or OSIX_CLOSE_RESTART (to be added to 
<unistd.h>).  posix_close() is the same as close() unless the
POSIX_CLOSE_RESTART flag is given, in which case, if interrupted by a
signal, posix_close() returns EINTR, and the fd is not closed, but the
only defined operation on the fd after this is another close()
(or posix_close()) (this allows the application to repeat
posix_close(fd, POSIX_CLOSE_RESTART) so it can find out when the close
has finally completed.   Passing any flag other than POSIX_CLOSE_RESTART
to posix_close can result in an EINVAL return, but in that case
posix_close() still acts like close() and closes the fd anyway.
URL: https://austingroupbugs.net/view.php?id=529 

du is to only ever count files once (including links) regardless of how
named on the command line of found in a directory traversal.
URL: https://austingroupbugs.net/view.php?id=539 

In awk "delete arrayname" is to be defined to delete the enture array.
URL: https://austingroupbugs.net/view.php?id=544 

In awk "nextfile" is added.
URL: https://austingroupbugs.net/view.php?id=607 

The symbol PTHREAD_NULL is to be added to pthread.h - it must be a
compile time constant.  It indicates "no thread" - and is needed as
the pthread_t type is opaque (can be a pointer, int, struct, (float?))
so there's no portable way to write the notion of "no thread".
(For us,
	#define PTHREAD_NULL ((pthread_t)NULL)
should be all that's needed).
URL: https://austingroupbugs.net/view.php?id=599 

mbsnrtowcs() is defined to count bytes in its result, not characters.
URL: https://austingroupbugs.net/view.php?id=616 

ffsl() and ffsll() are added - analogus to ffs() but with a long, or long long
arg respectively.
URL: https://austingroupbugs.net/view.php?id=617 

awk gains fflush
URL: https://austingroupbugs.net/view.php?id=634 

mkstemp() required to generate EINVAL (like mkdtemp() was) in case of
an invalid template (nbot ending in XXX...).
URL: https://austingroupbugs.net/view.php?id=652 

What looks to be (to me) a quite technical change to the way semop()
works (that is, was specified to work) - one I am not going to attempt
to understand:  See:
URL: https://austingroupbugs.net/view.php?id=628 

strerror_r is semi-deprecated, Issue 7 (current POSIX) already has a
strerror_l() function we do not seem to support (yet) - it is just
like strerror() except uses thread local storage (different threads cannot
step on each other's error messages) and includes a locale parameter
to say into which locale's language the message should be created.
Notes in the bug report claim that strerror_l() is a superior interface
to strerror_r() (and for some uses it might be - but personally I can't
think of a situation where I'd want to specify the locale to use, rather
to the user's default, yet I can imagine not wanting one call of strerror_?()
to step on the results of a previous one (even a previous one in the
same thread) - so I'd be keeping strerror_r whatever eventually happens to
it in POSIX (I know we will keep it for ABI/API compat, but it could be
removed from the manual etc, to discourage people from using it .. which
I would not do) and since it is in POSIX already, we probably should have
strerror_l() but I'd also do a strerror_rl() version.
URL: https://austingroupbugs.net/view.php?id=655 


Lastly for now (there is lots more of this to come, which is why no-one
yet has any idea when Issue 8 will be ready) one that is not yet quite as
far along as those above, that is, has not yet been edited into the text
for Issue 8, but which is a big one, and seems certain to eventually make
it in, is the addition of "close on fork" as an analog to "close on exec".
The only rational reason for this (but given it, the need is hard to fault)
is to deal with threads that fork just after another thread has opened a
new file descriptor.  Such a thread (currently) inherits that open fd, as
fd's are per process, not per thread, but has no idea it has done so, and
no reasonable way to find out (since after a fork() only the forking thread
remains alive in the child).   This involves extending fcntl(FD_GETFLAG)
and fcntl(FD_SETFLAG) to return/take a new flag (which will mean auditing
a bunch of code, as it has been common for fcntl(FD_GETFLAG) != 0 to imply
close on exec is set - since it was the only defined flag, apps didn't
always test for it by name ... which is kind of justified, as originally
the flag had no name.   There is also O_CLOFORK for use in open() pipe2()
(etc).   And an "f" mode flag for popen() (at least in the "wf" form)
(which would probably be useful in fopen() as well, but that is not to
be mandated, as apps can do fd = open("....", O_RDONLY|O_CLOFORK); and
then fdopen(fd, "r") to achieve the effect of fdopen("...", "rf") but
for popen() that's not possible.   Because implementing "rf" mode
correctly for popen is nearly impossible, that isn't to be required.
The issue with popen in "w" mode is that if another thread forks just
after the pipe has been created, it will have an (unknown) write side
pipe fd, so the reading child (from popen) would never see EOF, it will
simply hang forever waiting for that other fd to write something...
For more info on all the changes involved in this see
URL: https://austingroupbugs.net/view.php?id=1318 


I will send another message like this when there are enough new changes
to warrant it (unless general opinion is that we don't care).   We appear
to not have a tech-standards or similar list, hence I'm just using
current-users as that's who will be affected by all of this first, I
assume,

kre