Re: Anyone interested in implementing O_NOCLOBBER ?

[Robert Elz <kre%munnari.OZ.AU@localhost>]

    Date:        Thu, 16 Apr 2020 19:27:48 +0200
    From:        Joerg Sonnenberger <joerg%bec.de@localhost>
    Message-ID:  <20200416172748.GA86536%bec.de@localhost>

  | What is the point of this "restriction"? They wanted to make a set flag,
  | but allow people to not have to use sub shells when also redirecting
  | stderr?

Sorry, I'm not following the point there, what does any of this have
to do with sub shells or stderr ?

I expect you know all of this already, but ...

Noclobber mode started in csh in the 70's (maybe even in ashell, I no
longer remember which extras that ended up in csh started there).

Its point was entirely so that idiots could type

	cat foo >/my/important/file

and get some protection from losing their important files.   For this no
special mechanisms are needed (this worked fine on a 6th edition kernel).

That is, until the user who had done "set noclobber" tried to do

	cat foo >/dev/null

and it failed, because /dev/null already existed.
Same with "cat foo >/dev/tty".   They could use the syntax to override
noclobber mode (in sh it would be "cat foo >|/dev/null" - I know it
is different in csh, but I forget what it is, it has been many decades
since I used csh, and back then I never used noclobber mode (still don't)).

So the "only regular files" rule was added (still way back in the 70's).
Sym-links didn't exist yet of course.

This was all eventually copied into /bin/sh (there called "set -C").

(At least I believe that's the sequence - it is possible that Bourne
added "set -C" first, Berkeley people knew some of what was happening
in his shell - noclobber might have been copied in the other direction,
as some other csh features were).

Doesn't matter.

In any case, sometime after that, someone decided that noclobber mode
would be a good way of making lock files.   Also temporary files that
need unique names.   Using noclobber mode, if an error is returned,
then the lock failed, or a different name is needed.   But now we're
right into the area where race conditions cause things to break, two
scripts trying to make the same lock at the same time both test for the
file existing (it doesn't) so both create it (both succeed) - bad mojo.

Since this got reported as a bug (ignoring the fact that attempting to
use noclobber mode for these purposes in the first place was simply stupid)
shell authors started using O_EXCL (which had been invented by this time)
to narrow the race.   Only narrow, as the /dev/null issue still needs to
be handled - so if O_EXCL says EEXIST, the shell must stat the filename,
and see if it is a special or regular file, and try again without O_EXCL
if it is a special file.   But during that window, the file might be
changing - I do: ln /dev/null /lock/file - your open(/lock/file, O_CREAT|O_EXCL)
fails, your stat() says "char special file", and at this point I unlink
/lock/file and instead ln /your/important/file /lock/file just after which
you redo the open, without O_EXCL (probably also without O_CREAT as we know
the file exists - but that's irrelevant), and now you're about to
write all over your important file.   More bad mojo.

O_NOCLOBBER is intended as a solution to that - in that it is just like
O_EXCL, but doesn't error on special files, only regular ones, so this
race condition no longer exists (because there is just one (atomic) open
sys call which either works, or fails).

Simply avoiding using noclobber mode for these purposes would be a better
idea, but it seems now that "everyone knows" that noclobber mode can be
used (many even believe it is the "one true way"), and complain if it fails.

The symlink difference between O_EXCL and O_NOCLOBBER is due to a difference
in purpose - O_EXCL is an assertion (or requirement) "this name must not
already exist, if it does, error".   Given that, anything owning the name
in question causes an error (including a dangling symlink, or a device file).

On the other hand, O_NOCLOBBER is designed to avoid destroying important
files, if one does (not using noclobber)

	ln -s /no/such/file my-link
then
	cat file >my--link

then (all else being OK) /no/such/file now exists, and contains a copy of
file.   Since the file didn't previously exist, and nothing has harmed
(or changed at all) the symlink, no clobber mode (with the inent of avoiding
damage to existing files) shouldn't object.    Consequently, while O_EXCL
would error when the name given exists, O_NOCLOBBER doesn't when it is a
symlink pointing to a file that doesn't exist.

At least, that's the idea behind the 2 differences between O_EXCL and
O_NOCLOBBER.   Aside from those, the two are (supposed to be, since this
is all invention) identical.

In any case, nothing here is related to stderr (while noclobber mode applies
to all output redirections, it is far more commonly useful for stdout)
and certainly nothing related to sub shells.   For O_NOCLOBBER the
expectation is that other commands intending to make new output flies
(not destroying something that already existed) would use it as well.
Editors that rename the original file and write a new copy are an example.

kre