Re: sh(1) exec redirection

[Robert Elz <kre%munnari.OZ.AU@localhost>]

    Date:        Wed, 6 Nov 2024 08:03:37 -0500 (EST)
    From:        Mouse <mouse%Rodents-Montreal.ORG@localhost>
    Message-ID:  <202411061303.IAA03930%Stone.Rodents-Montreal.ORG@localhost>

  | But breaking existing use cases strikes me as a Very Bad Idea.

What existing use cases?

Passing random fds to unrelated (as in not part of a set of
related commands all built together) child processes is a very
very rare thing to happen.

Maybe you remember in the 6th end and before days (before close on
exec had been invented) when many applications (particularly
daemons and others which would run for a long time) started
with a sequence like:

	for (i = 3; i < BIG; i++)
		close(i);

because having random fd's passed in from careless parents who
didn't clean up everything before exec() was a common problem,
(In v6 days BIG was typically 20 (or 16 or something near that) as
that was the most fd's a process was allowed).

This change was made to the NetBSD sh in Jan 2016.   It is being complained
about almost 9 years later.   That speaks to large numbers of use cases,
doesn't it???

And even now, the original use that failed and caused the complaint
wasn't this, but a case that I agree is a bug, and should be fixed.

  | I think that, instead, some way should have been designed to set
  | CLOEXEC _without_ breaking existing use cases.

If there actually were any, this might be an issue.

  | Is this what the fdflags you mentioned is?

It is, and more - it can set or clear any of the mode bits that
can be set or cleared by fnctl() (so O_APPEND, ...) as well as
FD_CLOEXEC.

  | If so, shouldn't the CLOEXEC misfeature be rolled back,
  | since there's a now way to get that effect that's compatible
  | with decades of history?

No, for two reasons, one below in my reply to Edgar's message (I'm
combining them to everyone else has one less junk e-mail to delete,
and because I want this to be my last message on this topic).

The other is that it is far more common for scripts like this to exist

	exec 3<&0 </temporary/stdin
	# many commands that use the alternate stdin
		....
	# end of those commands (which might be in a loop)
	exec <&3 3<&-

than it is to have scripts like

	exec 3</some/file
	cmd -f 3		# tell command to use fd 3

and when deciding which category of commands should need to be
changed to either set, or clear, the close on exec flag, it seems
sane to me to pick the ones which are very rare (not many need updating)
over the ones which are quite common (and created much more often)
especially as most people don't even consider the issue when writing
scripts.

Edgar Fu�<ef%math.uni-bonn.de@localhost> said:
  | I can understand that NetBSD's sh derives (unless set -o posix is given)
  | from SUS in cases where SUS documents brain-dead historical behaviour. 

I think you mean diverges rather than derives (no criticism intended,
your English is many orders of magnitude better than my German, and I
once kind of pretended I could handle that ... once).

  | However, passing fd's opened by exec to commands called later on doesn't
  | appear obviously brain-dead to me. 

Not brain dead, just a very unusual way to operate - so much so that I
consider it reasonable that if you want to operate that way, you need
to do the extra work required to make it happen.  At least we provide a
mechanism to allow that to be done.

  | Wouldn't it be more appropriate to have a close-everything-on-exec
  | flag that /etc/rc can use instead of violating SUS on purpose? 

It isn't just /etc/rc - it is lots of scripts, usually written by users
who have no idea that the fd they are using as a temporary (they believe)
in the shell is getting handed off to commands they never expected to be
able to access it.

And, as is not at all uncommon when my memory is involved, it turns out I
was wrong about this.   What POSIX actually says is (in XCU 2.14, the page
which describes the "exec" special built-in utility):

	If exec is specified with no operands, any redirections associated
	with the exec command shall be made in the current shell execution
	environment. If any file descriptors with numbers greater than 2 are
	opened by those redirections, it is unspecified whether those file
	descriptors remain open when the shell invokes another utility.

So it turns out I should probably actually remove the "posix" test from
this, as conforming with POSIX does not require that fd's remain open to
be passed off to child shells.   We aren't violating anything.
(But I won't, it can remain, even though it is technically incorrect).

What's more, all ksh variants I have to test do the same as our shell 
currently does, and set close-on-exec if the fd > 2.   Other shells
typically don't -- and most have no way to set close on exec, so no way
for a script to avoid this problem, other that sticking

	3>&- 4>&- 5>&- 6>&- 7>&- 8>&- 9>&-

on every command that is run - just in case something somewhere in the
script, now or later, has opened one of those fd's.   (9 is the most that
can portably be used - in shells that allow more fds, like ours, it is
even harder).

Note that as ksh88 (which I don't have to test, but is very likely to
be the same as the ones I do have, in this regard at least) was the
"model" for the specification of the shell, and earlier versions of
the standard tended to be more aligned with how it worked, than how
other shells do, if the standard were to actually (now or in the past)
select something other than "it is unspecified", it is far more likely
that it would actually require that fd's > 2 not be passed on (ie:
close-on-exec .. or explicit close in the shell before an exec),

ksh93 provides much the same mechanism as we do to allow open fd's
to be passed to children -- except they (seem to) require the "3<&3"
to be on the command line of the child process, whereas we allow
	exec 3<whatever 3<&3
to pass fd 3 to all subsequent children (until it is closed).

mksh (and our /bin/ksh) doesn't seem to provide any explicit mechanism to do
this at all, I'd assume that to pass an open fd to a child, you'd need to
explicitly open it as
	cmd ... 3<whatever
which doesn't always work as desired, or open a different fd in the script
	exec 4<whatever
and then
	cmd 3<&4

In any case, this issue in the NetBSD shell (other than the behaviour of
	exec cmd 3</whatever
which I will alter) was settled years ago.   It is not going to change
again now.

kre