Re: bin/55979 (sh single quotes removes nul characters)

[Robert Elz <kre%munnari.OZ.AU@localhost>]

    Date:        Wed, 10 Feb 2021 10:39:13 -0800
    From:        Justine Tunney <jtunney%gmail.com@localhost>
    Message-ID:  <CANtdasRbJQCznmCStFQFsYuOrb8BjQdoQiv8XLfAxOEYgD2RAw%mail.gmail.com@localhost>

  | Try downloading https://justine.lol/hello.com again.
  | That file can be your test case.

I did that, and used it, though:

  | I can create a more minimal one too if you need it.

I did that for myself as well, once I worked out just what was
going on, and what was really needed to perform the test (small
tests that test nothing more than the bug are ideal ... of course
to make one of those you need to first know exactly what the
bug really is).

Turns out that back in 1995 a fix for another problem broke the
way that \0's were being dropped, and caused corrupted input (more
corrupted than just dropping the nul chars) to be handed to the
shell parser.

That we have had this bug for more than 25 years says something
about just how rare it is for shell scripts to actually contain
nul characters.   Of course, any that do are non-conforming, so
we could simply claim our current behaviour as correct (though
it clearly is an obvious bug ... now it has been found).

I am currently building an updated system (HEAD) with the fix
(and the MSAN detected bug fix as well, not that that one really
matters), after which (tomorrow, or later today depending upon
timezones) I'll run the ATF tests to check that nothing new appears
to have broken (as this fix should change nothing for a script without
nul characters in it, and the MSAN fix change is truly trivial,
that should not be a problem).

Expect to see the fix(es) in HEAD early next week.

After it has caused no problems there for a while, I'll request
a pullup to -9.   But as neither of these bugs bother almost
anyone, ever, and because its remaining lifetime isn't expected
to be all that long, I don't think I'll bother with pullups to -8.
(Of course, any other developer who wants to do the work to
incorporate the fixes there, and test them properly, could do that.)

If you want to test the updated sh on a -9 system, just copy the
sources from HEAD (after the fix appears) and build it on a -9
system (cd src/bin/sh; <update sources to HEAD>; make) (or as part
of a build of the -9 system on any host).  There should be no problems
doing that (but going to -8, while not all that hard, is slightly more
complex than just copy and build).   Note that building sh requires
access to the sources of its built-in commands (test, printf, kill, ...)
so the rest of the (userland at least) sources should be present.

kre

ps: no changes are needed to the way we detect binary files so as not to
interpret them as scripts - that turned out to all be misdirection, the
actual problem is closer to what was originally reported, though single
quoted strings have nothing specific to do with it, and nuls certainly
do not survive sh processing.