bin/55979: sh single quotes removes nul characters

[jtunney%gmail.com@localhost]

>Number:         55979
>Category:       bin
>Synopsis:       sh single quotes removes nul characters
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    bin-bug-people
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Sat Feb 06 12:10:00 +0000 2021
>Originator:     Justine Tunney
>Release:        9.1
>Organization:
>Environment:
NetBSD netbsd 9.1 NetBSD 9.1 (GENERIC) #0: Sun Oct 18 19:24:30 UTC 2020  mkrepro%mkrepro.NetBSD.org@localhost:/usr/src/sys/arch/amd64/compile/GENERIC amd64
>Description:
/bin/sh and /bin/ksh remove ASCII NUL characters embedded in single quoted strings. This is inconsistent with the behavior of shells on other platforms. POSIX requires this content be preserved:

    2.2.2 Single-Quotes
    Enclosing characters in single-quotes ( '' ) shall preserve the literal
    value of each character within the single-quotes. A single-quote cannot
    occur within single-quotes.

I need it because I'm the author of Cosmopolitan Libc which uses a polyglot executable format where binary content is concatenated to a shell script. I just added support for NetBSD. Right now it doesn't work with /bin/sh so I have to tell users to install bash. See https://github.com/jart/cosmopolitan

This use case is supported by POSIX.

    "The input file may be of any type, but the initial portion of the
     file intended to be parsed according to the shell grammar (XREF to
     XSH 2.10.2 Shell Grammar Rules) shall consist of characters and
     shall not contain the NUL character. The shell shall not enforce
     any line length limits."

    "Earlier versions of this standard required that input files to the
     shell be text files except that line lengths were unlimited.
     However, that was overly restrictive in relation to the fact that
     shells can parse a script without a trailing newline, and in
     relation to a common practice of concatenating a shell script
     ending with an 'exit' or 'exec $command' with a binary data payload
     to form a single-file self-extracting archive."

    http://austingroupbugs.net/view.php?id=1250
    http://austingroupbugs.net/view.php?id=1226#c4394

FreeBSD /bin/sh was recently updated to incorporate this change:

    https://github.com/freebsd/freebsd-src/commit/9a1cd363318b7e9e70ef6af27d1675b371c16b1a

Could NetBSD update its /bin/sh shell? 

Here's an explanation of the format and binaries for testing purposes. They do in fact support NetBSD.

    https://justine.lol/ape.html  <-- design doc
    https://justine.lol/hello.com <-- binary file
>How-To-Repeat:
printf "x='\1\0\1'\nprintf '%%s'"' "$x"\n' | /bin/sh | hexdump -C
00000000  01 01                                             |..|

printf "x='\1\0\1'\nprintf '%%s'"' "$x"\n' | /bin/ksh | hexdump -C
00000000  01 01                                             |..|

>Fix:
Possibly changing something to do with `sqsyntax` or `readtoken1` in your Almquist Shell fork in bin/sh/parse.c