bootstrap: bootstrap_sh vs sh_prog, libarchive shell construct usage

[Greg Troxel <gdt%lexort.com@localhost>]

Vivianne reported a problem bootstrapping on old Solaris where /bin/sh
could not handle a construct in libarchive's configure.

  ./configure: syntax error at line 3494: `ARCHIVE_MINOR=$' unexpected

which is due to

  ARCHIVE_MINOR=$(( (LIBARCHIVE_VERSION_N() / 1000) % 1000 ))

1) What do we intend to require of shells for configuring libarchive?
This construct appears to be straightforward valid POSIX sh:

  https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_06_04

but in the GNU world, generally the plan is for configure scripts to be
more ancient-sh compatible.

2) Vivianne sent a patch to set shprog also on Solaris, when
/usr/bin/bash was found.  (kre@ pointed out that the file name should
appear once and be copied from the previous variable, which is fine but
not the question I am raising.)

  From: Vivianne Ombretta <vivianne%chinstrap.org@localhost>
  Subject: bootstrap: Set shprog to bash on SunOS if available
  To: Greg Troxel <gdt%lexort.com@localhost>
  Cc: pkgsrc-users%netbsd.org@localhost
  Date: Sat, 1 Jul 2023 13:10:34 -0400 (17 hours, 24 minutes, 37 seconds ago)

  The system's sh is broken and results in a failed configure of
  libarchive. This should not affect other systems as only the
  SunOS case is modified.

  diff --git a/bootstrap/bootstrap b/bootstrap/bootstrap
  index e84e4dc2a..1ae955837 100755
  --- a/bootstrap/bootstrap
  +++ b/bootstrap/bootstrap
  @@ -852,6 +852,7 @@ SunOS)
          if [ -x "/usr/bin/bash" ]; then
                  bootstrap_sh=${SH:-/usr/bin/bash}
                  bootstrap_sh_set=set
  +               shprog="/usr/bin/bash"
          else
                  need_mksh=yes
          fi


I went to read bootstrap, and came away unclear on why we have
bootstrap_sh and shprog both and the circumstances under which one or
the other is set.  Excerpts:

  # If we determine that a native version is good enough to use then it is saved
  # and we later write the appropriate TOOLS_PLATFORM variable to mk.conf to
  # override any further processing from mk/tools.

  bootstrap_sh=${SH-/bin/sh}
  bootstrap_sh_set=${SH+set}

  OSF1)
          shprog=/bin/ksh

  SunOS)
          if [ -x "/usr/bin/bash" ]; then
                  bootstrap_sh=${SH:-/usr/bin/bash}
                  bootstrap_sh_set=set
          else
                  need_mksh=yes
          fi

  check_prog shprog sh

  if [ -n "${bootstrap_sh_set}" ]; then
          bmakexargs="${bmakexargs} --with-defshell=${bootstrap_sh}"
  fi

  install_sh="$shprog $wrkdir/bin/install-sh"

  # bootstrap mksh if necessary
  case "$need_mksh" in
  yes)    echo_msg "Bootstrapping mksh"
          copy_src $pkgsrcdir/shells/mksh/files mksh
          run_cmd "(cd $wrkdir/mksh && env $BSTRAP_ENV CC=\"${CC}\" $shprog Build.sh -r)"
          run_cmd "$install_sh -c -o $user -g $group -m 755 $wrkdir/mksh/mksh $wrkdir/bin/mksh"
          echo "TOOLS_PLATFORM.sh?=               $prefix/bin/mksh" >> ${TARGET_MKCONF}
          echo "TOOLS_PLATFORM.sh?=               $wrkdir/bin/mksh" >> ${BOOTSTRAP_MKCONF}
          echo_msg "Rebootstrapping bmake for mksh"
          bmakexargs="$bmakexargs --with-defshell=$wrkdir/bin/mksh"
          bootstrap_bmake
          ;;
  esac

  if [ "$bootstrap_sh_set" = "set" ]; then
          echo "TOOLS_PLATFORM.sh?=               ${bootstrap_sh}" >> ${TARGET_MKCONF}
          echo "TOOLS_PLATFORM.sh?=               ${bootstrap_sh}" >> ${BOOTSTRAP_MKCONF}
  fi


  echo_msg "Bootstrapping pkgtools"
  copy_src $pkgsrcdir/archivers/libarchive/files libarchive
  run_cmd "(cd $wrkdir/libarchive; env $BSTRAP_ENV CONFIG_SHELL=$shprog \
  $shprog ./configure $configure_quiet_flags --enable-static --disable-shared \


So it looks like:

  bootstrap_sh tends to be sh, but can be set to a better shell in
  platform-dependent code.   It can also be set to a bootstrapped mksh.
  Setting one or the other of these is not unusual across platforms.

  bootstrap_sh is used to set TOOLS_PLATFORM.sh.  This is obviously (but
  not documented) because the native shell will not be good enough to be
  used as sh for *pkgsrc in general*.

  if need_mksh is set, then TOOLS_PLATFORM.sh is set to it, as if
  bootstrap_sh was set to $prefix/bin/mksh (but also mksh is
  bootstrapped).

  On OSF1, we set need_mksh and shprog, declaring that the default shell
  is not good enough for pkgsrc and not good enough to bootstrap.

and that all makes sense.

Then, there is shprog, which is set to some program 'sh' in PATH.  That
is then used to run shell scripts as part of the bootstrap.  Questions:

  a) Do we have a belief, or a policy about configure files, that any
     shell we find is good enough?  And that OSF1 is an exception?

  b) Should we instead be defaulting shprog to bootstrap_sh, so that if
     we intend to use a better shell for pkgsrc, we just use it for
     configure/install/etc. during bootstrap?

  c) We respect SH to set bootstrap_sh, but why do we not allow it for
     shprog?

Given all that, any objections to the following?

The only downside I can see is that there are probably SunOS systems
where /bin/sh is good enough for bootstrapping, but given that we never
accept /bin/sh if /usr/bin/bash exists, using the found bash for
configure seems ok anyway.

I also wonder about defaulting shprog to bootstrap_sh if set, but don't
want to bite off the responsibilty to fix fallout, and we have OSF1
precedent.


--- bootstrap.~1.319.~	2023-06-29 06:13:09.567701715 -0400
+++ bootstrap	2023-07-02 07:14:32.157996634 -0400
@@ -393,7 +393,9 @@ checkarg_sane_relative_path() {
 #
 # If we determine that a native version is good enough to use then it is saved
 # and we later write the appropriate TOOLS_PLATFORM variable to mk.conf to
-# override any further processing from mk/tools.
+# override any further processing from mk/tools.  Note that while these are
+# "bootstrap_foo" they are not used during bootstrap, but are stored in
+# TOOLS_PLATFORM.foo in the generated mk.conf.
 #
 bootstrap_awk=
 bootstrap_sed=
@@ -852,6 +854,7 @@ SunOS)
 	if [ -x "/usr/bin/bash" ]; then
 		bootstrap_sh=${SH:-/usr/bin/bash}
 		bootstrap_sh_set=set
+		shprog=$bootstrap_sh
 	else
 		need_mksh=yes
 	fi
@@ -981,6 +984,7 @@ check_prog lnprog ln
 check_prog lsprog ls
 check_prog rmdirprog rmdir
 check_prog sedprog sed
+# shprog is used to run configure and other shell scripts during bootstrapping.
 check_prog shprog sh
 check_prog whoamiprog whoami