fixing libfetch as a first-class object

[Greg Troxel <gdt%lexort.com@localhost>]

We are talking about libfetch as if it is part of pkg_install.  But it
ends up in NetBSD base, and it's in pkgsrc.  So: is it really a private
lib of pkg_install, or is it a legitimate library?

The fetch(3) man page reads like a general purpose library that users
are welcome to use, so I'm going to accept that it's a separate library
usable for other purposes.

The man page explains that there are a lot of environment variables.  It
fails to discuss https and validation at all.

From the patches posted, it seems clear that libfetch

  supports https

  does not validate certs

and this is a bug, we have agreed.  As I explained earlier, my concern
about risk in pullups especially is unintended consequences and
complexity.  I am ok with "we decided to error out if validation failed,
and you can turn on switch X" causing problems because that is what we
intend to change.  People who have such setups are already used to
adding exceptions in firefox and passing --no-check-certificate to wget.
wget made the change in default, with the same rationale; I do not
perceive any "they should not have done so".  curl is similar.

So I think we should:

  Add HTTPS_VALIDATE_CERTS=yes/no to the environment section, which is
  aligned with the other variables already declared.

  Default this to on.  (When pulling up to 2023Q4, limit to netbsd-10,
  but turn it on generally for pkgsrc-current.)

  Explain this in the man page, which is exactly that if this is
  enabled, validation is done, and giving the default.

and if people generally feel strongly about it (there is broad agreement
that https should be validated by default, but i have not seen anybody
else comment about any of the details), also

  change libfetch to fail on a redirect from https to non-https
  (always).  (Really, fail redirect from TLS-like protocols (transport
  authentication/confidentiality with endpoint authentication) to
  non-TLS-like protocols.)

  Explain this in the man page.

  If thought necessary, add HTTPS_ALLOW_HTTP_REDIRECT=yes/no, defaulting
  to no.  (I am for now ok with skipping this step as I expect zero
  actual people to have enough trouble with this to make it worthwhile
  to add the complexity.)

  Explain the new variable in the man page.

And then no changes are needed to pkg_install or pkgin.  Anything else
that does "fetch me this https URL" will get validation as expected.
This fixes the bug in the right place: where the bug is.

I believe that if these changes were in libfetch already, there would be
zero motivation to change anything and we wouldn't be having this
discussion.