validation of https in libfetch: perhaps coming soon to 2023Q4 as a pullup

[Greg Troxel <gdt%lexort.com@localhost>]

Over on tech-pkg we have been discussing (at greater length than perhaps
should have been necessary in part due to me trying to deal with the
branch, this, and non-computer things):

  changing pkgsrc so that
    pkg_add https://some.server/some/path/some-package.tar.gz
  will do certificate validation, meaning expecting a certificate with
  the right name, not expired, signed by a CA configured as a trust
  anchor in the system.  Nothing tricky or odd, just "actually validate"

  applying that change to the 2023Q4 branch
    mostly, limited to NetBSD 10 and up

Taylor has done all the work.

The overall rationale is

   package fetches should validate as a security mechanism to help guard
   against malware

   NetBSD 10 now configures trust anchors in the base system, so this is
   feasible

   This is a bugfix as libfetch probably always should have validated.

   https fetches done by libfetch should be validated regardless of the
   caller.  https is simply a protocol that expects validation.

   It's ok to pull the bugfix up to the branch.

   Maybe we should not impose the bugfix, because it changes behavior,
   on users of other than NetBSD >= 10.

   Because there's an env var to set to opt out of validation, it won't
   really cause anyone any real trouble.

This leaves for a possible/likely future change

   Make libfetch refuse to follow https to http redirects.

So, assuming:

  the change lands in pkgsrc-current soon

  after a bit we'll pull it up to the branch

then we have a choice:

  1) ifdef it so that it only applies (on the branch) to NetBSD >= 10

  2) don't ifdef it, so it applies to all platforms.



Earlier I wanted 1, so that we'd have limited changes.  I have come to
see it as a bugfix with an easy opt out, so that makes me want to just
rip the bandaid off and fix the bug.


So, if you use the stable branch on other than NetBSD 10 and you fetch
packages over https, or anything else with libfetch, do you

  fetch from places with valid certs and so this is fine
    Note that valid  is with respect to the trust anchors you have configured!

  fetch from places without valid certs, but if you set the env var it
  won't validate, you are ok with it?

  fetch from places without valid certs, but setting the env  var in
  your world as a change during the branch is painful, so you are not ok
  with this?


I hope this made sense; please speak up if turning on https validation
for all systems, with an env var to opt out, is going to cause you
trouble.


Greg