Re: Cert validation in pkg_add

[Taylor R Campbell <riastradh%NetBSD.org@localhost>]

> Date: Thu, 21 Dec 2023 10:34:05 -0500
> From: Greg Troxel <gdt%lexort.com@localhost>
> 
> I also see it as core UNIX and NetBSD doctrine to let people do what
> they want, even if it's something someone else thinks they shouldn't do.
> Besides basic respect for users, I am mindful that people use software
> in ways that we don't understand, and I don't think it's ok to say "I
> don't understand your situation but if you are doing X you are wrong".
> Your patch isn't doing this -- it has an easy disable, but I think this
> needs to be kept in mind.

I'm a little confused by this.  Can you specify exactly which aspect
of my patch you disagree with?

1. configuring https:// (in PKG_PATH or cmdline) flips on validation
2. when validation is on, pkg_add refuses to let https server
   downgrade security by redirect to non-https (http, ftp)
3. no change if you set INSECURE_TRANSPORT=1
4. no change if you configure http:// or ftp://
   => This implies: when http:// is configured, even if http server
      attempts security upgrade by redirecting to https, pkg_add
      doesn't bother validation (maybe revisit later)
5. no change if not NetBSD>=10.0 (revisit post-branch)


Below I'm quoting excerpts to explain where I think we agree, and why
I'm confused, but you can skip everything below if you prefer; I'm
only adding it in case it helps with understanding the finer points.

> So that leads me to be ok with pkg_add doing https validation by default
> and a setting to disable it.

We seem to be in agreement about this, so far.

In particular, I think we agree that if you configure an https URL
(meaning, you set it in PKG_PATH or pass it as a command-line
argument), and it leads to a (possibly empty) chain of https redirects
before serving a file, pkg_add should require validation of all https
servers in this case.  OK?

> Taylor R Campbell <riastradh%NetBSD.org@localhost> writes:
> 
> > On the other hand:
> >
> > - If user-specified http is redirected to server-specified https, the
> >   adversary has already had the opportunity to intercept http by the
> >   time pkg_add could begin to act on the redirect.
> >
> >   So pkg_add can't guarantee authentication if you use http URLs,
> >   whether or not it validates certs on redirect to https.  But
> >   validating certs in this case might break existing setups.
> 
> pkg_add does not have a guarantee of authentication when fetching
> packages and we should not pretend that it does.  If someone configures
> http or ftp, that's what they have done.  That's just like choosing to
> use NFS or SMB.  There's no reason anyone who does that should expect
> any cryptographic integrity/authentication, and we shouldn't invent a
> requirement for it.

It sounds like we are in agreement about this: if you configure an
http URL, you don't (can't) expect authentication, so if the http
server redirects to https, pkg_add wouldn't violate your expectations
if it _didn't_ validate https in that case.

In other words, it sounds like what you're saying here (and what I
think I agree with) is that we could go either way with validation in
the `security upgrade' case of http redirecting to https:

- It is _weakly better for security_ if pkg_add validates https in
  this case even though a MITM could have forged the original http
  reply -- it reduces the attack surface slightly but not really that
  much.

- It is _much riskier for compatible functionality_, however, if
  pkg_add can stop working in this case because of missing certs even
  though the user didn't ask for https in the first place.

So that's why I'm OK on security grounds with doing https validation
on redirect from http in a security upgrade, but why I'm suggesting it
is safer to avoid breakage by _not_ doing this.

If you want to risk _more_ breakage, I can amend the patch to flip on
validation in this case anyway.  I suspect it may be a little more
complicated -- and it definitely has a higher risk of breakage, even
if no bugs creep in with the extra complexity -- but if you want that
then I can do it.

> It is fair to say that if someone configures a https URL then fetches
> *to that URL* should have pkix validation.

Yes, that's the main thing that I am trying to implement and what I'm
calling the security contract.

> > - pkg_add automates the installation (and, via INSTALL scripts,
> >   execution) of software from the remote host the moment you run it.
> >
> >   So if user-specified https is redirected to server-specified http,
> >   there's no opportunity like an interactive web browser provides for
> >   a user to check for `https' or a lock icon in the address bar after
> >   a page is done loading, but before entering their social security
> >   bank account password's maiden name.
> >
> >   Thus, if pkg_add were to quietly honor https to http redirect, it
> >   would quietly render users more vulnerable than an interactive web
> >   browser would.
> 
> I think this is a stretch.  When you specify an https URL, you are
> saying
> 
>    This is the location I want to get bits from.
> 
>    Implicitly, PKIX validation should happen on fetches, so the bits are
>    really from that server (broadly construed, server might have CDN).
> 
> I view any https server giving a redirect to anything non-https as a
> bug.  I am not aware of any reports of places serving packages over
> https that return redirects to http.  If there are we should report that
> to them and ask them to fix.

It sounds like we are in agreement that:

(a) https redirecting to non-https (`security downgrade') is bad
(b) it is important to report that case as a bug to fix

What I'm unclear on is: How would we _discover_ this to report it if
pkg_add silently allows security downgrade of https redirecting to
non-https?

Users don't have the opportunity when they run `pkg_add https://...'
to see that the package was loaded from http:// instead before they
can make a decision, the way they do in an interactive web browser.

> > So I think it is reasonable for the contract to be: If you specify
> > https://... URLs, then pkg_add guarantees it only installs packages
> > retrieved over authenticated transport.  (Otherwise, no change.)
> 
> This means an http: URL that redirects to https will not get validation,
> too.

That's true with the `Otherwise, no change' part.  We can strike that,
and add _more_ changes to potentially break _more_ scenarios, if you
want.

What's puzzling me, though, is that you seem to be concerned with the
riskiness of the change, and yet you seem to be asking for a riskier
version of the change.  That's fine by me on security grounds -- but I
was specifically trying to minimize breakage by making the changes
narrowly scoped to what the user explicitly asks for as follows:

1. Flip on security contract (require validation, refuse downgrade) if
   you explicitly configure `https://'.  (Plus: INSECURE_TRANSPORT=1
   as override if you really need.)

2. No change whatsoever if you don't configure `https://'.

> I see this as ending up way too complicated.  Instead, simply
> 
>   Validate certifications for all https connections if
>   HTTPS_VALIDATE_CERTS=yes.
> 
>   Omit validation if HTTPS_VALIDATE_CERTS=no.

Just to clarify, does your proposal for HTTPS_VALIDATE_CERTS=yes
entail either of the following?

1. If http redirects to https (security upgrade), validate certs
   anyway -- even though the client is already vulnerable to MITM.

2. If https redirects to http (security downgrade), allow that anyway
   -- even though it means the client would become _newly_ vulnerable
   to MITM despite earlier validation of user-configured https.

If not, which case do you want to handle differently and how?

>   Set the default value to yes for NetBSD 10.
> 
>   Set the default value to yes for any system where it is known that as
>   part of proper base system setup that trust anchors are configured.
> 
>   Probably, just set it to yes everywhere, because it is a bug to fetch
>   https without validatin, unless the user has asked not to validated.
>   <pmc>This is probably a bit much for 2023Q4 pullup.</pmc>

For now I'm not proposing to do anything outside NetBSD 10, so we can
address that post-branch.

>   Make no changes at all to ftp/http.  There is no bug to be fixed.

I'm unclear on whether you mean `no changes at all when user
configures ftp/http' or `no changes at all when pkg_add is directed,
whether by local configuration or by remote redirect, to ftp/http'.

> Implicit in the above is that I still object to INSECURE_TRANSPORT as
> being about "you should not do this" vs "let's fix the bug of not
> validating".

What I'm trying to get at with INSECURE_TRANSPORT is: override the
security contract (for the transport layer) and just do what I and the
alleged remote server say.  I think that is clearer than leaving one
to wonder with HTTPS_VALIDATE_CERTS=1/0 what happens securitywise and
compatibilitywise in downgrade or upgrade cases.