Re: Cert validation in pkg_add

[Greg Troxel <gdt%lexort.com@localhost>]

Taylor R Campbell <riastradh%NetBSD.org@localhost> writes:

>> Date: Thu, 21 Dec 2023 10:34:05 -0500
>> From: Greg Troxel <gdt%lexort.com@localhost>
>> 
>> I also see it as core UNIX and NetBSD doctrine to let people do what
>> they want, even if it's something someone else thinks they shouldn't do.
>> Besides basic respect for users, I am mindful that people use software
>> in ways that we don't understand, and I don't think it's ok to say "I
>> don't understand your situation but if you are doing X you are wrong".
>> Your patch isn't doing this -- it has an easy disable, but I think this
>> needs to be kept in mind.
>
> I'm a little confused by this.  Can you specify exactly which aspect
> of my patch you disagree with?
>
> 1. configuring https:// (in PKG_PATH or cmdline) flips on validation

I believe that by default any https fetch should have validation.

> 2. when validation is on, pkg_add refuses to let https server
>    downgrade security by redirect to non-https (http, ftp)

Nope.  That's not supported in any standard that I have found.  If the
server does that, it's a bug in the server, but that's no different than
serving malware.  The user has chosen to trust the server to provide
bits, and there is no reason to second guess that.

> 3. no change if you set INSECURE_TRANSPORT=1

I disagree with the name INSECURE_TRANSPORT, and that it affects
http/ftp, but yes, as phrased it is not really objectionable.

> 4. no change if you configure http:// or ftp://
>    => This implies: when http:// is configured, even if http server
>       attempts security upgrade by redirecting to https, pkg_add
>       doesn't bother validation (maybe revisit later)

I don't think this makes sense.  We are having this entire discussion
because of a belief that the standards say, or community norms say, that
https (TLS really) should do pkix validation on the client side of
server certs.  If someone configures an http URL, that should be used to
fetch, and if the server sends a redirect to https, then that fetch
should use the configured validation settings (defaulting to validating,
unless HTTPS_VALIDATE_CERTS=no, eventually).

> 5. no change if not NetBSD>=10.0 (revisit post-branch)

Yes, that's fine.  However this sets precedent, so everything has to be
ok for all systems and all circumstances, even if not enabled due to risk.

> Below I'm quoting excerpts to explain where I think we agree, and why
> I'm confused, but you can skip everything below if you prefer; I'm
> only adding it in case it helps with understanding the finer points.
>
>> So that leads me to be ok with pkg_add doing https validation by default
>> and a setting to disable it.
>
> We seem to be in agreement about this, so far.
>
> In particular, I think we agree that if you configure an https URL
> (meaning, you set it in PKG_PATH or pass it as a command-line
> argument), and it leads to a (possibly empty) chain of https redirects
> before serving a file, pkg_add should require validation of all https
> servers in this case.  OK?

I think that (unless validation is disabled) every https fetch should be
validated.  Which is a very simple rule with no exceptions and no
bookkeeping, and yes it leads to the same conclusion in the case you
cite.

> It sounds like we are in agreement about this: if you configure an
> http URL, you don't (can't) expect authentication, so if the http
> server redirects to https, pkg_add wouldn't violate your expectations
> if it _didn't_ validate https in that case.

No, we don't agree at all.  That is a violation.  The standards say
https is supposed to be validated*.  Just because I configured http and
the server sent a redirect doesn't mean I waive following the standard.
I see this as 100% about straightforwardly following standards and not
at all about trying to bandaid around TNF not signing packages.

* Or we are assuming that the combination of standards and community
  consensus says this.

It is not about what I expect of overall security properties.  It is far
far simpler, that I expect http fetches to just happen, and https
fetches to validate.  I expect well-configured servers that operate over
https to never issue a redirect to non-https, but I don't expect a "wget
like" program to object to this.

I also don't see how not validating https is useful when it is a
redirect from http.  If people do odd things and the straightforward
approach from the standards doesn't do what they might find useful, oh
well.

> In other words, it sounds like what you're saying here (and what I
> think I agree with) is that we could go either way with validation in
> the `security upgrade' case of http redirecting to https:
>
> - It is _weakly better for security_ if pkg_add validates https in
>   this case even though a MITM could have forged the original http
>   reply -- it reduces the attack surface slightly but not really that
>   much.

I do not think it is proper to reason about how to behave in terms of
better for security.  There are existing standards and norms and they
are straightforward.  http just happens and https should get pkix
validation with system-configured trust anchors.

> - It is _much riskier for compatible functionality_, however, if
>   pkg_add can stop working in this case because of missing certs even
>   though the user didn't ask for https in the first place.

If the server sends an https redirect, and the server doesn't present a
valid certificate, and the user hasn't asked to disable https
validation, then failing with a visible "failed to validate certificate
from URL" is perfectly ok.n

> So that's why I'm OK on security grounds with doing https validation
> on redirect from http in a security upgrade, but why I'm suggesting it
> is safer to avoid breakage by _not_ doing this.

Either "validating certs for https is normal and should just happen" or
that is not true.   I think that validation for https unless configured
off meets reasonable expectations based on standards, is extremely
straightforward, and is not going to be confusing to deal with.  All
these extra rules 

> If you want to risk _more_ breakage, I can amend the patch to flip on
> validation in this case anyway.  I suspect it may be a little more
> complicated -- and it definitely has a higher risk of breakage, even
> if no bugs creep in with the extra complexity -- but if you want that
> then I can do it.

No, I want you to completely remove all code to disable ftp/http under
any circumstances and have a single setting to validate or not validate
https.  This is very simple, with a single variable, configurable by the
user, to have a yes/no.

>> It is fair to say that if someone configures a https URL then fetches
>> *to that URL* should have pkix validation.
>
> Yes, that's the main thing that I am trying to implement and what I'm
> calling the security contract.

But I don't like "security contract".  I think it's important to keep in
mind that the architecture for package security is signed packages.  We
are both fixing a bug and also pushing wet noodles to work around it,
and I think we should only fix the bug.

But I agree with "by default, all https fetches get validation".  The
justification for me is that this is stadanrds/community-norms.

Yes, it's true that with "configure https URL to a non-broken server and
leave validation enabled" leads to "all packages fetched from a
validated connection".  But that's not really about packages as much as
"that's part of what https does and that's why it should be used instead
of http".

>> I view any https server giving a redirect to anything non-https as a
>> bug.  I am not aware of any reports of places serving packages over
>> https that return redirects to http.  If there are we should report that
>> to them and ask them to fix.
>
> It sounds like we are in agreement that:
>
> (a) https redirecting to non-https (`security downgrade') is bad

And it is a bug in the server.

> (b) it is important to report that case as a bug to fix

It is easy enough to scan public URLs.  I see this as a mostly
theoretical problem, and there are a vast number of server problems that
lead to more trouble, like allowing attackers to change files.

Consider:

  attacker compromises server
  attacker puts in a http redirect
  attacker doesn't change the file
  attacker intercepts http fetch with malware

> What I'm unclear on is: How would we _discover_ this to report it if
> pkg_add silently allows security downgrade of https redirecting to
> non-https?

We started out saying "we should validate".  I think we should make the
small change to validate and not make this complicated.

> Users don't have the opportunity when they run `pkg_add https://...'
> to see that the package was loaded from http:// instead before they
> can make a decision, the way they do in an interactive web browser.

And they don't get to audit the server to see anything else.

Web browsers are different, because web content is generally dreadful,
unlike serving files over https.  There are third-party refs in html all
the time and that's a mess security wise.  And, a large part of the
motivation for https in web is that users upload passwords.

> What's puzzling me, though, is that you seem to be concerned with the
> riskiness of the change, and yet you seem to be asking for a riskier
> version of the change.  That's fine by me on security grounds -- but I
> was specifically trying to minimize breakage by making the changes
> narrowly scoped to what the user explicitly asks for as follows:

I am concerned about different risks than you think I am.

We are talking about a change to validate https by default.  That's
going to throw errors instead of fetching when https doesn't have a
valid certificate.  That's ok because that is exactly what we are trying
to do.

What I object to is more code with more ifs to try to construct security
contracts that are unneceesary, as every bit of code can have issues and
also I don't think we should cross into that kind of complexity.

And, I want the behavior to be really simple and understandable.   So
"if validation is configured on (could be deafult), validate https" does
what we need and is extremely simple, both to describe and to code.
Yes, if serveres do odd things then it takes thought to predict, but it
will be seen that the code does exactly what it said it would do.

>> I see this as ending up way too complicated.  Instead, simply
>> 
>>   Validate certifications for all https connections if
>>   HTTPS_VALIDATE_CERTS=yes.
>> 
>>   Omit validation if HTTPS_VALIDATE_CERTS=no.
>
> Just to clarify, does your proposal for HTTPS_VALIDATE_CERTS=yes
> entail either of the following?
>
> 1. If http redirects to https (security upgrade), validate certs
>    anyway -- even though the client is already vulnerable to MITM.

Yes.  "validate https" means exactly that.  It is irrelevant that the
first fetch could have been compromised.   Not to the client - it bwas
doing something unsafe - but it's

> 2. If https redirects to http (security downgrade), allow that anyway
>    -- even though it means the client would become _newly_ vulnerable
>    to MITM despite earlier validation of user-configured https.

I see a server returning a http redirect and returning bad bits as both
a server bug and I don't think we can deal with that in general.

>>   Set the default value to yes for NetBSD 10.
>> 
>>   Set the default value to yes for any system where it is known that as
>>   part of proper base system setup that trust anchors are configured.
>> 
>>   Probably, just set it to yes everywhere, because it is a bug to fetch
>>   https without validatin, unless the user has asked not to validated.
>>   <pmc>This is probably a bit much for 2023Q4 pullup.</pmc>
>
> For now I'm not proposing to do anything outside NetBSD 10, so we can
> address that post-branch.
>
>>   Make no changes at all to ftp/http.  There is no bug to be fixed.
>
> I'm unclear on whether you mean `no changes at all when user
> configures ftp/http' or `no changes at all when pkg_add is directed,
> whether by local configuration or by remote redirect, to ftp/http'.

I mean make no changes to the code surrounding http/ftp fetching at
all.  So there is no checking for the V flag in libfetch and failing it.

>> Implicit in the above is that I still object to INSECURE_TRANSPORT as
>> being about "you should not do this" vs "let's fix the bug of not
>> validating".
>
> What I'm trying to get at with INSECURE_TRANSPORT is: override the
> security contract (for the transport layer) and just do what I and the
> alleged remote server say.  I think that is clearer than leaving one
> to wonder with HTTPS_VALIDATE_CERTS=1/0 what happens securitywise and
> compatibilitywise in downgrade or upgrade cases.

I disagree.  I object to declaring a security contract.  This is simply
a bug fix that https should do validation.  I find
HTTPS_VALIDATE_CERTS=yes/no to be extremely straightforward.  It is the
same as wget's --no-check-certificates option:

       --no-check-certificate
           Don't check the server certificate against the available
           certificate authorities.  Also don't require the URL host name to
           match the common name presented by the certificate.

           As of Wget 1.10, the default is to verify the server's certificate
           against the recognized certificate authorities, breaking the SSL
           handshake and aborting the download if the verification fails.
           Although this provides more secure downloads, it does break
           interoperability with some sites that worked with previous Wget
           versions, particularly those using self-signed, expired, or
           otherwise invalid certificates.  This option forces an "insecure"
           mode of operation that turns the certificate verification errors
           into warnings and allows you to proceed.

And, INSECURE_TRANSPORT blurs "don't allow http".  I think you're
proposing that without setting that, configuring http allows http.  But
it's not obvious from the name what it really means.


A separate question from fixing the https-isn't-validated bug might be

  Is an https server sending an http redirect such an egregious error
  that we are ok with erroring out with no escape valve?  The fix is to
  fix the server of course.


Despite an earlier comment (not by you!), I think it would be totally
unreasonable to not have an escape valve for https validation.  pkg
tools get used in all sorts of environments.  I used them in a network
of over 50 NetBSD machines disconnected from the Internet, where the
overall security argument was almost entirely disjoint from this
discussion.  I think we had SSL/TLS but I'm really not sure about
whether the certificate situation was set up right.  Sure, it's always
better to be better, but it's an effort tradeoff, and in that case it
would not have made any difference at all in overall security.

But, telling people that have odd setups that return an http redirect
over https and thus have issues "don't do that then" seems ok, even with
my odd experience.