Re: Call for testing: certctl, postinstall, TLS trust anchors

[Taylor R Campbell <riastradh%NetBSD.org@localhost>]

> Date: Sun, 08 Oct 2023 10:54:13 -0400
> From: Greg Troxel <gdt%lexort.com@localhost>
> 
> (I've been putting off thinking about and dealing with this due to
> juggling too many other things.)

No worries, glad you found some time to review it!

> Taylor R Campbell <riastradh%NetBSD.org@localhost> writes:
> 
> > The new certctl(8) tool is provided to manage the TLS trust anchors
> > configured in /etc/openssl/certs with a simple way to change the
> > source of trust anchors or distrust individual ones -- and with a
> > manual override, if you would rather use another mechanism to do it,
> > like the commands available in the security/mozilla-rootcerts or
> > security/ca-certificates packages, or the special-purpose
> > security/mozilla-rootcerts-openssl package.
> 
> This says "TLS trust anchors", but I wonder if that's accurate.  Isn't
> this "pkix trust anchors, for which the most common case is TLS"?  I
> have not dug in to the openssl library calls, but my impression is that
> openssl the installed software does pkix validation in general, and the
> installed trust anchors will be used for invocations to validate pkix certs
> separately from TLS.

If you know of applications that uses /etc/openssl/certs on NetBSD (or
/etc/ssl/certs on FreeBSD, or /etc/pki/tls/certs on Fedora), or SSLDIR
in pkgsrc, to find trust anchors for anything other than TLS, I'd like
to hear about it.

My guess is that such cases are very rare if they exist at all, and
they are totally dominated by applications that rely on it for TLS
validation.

> After reading, I think what's going on is
> 
>   1) mozilla rootcert situation is a bit of a mess smantically

Not really.  The CAs are clearly marked in certdata.txt for different
purposes.  I imported the content according to the metadata about
Mozilla's trust; more below.

>   2) certctl is installing the subset that is intended for TLS

Correct.

You can also use the same certctl(8) program to install trust anchors
in another path for other purposes, like /etc/smime/certs.  I made
sure it would work (both for testing and for other possible uses) but
didn't lift a finger to make it happen because TLS is critical and
everything else is niche.

>   3) the installed certs will be used for all uses, not just TLS
>   (e.g. SMIME), and because certs intended for SMIME but not "server"
>   are missing, the wrong thing will happen sometimes, but because many
>   CAs do both (?) it will often be ok.

See above: if you know of applications that rely on /etc/openssl/certs
for S/MIME, and it's not just a joke (which most open-ended
interorganizational use of S/MIME that I'm aware of is --
intraorganizational uses managed by a corporate IT department purely
for internal or partner use aside), I'm curious to see.

>   4) This is all not obvious, and
>      a) It's not the least bit clear that the right thing is happening.
>      b) I expect ~nobody to understand this.

Is any of the text in the certctl(8) or hier(7) man pages unclear
about this?  I tried to clarify the purpose of /etc/openssl/certs for
TLS trust anchors specifically in that text.

> Looking in /usr/share/certs/mozilla, it continues to be non-obvious.
> The idea that 'all' has "untrusted CAs" seems crazy; if they aren't
> trusted, why are they in the root set, which is by definition the set of
> CAs which meet the rules and are therefore trustworthy?

`all' has everything in certdata.txt.

`server' has only what Mozilla has chosen to trust for TLS.

`email' has only what Mozilla has chosen to trust for S/MIME.

> I see code is empty.  I'm going to ignore this.

Yes.  The certdata.txt format has a way to say that trust anchors are
fit for code-signing, so for completeness I exposed that via the
directory /usr/share/certs/mozilla/code, but (a) there are no trust
anchors in certdata.txt that use it, and (b) there is nothing in
NetBSD that would use such trust anchors anyway.

> With a bit of ls/sort/uniq/comm, I see that there are certs in all that
> do not appear in email or server:
> 
>   Explicitly_Distrust_DigiNotar_Root_CA.pem
>   Symantec_Class_1_Public_Primary_Certification_Authority_-_G6.pem
>   Symantec_Class_2_Public_Primary_Certification_Authority_-_G6.pem
>   TUBITAK_Kamu_SM_SSL_Kok_Sertifikasi_-_Surum_1.pem
>   TrustCor_ECA-1.pem
>   TrustCor_RootCert_CA-1.pem
>   TrustCor_RootCert_CA-2.pem
>   Verisign_Class_1_Public_Primary_Certification_Authority_-_G3.pem
>   Verisign_Class_2_Public_Primary_Certification_Authority_-_G3.pem

That looks correct.  It matches what I see in certdata.txt and the
special annotation about Tubitak Kamu SM at
<https://wiki.mozilla.org/CA/Additional_Trust_Changes>.  See
external/mpl/mozilla-certdata/share/Makefile for notes on how this
particular sausage is made.

These exclusions also match my knowledge of the history:

- Digi-Notar was infamously compromised by the Iranian state a decade
  ago.

- Symantec's CA was nixed after a series of certificate misissuance
  incidents and audit failures a few years ago:
  https://blog.mozilla.org/security/2018/03/12/distrust-symantec-tls-certificates/
  Since Symantec had bought Verisign's CA business that probably
  covered the two Verisign certificates as well.

- TrustCor was nixed after shady behaviour in response to public
  scrutiny last year:
  https://groups.google.com/a/mozilla.org/g/dev-security-policy/c/oxX69KFvsm4/m/WJXUELicBQAJ  

> How is SMIME supposed to work?  Are SMIME validators, which presumably
> use openssl as an engine, supposed to maintain a different trust anchor
> set?  Where? 

As far as I'm aware, S/MIME is only ever seriously deployed within a
single organization at a time (or a closed set of partnering
organizations).  So I don't expect anything about it to seriously work
out of the box and I have no idea what public CAs do about it.

I'm prioritizing effort on TLS, but I _installed_ the email
certificates as pem files under /usr/share so that they're available
in case someone wants to do something with them like declare
/etc/smime/certs as the place to find the trust anchors and configure
them with certctl(8) using a different config file.

> I see that mozilla-rootcerts-openssl has 169 certificates, so that must
> be all, which appears to be a really serious bug.

Yes, pkgsrc security/mozilla-rootcerts (and mozilla-rootcerts-openssl)
includes non-TLS trust anchors.  I meant to report this somewhere but
ran out of round tuits.

> Maybe we don't want to deal with this, but I think it needs to be
> clearer, especially as this upgrade to certctl from
> mozilla-rootcerts-openssl does:
> 
>   resolves a security issue where "untrusted" certs were trust anchors (yay)
> 
>   removes trust anchors for email, likely breaking some SMIME uses (but
>   not sure in practice how much, given tbird uses internal and gpg2 uses
>   gnutls).  (theoretical boo)

I expect this is extremely theoretical.

> I'll also observe that it's mostly because I have avoide digging in
> until today, but this larger situation of the subsets, what was and what
> is, is news to me today.  

The situation with security/mozilla-rootcerts is actually worse
because it doesn't interpret the DISTRUST_AFTER annotations, so CAs
that _were_ trusted for TLS but have _now_ been sunset are still
included.  That was news to me too...

> > So it would be helpful if you could test updating NetBSD in whatever
> > way you do it (sysinst, untar/etcupdate/postinstall, etcmanage,
> > something even more bespoke), and let me know if anything goes wrong
> > with the TLS trust anchors:
> 
> I did an update via [...]

Thanks!

> Given the man page, I expected to have an empty /usr/openssl/untrusted
> directory but that is not in any of the sets.  Perhaps the idea is that
> it is created on demand, but I didn't figure that out from the man page.

It is created on demand.  Not sure it's worth including in sets, just
more trouble with set lists for an empty directory with no
user-serviceable parts.

> I don't understand the "distrust" mechanism.  The intent is obvious, but
> there must be some data stored someplace.  I think it's
> /etc/openssl/untrusted, where if a cert is present (or symlinked?) and
> matches bit-for-bit, it's excluded.  But that is really non-obvious from
> the certct man page, even for me.   I realize that may be a
> implementation detail not specified by the interface, but a
> 
>   (While not part of the defined interface to certctl, the untrust
>   mechanism is implemented by placing a copy of the untrusted
>   certificatee in /etc/openssl/untrusted.)

It is very much an implementation detail and I didn't want to put too
much thought into the details.  The really important thing is that the
interface work so that we can distribute SAs that give remediation
steps like:

# certctl untrust TrustCor_ECA-1.pem

and they will work.  I am not committing to the implementation because
the expedient thing was grody but good enough for now, and I may want
to change it.

> What's not obvious and is part of the interface is that if I untrust a
> cert (e.g. because I don't believe that the CA is sound), then when that
> CA cert is updated in the mozilla set, will be untrust persist, or not?
> I am guessing not, and this is hard to fix but I think it's important to
> tbe clear on "what does it mean to untrust a cert", because people may
> think it does what they want instead of more narrowly what it says.

If you do `certctl untrust TrustCor_ECA-1.pem', then it will remain
untrusted on update.

> What's not obvious is a recommended place to put a CA cert (not in the
> mozilla set) that I wish to have as a trust anchor.   I realize from
> reading that it can be literally any pathname on the entire system.  But
> certctl says "you can no longer use /etc/openssl/certs like used to".
> 
> I think I lean to /etc/openssl/manual, but that is probably just me.

I figured that you might install other trust anchors as static data
files from packages, e.g. /usr/pkg/share/gdt-rootcerts, and link them
in through another `path' directive in /etc/openssl/certs.conf.

> > 5. Do you hit any messages or warnings or failures that you don't
> >    understand?
> 
> Not really, but certctl untrusted man page paragraph is confusing.  It
> should simply say have been excluded and the "so that" should be omitted
> or at most a perenthetical remark.   It feels like opencoding the doc
> for another command as if it is not just a reference, and I find such
> things confusing.

How about this?

     untrusted   List absolute paths to certificates that have been
                 excluded by certctl untrust.  certctl rehash will not
                 put these in certsdir.  Paths are printed one per
                 line, encoded in vis(1) format to escape any shell
                 metacharacters.

And, correspondingly:

     list        List absolute paths to trusted certificates.  certctl
                 rehash will populate certsdir with these.  Paths are
                 printed one per line, encoded in vis(1) format to
                 escape any shell metacharacters.

> What is not clear is why the email ones are left out, and if they'd be
> not used for server validation due to key usage flags, or the rationale
> for the policy shift from the pkgsrc package approach.   I'm not saying
> it's wrong, but I had viewed this as "do the same thing, but move it to
> base and change the mechanism", and now it's clear to me that it's more.

I'm not sure it's as much a policy shift as implementing the policy
that was suggested in the DESCR and implicitly assumed by
applications, and then realizing mozilla-rootcerts was doing it wrong
all along.  From DESCR:

   It also provides a script, mozilla-rootcerts, which can be used to
   install the root CA certificates distributed by the Mozilla Project
   into a location that makes them usable by TLS implementations, extract
   them to the current working directory, or rehash the existing
   certificates.

   NB: This package provides certificates, but does not as a consequence
   of installation place them in a location that makes them immediately
   usable by SSL/TLS implementations.