Re: fixing libfetch as a first-class object

[Greg Troxel <gdt%lexort.com@localhost>]

John Klos <john%klos.com@localhost> writes:

I'm going to ignore pkgin mostly, not because it doens't matter, but
because it's a case of "whatever we do for pkg_add, also do it for
pkgin", and I think that statement is widely agreed on.

> I've read Taylor's carefully considered suggestions and patches, and I
> can't see where they'd be problematic. We've had much more disruptive
> changes that we've all been asked to simply accept rather than taking
> a more reasoned approach, yet here we're seemingly worried about edge
> cases that I just can't see are more important than the goal of having
> an expected trust mechanism in place.
>
> Please let me know if any part of my summary is incorrect:
>
> NetBSD < 10 ignores the lack of certificates, which is neither a good
> default, even if we want to avoid changing things, nor does it make
> sense, as in that behavior defies common sense.

I don't follow this.  NetBSD < 10 does not have the base system install
trust anchors.  Instead that is the responsibility of users.  Many
choose openssl-rootcerts-openssl.  ftp(1) does validate https by default
(disablable by env var), which will mean downloads fail if no trust
anchors are configured.  libfetch does not validate, and therefore
pkg_add and pkgin do not.

> NetBSD >= 10 fails if certificates aren't installed. This matches the
> principle of least astonishment and is a good default. Even if people
> accustomed to NetBSD pre-10 might be surprised should they update yet
> not address their lack of certificates, it'd be foolish to avoid this
> change because we're worried about these few people who upgrade
> without fixing certs.

I again don't follow this.  NetBSD >=10 behaves just as < 10, except
that a fresh install or a valid upgrade results in configured trust
anchors.

Note that I have never been opposed to the base system providing default
trust anchors.  What I objected to was pkgsrc overriding the base
system's judgement, and not respecting the user's manual configuration.
The NetBSD 10 approach is great and avoids all of those issues, and
Taylor did an awesome job addressing all the concerns.

> That said, I think we're all in agreement that pkgsrc / libfetch /
> pkg_add / pkgin should validate https.

Very broadly yes, but not really.

  Taylor's patch makes pkg_add set a flag to libfetch to validate https
  and disable http/ftp when the URl is https.  Unless a configuration is
  set not to do this.

  My proposal is to make libfetch validate https by default, unless
  there is configuration not to.  And not to change pkg_install/pkgin at
  all, because there is no need.

The difference is

  I object to defining a "security contract" due to general comfort with
  overpromising.  I see this as far simpler: by default https should
  validate, either because the standard more or less says it should, or
  because ~everybody in 2023 at least thinks it should so it is
  effectively as if it is in the standard.  Right not libfetch doesn't.
  We have a bug and we should just fix it.  Anyone relying on buggy
  behavior will have to cope, and as long as we make it relatively easy
  to do so (options not moralizing), that's ok.  This is always how it
  is when we fix bugs.

  The modify-pkg_add-to-call-with-flag approach has similar behavior
  when viewed at the "using pkg_add" level.  But I am concerned about:
  What is the interface to libfetch?  How is it documented?  If some
  other random program that uses libfetch (a documented public API) asks
  for https, should it get validation?  I think it should.

  Taylor's patch is trying to change almost nothing except pkg_add.  I
  get the point, but I think it's overly deferential to bug preservation
  at the expense of architectural complexity.

  I want to have clear, easy to explain semantics.  "https is validated"
  to me is that.  There's no need to say "unless it was a redirect from
  http" and "https to http is not allowed".  If we are going to disallow
  https->http, I'd like that to be separate in a library, because it is
  conceptually different, even if people might care about both, and we
  don't know what the next library user is going to need.

Another thing is

  I see this as not an emergency.  libfetch has failed to validate ever
  since it learned https.   NetBSD 10 has been almost released for a
  really long time.

vs

  It's really important for NetBSD 10 pkg_add to validate.

> I think there's some disagreement about whether https should be
> allowed to downgrade to http. I can see how one can argue that the it
> should, since we supposedly trust the https server, but I can also see
> why people wouldn't want that. That seems like something that might
> end up being a flag people can set.

Agreed that we probably see this slightly differently and I think we all
see it as a second-order issue.

I'd want to ask

  What do the standards say?

  What is the practice across command-line downloading programs like
  ftp(1), wget, curl?  Is there in fact community consensus?

> When it comes to http redirecting to https, the question arises
> whether this should fail if certificates aren't installed. Personally,
> I think it should fail, but I can just as easily see why, if we're in
> the business of avoiding unexpected changes, pkg_add and friends
> should proceed even without certificates. Again, this could be a flag.

I think it follows that if http redirects to https, the https fetch
should follow the same validation rules as if https were done in the
first place.   Yes, if this bites somehow, they'll have to fix their
server not to do that, or turn on the "don't validate" flag.

> So if we agree that https->http downgrading is a flag that defaults to
> fail and http->https upgrading is a flag that defaults to validate,
> then what else is there that's keeping this from being added?

Having logic spread over pkg_add instead of just fixing libfetch.

> Just like in our very long threads in port-vax@, perhaps I've missed
> some things, but is this a fair summary?

You left out all the parts that are actually what the disagreement is
about, but what you did say is accurate :-)

I will have a go at a change to libfetch, just the validate-by-default,
opt out part.  I expect that to be really simple, and I think it will
get us almost all of what we want, and what we need for NetBSD 10 not to
be embarassing.

At this point I'm ok with validate-by-default always in pkgsrc-current,
and limit-to-netbsd-10 on the branch.

If people are ok with validation for all systems on the branch, I could
see that being ok.  This is after all a bugfix.