tech-pkg archive
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index][Old Index]
Re: fixing libfetch as a first-class object
I don't follow this. NetBSD < 10 does not have the base system install
trust anchors. Instead that is the responsibility of users. Many
choose openssl-rootcerts-openssl. ftp(1) does validate https by default
(disablable by env var), which will mean downloads fail if no trust
anchors are configured. libfetch does not validate, and therefore
pkg_add and pkgin do not.
lain: {134} uname -a
NetBSD lain.zia.io 9.0_STABLE NetBSD 9.0_STABLE (LAIN) #0: Fri Jun 12 01:17:05 UTC 2020 john%lain.zia.io@localhost:/usr/obj-amd64/sys/arch/amd64/compile/LAIN amd64
lain: {135} rm /etc/openssl/certs/*
lain: {136} ftp "https://cdn.netbsd.org/pub/pkgsrc/distfiles/0verkill-0.16.tgz";
Trying [2a04:4e42:87::262]:443 ...
Requesting https://cdn.netbsd.org/pub/pkgsrc/distfiles/0verkill-0.16.tgz
100% |***************************************************| 270 KiB
854.54 KiB/s 00:00 ETA
276579 bytes retrieved in 00:00 (853.84 KiB/s)
Nothing has been changed to skip cerficate validation.
NetBSD >= 10 fails if certificates aren't installed. This matches the
principle of least astonishment and is a good default. Even if people
accustomed to NetBSD pre-10 might be surprised should they update yet
not address their lack of certificates, it'd be foolish to avoid this
change because we're worried about these few people who upgrade
without fixing certs.
I again don't follow this. NetBSD >=10 behaves just as < 10, except
that a fresh install or a valid upgrade results in configured trust
anchors.
newanath# rm /etc/openssl/certs/*
newanath# uname -a
NetBSD newanath.zia.io 10.0_RC1 NetBSD 10.0_RC1 (GENERIC) #0: Tue Dec 26 15:27:07 UTC 2023 root%newanath.zia.io@localhost:/usr/obj-amd64/sys/arch/amd64/compile/GENERIC amd64
newanath# rm /etc/openssl/certs/*
newanath# ftp "https://cdn.netbsd.org/pub/pkgsrc/distfiles/0verkill-0.16.tgz";
Trying [2a04:4e42:31::262]:443 ...
FFFFFFFFFFFFFFFF:error:0A000086:SSL routines:tls_post_process_server_certificate:certificate verify failed:/usr/src/crypto/external/bsd/openssl/dist/ssl/statem/statem_clnt.c:1889:
Trying 151.101.209.6:443 ...
FFFFFFFFFFFFFFFF:error:0A000086:SSL routines:tls_post_process_server_certificate:certificate verify failed:/usr/src/crypto/external/bsd/openssl/dist/ssl/statem/statem_clnt.c:1889:
ftp: Can't connect to `cdn.netbsd.org:https'
Note that I have never been opposed to the base system providing default
trust anchors. What I objected to was pkgsrc overriding the base
system's judgement, and not respecting the user's manual configuration.
The NetBSD 10 approach is great and avoids all of those issues, and
Taylor did an awesome job addressing all the concerns.
That said, I think we're all in agreement that pkgsrc / libfetch /
pkg_add / pkgin should validate https.
Very broadly yes, but not really.
Taylor's patch makes pkg_add set a flag to libfetch to validate https
and disable http/ftp when the URl is https. Unless a configuration is
set not to do this.
Let's skip any discussion about "disable http/ftp" separate, since Taylor
said after discussion this would be excluded from these patches.
My proposal is to make libfetch validate https by default, unless
there is configuration not to. And not to change pkg_install/pkgin at
all, because there is no need.
The difference is
I object to defining a "security contract" due to general comfort with
overpromising. I see this as far simpler: by default https should
validate, either because the standard more or less says it should, or
because ~everybody in 2023 at least thinks it should so it is
effectively as if it is in the standard. Right not libfetch doesn't.
We have a bug and we should just fix it. Anyone relying on buggy
behavior will have to cope, and as long as we make it relatively easy
to do so (options not moralizing), that's ok. This is always how it
is when we fix bugs.
So far, understood.
The modify-pkg_add-to-call-with-flag approach has similar behavior
when viewed at the "using pkg_add" level. But I am concerned about:
What is the interface to libfetch? How is it documented? If some
other random program that uses libfetch (a documented public API) asks
for https, should it get validation? I think it should.
Understood, and agreed.
Taylor's patch is trying to change almost nothing except pkg_add. I
get the point, but I think it's overly deferential to bug preservation
at the expense of architectural complexity.
I don't want to speak for anyone else, but it seems to me that Taylor was
defaulting to doing less intrusive things.
I want to have clear, easy to explain semantics. "https is validated"
to me is that. There's no need to say "unless it was a redirect from
http" and "https to http is not allowed". If we are going to disallow
https->http, I'd like that to be separate in a library, because it is
conceptually different, even if people might care about both, and we
don't know what the next library user is going to need.
So http->https should have validation, like NetBSD 10. Got it. And
https->http should be allowed, even if it's not a great idea. Got it.
Another thing is
I see this as not an emergency. libfetch has failed to validate ever
since it learned https. NetBSD 10 has been almost released for a
really long time.
vs
It's really important for NetBSD 10 pkg_add to validate.
I see the desire for consistency. Even if NetBSD 10 released tomorrow,
though, I agree this could be added to pkgsrc and left out of 2023Q4, even
though it'd be nice if the first set of binary packages for NetBSD 10
included this.
I think there's some disagreement about whether https should be
allowed to downgrade to http. I can see how one can argue that the it
should, since we supposedly trust the https server, but I can also see
why people wouldn't want that. That seems like something that might
end up being a flag people can set.
Agreed that we probably see this slightly differently and I think we all
see it as a second-order issue.
I'd want to ask
What do the standards say?
What is the practice across command-line downloading programs like
ftp(1), wget, curl? Is there in fact community consensus?
Indeed. Definitely orthogonal but worth discussing. In our case, it'd
be good to wonder in the specific context of pkgsrc whether it's a good
idea or not, and why or why not it should be allowed, or warrants a
warning, et cetera. For some things, it hardly matters, but if I choose a
package repository, I really would neither want nor expect it to redirect
to http in the future. I'd prefer a warning / failure. But that's another
discussion (just saying that concensus is great, but might still not be
the best for this situation).
When it comes to http redirecting to https, the question arises
whether this should fail if certificates aren't installed. Personally,
I think it should fail, but I can just as easily see why, if we're in
the business of avoiding unexpected changes, pkg_add and friends
should proceed even without certificates. Again, this could be a flag.
I think it follows that if http redirects to https, the https fetch
should follow the same validation rules as if https were done in the
first place. Yes, if this bites somehow, they'll have to fix their
server not to do that, or turn on the "don't validate" flag.
Right.
So if we agree that https->http downgrading is a flag that defaults to
fail and http->https upgrading is a flag that defaults to validate,
then what else is there that's keeping this from being added?
Having logic spread over pkg_add instead of just fixing libfetch.
I see. That makes sense.
Just like in our very long threads in port-vax@, perhaps I've missed
some things, but is this a fair summary?
You left out all the parts that are actually what the disagreement is
about, but what you did say is accurate :-)
Ha ha ha... Well, I'm better informed now, thank you, and if even one
person understands the thread better as a result of this summary, then
it's worth it.
I will have a go at a change to libfetch, just the validate-by-default,
opt out part. I expect that to be really simple, and I think it will
get us almost all of what we want, and what we need for NetBSD 10 not to
be embarassing.
At this point I'm ok with validate-by-default always in pkgsrc-current,
and limit-to-netbsd-10 on the branch.
If people are ok with validation for all systems on the branch, I could
see that being ok. This is after all a bugfix.
That'll be good to see.
Thanks,
John
Home |
Main Index |
Thread Index |
Old Index