Re: CVS commit: pkgsrc/pkgtools/cwrappers/files/bin

[Jonathan Perkin <jperkin%joyent.com@localhost>]

* On 2021-11-08 at 13:27 GMT, Christos Zoulas wrote:

> > On Nov 8, 2021, at 8:15 AM, Jonathan Perkin <jperkin%joyent.com@localhost> wrote:
> >
> >  https://github.com/joyent/pkgsrc-joyent/tree/master/openjdk12
> >  https://github.com/joyent/pkgsrc-joyent/tree/master/openjdk13
> >  https://github.com/joyent/pkgsrc-joyent/tree/master/openjdk14
> >  https://github.com/joyent/pkgsrc-joyent/tree/master/openjdk15
>
> That could be the case, but there are no NetBSD ones. I thought PKGSRC 
> was supposed to be OS neutral and maintained in a single place, but I 
> can see now this is not the case.

This is generally true, however for a limited number of packages it 
makes more sense for them to be in a separate repository as they use a 
different upstream source tarball (and so can't share patches).  For 
example I maintain my own GCC packages too:

  https://github.com/joyent/pkgsrc-joyent/tree/master/gcc9
  https://github.com/joyent/pkgsrc-joyent/tree/master/gcc10

which use the upstream illumos branch, as well as cleaning up a few 
things I don't like about the main pkgsrc packages.

Anyone working in pkgsrc knows about these (and is probably tired of me 
talking about them), and certainly the OpenJDK maintainers are aware of 
the work I did for 12-15.

> I see that you have packages for each one of the versions which 
> indicates to me that you used the previous version to bootstrap the 
> next one. Have you tried to bootstrap from something else, like linux 
> and it worked? Going from 11 to 16 and 17 and needing to build 13 and 
> 15 is no fun, specially if you are not planning to use them...

Yes, I had to create each package to bootstrap through to 15 even though 
generally only 11 and 15 are in use by users, but it wasn't a huge 
amount of work (and the intermediaries don't need to be updated).

I've not cross-compiled, all bootstraps are created for native.

> Finally can you please explain your statement that $ORIGIN is 
> unpredictable?

First of all we need to remind ourselves that it was only a few months 
ago that TNF was found to be distributing binary packages that were 
dynamically linked against the WRKOBJDIR of the build user.  This 
happens to be under /work for the TNF build hosts, but could easily have 
been /tmp or /var/tmp as is common in many setups, and I think we can 
all agree that would have been sub-optimal to say the least.

Despite all of the work we've done, and the strictness we try to enforce 
both with the wrappers and the post-build and post-install checks, these 
were still able to get through, and could have been catastrophic.

Secondly it's worth pointing out that in all cases over the years where 
folks have argued that we absolutely must have $ORIGIN to support some 
particular package, we've been able to work through and patch things in 
a clean way so that they instead use absolute paths.  Even for very 
complicated build systems like GHC which was the cause of the security 
issue above, it turned out to be a simple 4-line diff.

We ship something like 25,000 packages now, none of which use $ORIGIN, 
and all built and checked as strictly as we're able, and so even the 
mere thought of changing this in a fundamental way that could result in 
losing some of that strictness has to be questioned as seriously as 
possible.

Ultimately the problem with $ORIGIN is that it's very very difficult to 
check properly, and introduces a host of complications.  You're now 
going to have to parse the output object path of every compiler and 
linker call so that you can then try to calculate whether the $ORIGIN 
path relative to that is valid, is within the allowed buildlink paths, 
is using the correct path for PREFER_PKGSRC settings, etc.  What if the 
object is written to a temporary directory and is then moved somewhere 
else later in the build?  Do all of those rules still apply?  We can't 
detect the move and re-check, so now we're just basically hoping it's 
still good and that the new location isn't accidentally going to places 
it shouldn't.

You're also going to have to rewrite all of mk/check to do the same but 
instead based on where the object will eventually reside on the file 
system, and whether all of the files it may load at runtime are 
compatible.

Doing all this properly in a way that is 100% reliable is, well, going 
to be difficult, if even possible.

Even with our strict absolute path requirements it was still possible 
for some bad packages to end up being distributed to users.  I remain 
convinced that we should be striving to lock things down even further, 
and be even more strict about the checks that we apply to avoid any 
possibility of that happening again, rather than moving in the opposite 
direction by allowing $ORIGIN.

--
Jonathan Perkin  -  Joyent, Inc.  -  www.joyent.com