> On Nov 8, 2021, at 10:02 AM, Jonathan Perkin <jperkin%joyent.com@localhost> wrote: > > * On 2021-11-08 at 13:27 GMT, Christos Zoulas wrote: > >>> On Nov 8, 2021, at 8:15 AM, Jonathan Perkin <jperkin%joyent.com@localhost> wrote: >>> >>> https://github.com/joyent/pkgsrc-joyent/tree/master/openjdk12 >>> https://github.com/joyent/pkgsrc-joyent/tree/master/openjdk13 >>> https://github.com/joyent/pkgsrc-joyent/tree/master/openjdk14 >>> https://github.com/joyent/pkgsrc-joyent/tree/master/openjdk15 >> >> That could be the case, but there are no NetBSD ones. I thought PKGSRC was supposed to be OS neutral and maintained in a single place, but I can see now this is not the case. > > This is generally true, however for a limited number of packages it makes more sense for them to be in a separate repository as they use a different upstream source tarball (and so can't share > patches). For example I maintain my own GCC packages too: > > https://github.com/joyent/pkgsrc-joyent/tree/master/gcc9 > https://github.com/joyent/pkgsrc-joyent/tree/master/gcc10 > > which use the upstream illumos branch, as well as cleaning up a few things I don't like about the main pkgsrc packages. > > Anyone working in pkgsrc knows about these (and is probably tired of me talking about them), and certainly the OpenJDK maintainers are aware of the work I did for 12-15. That makes sense I guess, although it would be still be better to have everything in one place... > >> I see that you have packages for each one of the versions which indicates to me that you used the previous version to bootstrap the next one. Have you tried to bootstrap from something else, like >> linux and it worked? Going from 11 to 16 and 17 and needing to build 13 and 15 is no fun, specially if you are not planning to use them... > > Yes, I had to create each package to bootstrap through to 15 even though generally only 11 and 15 are in use by users, but it wasn't a huge amount of work (and the intermediaries don't need to be > updated). > > I've not cross-compiled, all bootstraps are created for native. It is a lot of work considering you have to do it for each architecture... It is a lot simple if you are able to use a foreign jdk. And that of course does not guarantee that the build will work (because it is using bootstrap libraries for new binaries), but we have been lucky so far. >> Finally can you please explain your statement that $ORIGIN is unpredictable? > > First of all we need to remind ourselves that it was only a few months ago that TNF was found to be distributing binary packages that were dynamically linked against the WRKOBJDIR of the build > user. This happens to be under /work for the TNF build hosts, but could easily have been /tmp or /var/tmp as is common in many setups, and I think we can all agree that would have been sub-optimal > to say the least. Yes, that was pretty bad. > Despite all of the work we've done, and the strictness we try to enforce both with the wrappers and the post-build and post-install checks, these were still able to get through, and could have been > catastrophic. Yes, was there a post-mortem for that? I.e. what have we done to prevent something like this from happening in the future? > Secondly it's worth pointing out that in all cases over the years where folks have argued that we absolutely must have $ORIGIN to support some particular package, we've been able to work through > and patch things in a clean way so that they instead use absolute paths. Even for very complicated build systems like GHC which was the cause of the security issue above, it turned out to be a > simple 4-line diff. I just want to understand how to fix the jdk build issue. I think $ORIGIN is elegant as a concept because it allows relocatable packages, but I am not advocating that we should use it anywhere else. I think we should use it though for packages like the jdk where LD_LIBRARY_PATH interferes with the build. > We ship something like 25,000 packages now, none of which use $ORIGIN, and all built and checked as strictly as we're able, and so even the mere thought of changing this in a fundamental way that > could result in losing some of that strictness has to be questioned as seriously as possible. I am not advocating to start using $ORIGIN anywhere else. In fact what is bothering me with my patch (and I want to fix it), is that I only want to allow $ORIGIN on packages I specify that I want it (via some configuration option), and not on every package. > Ultimately the problem with $ORIGIN is that it's very very difficult to check properly, and introduces a host of complications. You're now going to have to parse the output object path of every > compiler and linker call so that you can then try to calculate whether the $ORIGIN path relative to that is valid, is within the allowed buildlink paths, is using the correct path for PREFER_PKGSRC > settings, etc. What if the object is written to a temporary directory and is then moved somewhere else later in the build? Do all of those rules still apply? We can't detect the move and > re-check, so now we're just basically hoping it's still good and that the new location isn't accidentally going to places it shouldn't. I think limiting it to the jdk and auditing manually its $ORIGIN uses is tractable. > You're also going to have to rewrite all of mk/check to do the same but instead based on where the object will eventually reside on the file system, and whether all of the files it may load at > runtime are compatible. > > Doing all this properly in a way that is 100% reliable is, well, going to be difficult, if even possible. I am not signing up for that. :-) > Even with our strict absolute path requirements it was still possible for some bad packages to end up being distributed to users. I remain convinced that we should be striving to lock things down > even further, and be even more strict about the checks that we apply to avoid any possibility of that happening again, rather than moving in the opposite direction by allowing $ORIGIN. I understand, but I still want to be able to build the jdk correctly (i.e. without using the bootstrap libraries in the new native binaries, or being able to use a foreign jdk), and I don't see a way forward with LD_LIBRARY_PATH. christos
Attachment:
signature.asc
Description: Message signed with OpenPGP