Summary of implicit DEPENDS problem

[Jonathan Perkin <jperkin%mnx.io@localhost>]

Here's my attempt to summarise the issue with implicit depends.

Before listing the general issues, I should point out that I discovered 
that my environment was exacerbating the problem.  Our GCC packages set 
--with-local-prefix to PREFIX, so that users can build software using 
libraries installed by pkgsrc easily outside of a pkgsrc environment.

Unfortunately this effectively bypasses the buildlink wrappers.  pbulk helps 
to ensure only dependencies are available during each package build, but due 
to the issues enumerated below, we can still end up linking against 
libraries that are installed as dependencies but not buildlinked and have 
it pass check-shlibs, resulting in shipping broken packages.

I've since fixed this part at least by enhancing our gcc-libs specs files to 
set link_* to sanitised library paths:

  https://github.com/TritonDataCenter/pkgsrc-extra/commit/83ec13fb4ade1c115334fa7e73f0cdd3284c5aae

Ok, onto the issues.

 * Problem 1: check-shlibs is insufficient.

   In its current form, mk/check/check-shlibs-elf.awk only checks packages
   that are listed in the depends file, and the depends file only contains
   direct dependencies.  If a file is linked against a library from an
   indirect dependency then it is ignored, even though that library may not
   be a runtime dependency.

   For a concrete example, nodejs had a build dependency on python which
   included python's buildlink3.mk, which in turn pulled in libiconv.
   nodejs then linked against libiconv, but as libiconv wasn't listed in
   the depends file at all, check-shlibs simply ignored it.  We ended up
   shipping a broken node package that failed at runtime because it was
   missing a dependency on libiconv.

   One might ask why we don't just uncomment the "Not yet" that would
   enable checking all packages:

     https://github.com/TritonDataCenter/pkgsrc/blob/trunk/mk/check/check-shlibs-elf.awk#L99-L100

   The simple answer is because it breaks a huge amount of packages.  I
   tried this in a bulk build and gave up pretty quickly.  Outside of a
   pbulk environment where there may be lots of random packages installed
   for the build to pick up, the results would only be worse.

 * Problem 2: BUILDLINK_DEPMETHOD doesn't propagate correctly

   BUILDLINK_DEPMETHOD for each package defaults to "full", unless it is
   overridden somewhere.  This means that dependencies that are included
   by a package with DEPMETHOD set to "build" are incorrectly tagged as
   "full".  For example consider the following:

    category/package/Makefile:
     BUILDLINK_DEPMETHOD.foo= build
     .include "../../category/foo/buildlink3.mk"

    category/foo/buildlink3.mk:
     .include "../../category/bar/buildlink3.mk"

   BUILDLINK_DEPMETHOD.foo will be set to "build", but
   BUILDLINK_DEPMETHOD.bar will be set to "full".

   This causes problems with check-shlibs, as it will think bar is a full
   dependency, but in reality it is not and won't be available at runtime.

   My current patch-set fixes this by using pushing depmethod onto a stack
   while the buildlink tree is parsed, similar to how USE_BUILTIN is
   correctly propagated, but we then run into the next problem.

 * Problem 3: buildlink3.mk inclusion guards

   Every buildlink3.mk file contains the following form:

    .if !defined(FOO_BUILDLINK3_MK)
    FOO_BUILDLINK3_MK:=
    ..logic to .include other bl3, set DEPMETHOD, etc.
    .endif

   This means that packages never see the full dependency graph, they only
   see direct dependencies, and unique indirect dependencies.  Let's use
   harfbuzz as a real-world example of how this breaks things, using the
   tree prior to my commit to work around the problem.

     $ cd fonts/harfbuzz
     $ bmake show-buildlink3
     gtk-doc
         glib2
             iconv
             gettext
                 iconv
             libffi
             pcre2
             zlib
             pthread
     glib2
     gobject-introspection
     ...

   gtk-doc's buildlink3.mk explicitly sets DEPMETHOD=build, which is
   reasonable as it's only ever used at build time.  It in turn includes
   glib2, which is then tagged as "build" too.

   Later, harfbuzz also directly includes glib2 as it is required as a full
   dependency.  However, because of the buildlink3 inclusion guards, while
   glib2 will now correctly be set to "full" (because that's the default for
   top-level inclusions), none of its .includes will be re-parsed and also
   tagged as "full".  You can see that clearly in the "show-buildlink3"
   output as the second "glib2" never descends into its dependencies again.

   This is why my hacky fix for now is to include glib2 prior to gtk-doc,
   so that glib2 and all its dependencies are correctly logged as "full",
   before including gtk-doc as "build".

   I don't know how to fix this without a radical overhaul of buildlink3.mk
   files.  Joerg suggested including additional metadata in BUILDLINK_TREE
   identifiers, but I still don't see how this gets around the inclusion
   guard problem.

My current patch to resolve the issues that I'm able to at this time is 
available here:

  https://github.com/TritonDataCenter/pkgsrc/commit/b6c52a5e078640b9b768d08e760c4f897ca3aedc

This:

  * Fixes problem #2, but is incomplete due to problem #3.

  * Adds IMPLICIT_DEPENDS and IMPLICIT_BUILD_DEPENDS, ensuring that the
    depends file includes them too.

  * Enhances check-shlibs to account for implicit dependencies, and fails if
    a package links against any file that is not either a "full" or
    "implicit-full" dependency.

As per the commit message, this isn't ready for integrating into pkgsrc yet 
though.  Due to problem #3 we see occasional false positives if an indirect 
dependency is tagged as "build" when it is later correctly tagged as "full".

I'm happy to keep this in my tree for now and commit workarounds as I find 
them, as I don't think other folks would be happy with any false positives.

Ideas on how to solve this completely would be appreciated.  For now I'm 
mainly focussed on ensuring my binary packages are not broken as I consider 
that to be a critical failure that I'm pretty embarrassed about :/

Cheers,

--
Jonathan Perkin   -   mnx.io   -   pkgsrc.smartos.org
Open Source Complete Cloud   www.tritondatacenter.com