Re: BLAS in pkgsrc: present and future

[Greg Troxel <gdt%lexort.com@localhost>]

"Dr. Thomas Orgis" <thomas.orgis%uni-hamburg.de@localhost> writes:

> Am Thu, 30 May 2019 09:05:51 -0400
> schrieb Greg Troxel <gdt%lexort.com@localhost>:
>
>> There are cases for things like choosing the postgres version that
>> already have the separation of user-settable, package-settable, and
>> internal variables.   I would suggest looking at that, and also the gcc
>> version selection logic.
>
> Hm, mk/jpeg.buildlink3.mk seems to be a _bad_ example … motif … mysql …
> where is the postgres thing? Ah, pgsql it is:
>
> # $NetBSD: pgsql.buildlink3.mk,v 1.48 2018/10/23 16:02:51 adam Exp $
> #
> # User-settable variables:
> #
> # PGSQL_VERSION_DEFAULT
> #
> # Package-settable variables:
> #
> # PGSQL_VERSIONS_ACCEPTED
> #
> # Variables set by this file:
> #
> # PG_LIB_EXT
> # PGSQL_TYPE
> # PGPKGSRCDIR
>
> So it does set variables without underscore for the package Makefile to
> use. Oh, and it also sets PGSQL_PREFIX and PGSQL_VERSION. Those might
> be included in the list above.

A fair point.  I was blurring "output variables" and "internal".  It
does also set _PGSQL_VERSION internally, which is (properly) not
documented.

> The packages do not have one common way to choose a BLAS
> implementation. Many need the values of BLAS_LIBS and LAPACK_LIBS
> inserted as env/make/configure variables somehow. Also, some just want
> the name of the implementation to trigger their own configuration
> machinery. It seems to me that this is quite like the situation with
> postgresql (including the actually set variable PGSQL_VERSION).

Yes, that sounds ok.

>> would expect that the package would set the ACCEPTED variable and then
>> .include ../../mk/blas.mk
>
> … and do something with the provided variables. There might be cases
> that just look for certain libs being available (-lblas, -lopenblas, -mkl),
> but the names are not standardized enough for that to work universally
> (build variants of OpenBLAS, for example). Also, if multiple … oh dear.

The blas.mk file will standardize the names, by defining in the
documentation what values the output variable can have.  This  is in
effect a token in a new namespace.

> What happens if, since we allow multiple BLAS packges being installed,
> the dependency chain of a package pulls in BLAS directly and indirectly?
> There is the package triggering the indirect dependency … having its
> own BLAS_ACCEPTED and resulting implementation being chosen at build
> time. The current package has a different BLAS_ACCEPTED, resulting in a
> different BLAS being chosen. This setting will overrule whatever was
> chosen during the build of the BLAS-using dependency, I presume? 

This is basically a mess.  It's not really about the selection code, but
about the idea that linking in multiple implementations doesn't work,
unless they have private namespaces.

Absent logic to query dependencies for blas choice and intersect, then
the current package's invocation of blas.mk will produce an answer,
disconnected from what the others did.

> So we can have a package with binaries needing libblas-a.so being used
> as a dependency for a package that decides that libblas-b.so is used
> instead. This kind of situation I can avoid by choosing a fixed BLAS
> that triggers package failures if it is not supported.

True.

> How would this situation even be detected with the PREFERRED/ACCEPTED
> logic? In what interesting ways will the build already fail?

The real question is 

>> > Is it still fine to have BLAS_LIBS and LAPACK_LIBS defined? Should
>> > those be _PKGSRC_BLAS_LIBS, _BLAS_LIBS?  
>> 
>> If those are defined by the bl3 file and expected to be used in
>> CONFIGURE_ARGS in depending packages, those variables sound
>> appropriate.
>
> With or without underscore? So far, I don't see a case that exports
> names with underscore to be used in Makefiles.

Without.   I was off earlier.  There are
  user-settable variables
  package-settable variables
  mk-set variables to be used in package makefiles
  internal variables

The PKGSRC_ prefix is generally most necessary on the first type, as
they go in /etc/mk.conf.  We do not have a culture of prefixing PKGSRC_
on mk-set variables and package-settable variables.
  
>> I think if you look at the gcc, pgsql, kerberos and ncurses code
>> (really all mk/foo.mk for foo a kind of program name),
>
> GCC (compiler/gcc.mk) is a bit hefty for me now, but all others use
> variables like FOO_TYPE. Btw. What do these lines do?
>
> BUILD_DEFS+=            CURSES_DEFAULT
> BUILD_DEFS_EFFECTS+=    CURSES_TYPE

That registers those variables and their value in the binary package, so
it's available to be read later.


I am expecting that you will write a wip/mk/blas.mk and we will iterate
improving it, and once converged, hoist it to pkgsrc/mk/blas.mk.    Is
that what you are expecting?