Re: CVS commit: pkgsrc/security/netpgpverify/files

[Sorry for the long post, more an aide memoire for me in case there's a "next time" -- agc]

I ran the testit.sh script in netpgpverfiy/files a number of times to make sure that what I'd done was good, and that worked fine - I've just run it again, and it worked fine.

Bootstrapping, though - this is where it gets complicated - the relevant compilation command line from the bootstrap output (on NetBSD) is:

cc -O2 -D_FORTIFY_SOURCE=2 -I/usr/include -D_LARGEFILE_SOURCE -D_LARGE_FILES -D_FILE_OFFSET_BITS=64 -DDEF_UMASK=0022 -I/d/repo/pkgsrc/bootstrap/work/wrk/pkgtools/pkg_install/work/libfetch -I/d/repo/pkgsrc/bootstrap/work/wrk/pkgtools/pkg_install/work -DHAVE_NBCOMPAT_H=1 -I/d/repo/pkgsrc/bootstrap/work/wrk/pkgtools/pkg_install/work/libnbcompat -I/usr/include -c libverify.c
libverify.c: In function 'read_compressed':
libverify.c:1319:9: warning: implicit declaration of function 'netpgpv_BZ2_bzDecompressInit'; did you mean 'BZ2_bzDecompressInit'? [-Wimplicit-function-declaration]
ok = (netpgpv_BZ2_bzDecompressInit(&bz, 1, 0) == BZ_OK);
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
BZ2_bzDecompressInit
libverify.c:1333:9: warning: implicit declaration of function 'netpgpv_BZ2_bzDecompress'; did you mean 'BZ2_bzDecompress'? [-Wimplicit-function-declaration]
ok = (netpgpv_BZ2_bzDecompress(&bz) == BZ_STREAM_END);
^~~~~~~~~~~~~~~~~~~~~~~~~
BZ2_bzDecompress

so it looks like bootstrap scripts are eventually picking up the bzlib.h in /usr/include.

I fixed that locally by renaming files, and am getting a related error, this time during linking.

Turns out libnetpgpverify.a in bootstrap is missing some files in its archive. No bzlib.o or zlib.o

This is caused by the Makefile in work/wrk/pkg_install/work/netpgp/ being incomplete

# $NetBSD: Makefile.lib.in,v 1.2 2016/06/30 21:10:59 agc Exp $

LIB= libnetpgpverify.a

OBJS= b64.o bignum.o bufgap.o digest.o
OBJS+= libverify.o misc.o
OBJS+= pgpsum.o rsa.o

OBJS+= sha1.o sha2.o md5c.o rmd160.o

PREFIX=/usr/local
MANDIR=/usr/local/man

all: ${LIB}

${LIB}: ${OBJS}
ar crv ${LIB} ${OBJS}

install:
${BSD_INSTALL_LIB} ${LIB} ${DESTDIR}${PREFIX}/lib
${BSD_INSTALL_DATA} verify.h ${DESTDIR}${PREFIX}/include/netpgp
${BSD_INSTALL_DATA} libnetpgpverify.3 ${DESTDIR}${PREFIX}/${PKGMANDIR}/man3

clean:
rm -rf *.core ${OBJS} ${LIB}
cleandist:
rm -rf *.core ${OBJS} ${LIB} Makefile

(i.e. it's missing the line adding bzlib.o and zlib.o to OBJS)

Turns out that happens when bootstrapping pkg_install:

cd /d/repo/pkgsrc/pkgtools/pkg_install/work/netpgp && /usr/bin/sed -e '/zlib/d' Makefile.lib.in >Makefile.in && /bin/rm -f bzlib.h zlib.h && /bin/sh ./configure && /usr/bin/env USETOOLS=no PKGSRC_MACHINE_ARCH=x86_64 OPSYS=NetBSD CATMAN_SECTION_SUFFIX=no MANINSTALL=maninstall\ htmlinstall FETCH_WITH_INET6=yes FETCH_WITH_OPENSSL=yes CXXCPP=cpp CC=gcc CFLAGS=-O2\ -D_FORTIFY_SOURCE=2\ -I/usr/include CPPFLAGS=-D_LARGEFILE_SOURCE\ -D_LARGE_FILES\ -D_FILE_OFFSET_BITS=64\ -DDEF_UMASK=0022\ -I/d/repo/pkgsrc/pkgtools/pkg_install/work/libfetch\ -I/d/repo/pkgsrc/pkgtools/pkg_install/work\ -DHAVE_NBCOMPAT_H=1\ -I/d/repo/pkgsrc/pkgtools/pkg_install/work/libnbcompat\ -I/usr/include CXX=c++ CXXFLAGS=-O2\ -D_FORTIFY_SOURCE=2\ -I/usr/include COMPILER_RPATH_FLAG=-Wl,-R F77=gfortran FC=gfortran FFLAGS=-O LANG=C LC_ALL=C LC_COLLATE=C LC_CTYPE=C LC_MESSAGES=C LC_MONETARY=C LC_NUMERIC=C LC_TIME=C LDFLAGS=-lssl\ -lcrypto\ -L/d/repo/pkgsrc/pkgtools/pkg_install/work/libfetch\ -L/d/repo/pkgsrc/pkgtools/pkg_install/work/netpgp\ -L/d/repo/pkgsrc/pkgtools/pkg_install/work/libnbcompat\ -L/usr/lib\ -Wl,-R/usr/lib\ -Wl,-R/usr/pkg/lib LINKER_RPATH_FLAG=-R PATH=/d/repo/pkgsrc/pkgtools/pkg_install/work/.cwrapper/bin:/d/repo/pkgsrc/pkgtools/pkg_install/work/.buildlink/bin:/d/repo/pkgsrc/pkgtools/pkg_install/work/.tools/bin:/d/repo/pkgsrc/pkgtools/pkg_install/work/.gcc/bin:/usr/pkg/bin:/home/agc/scripts:/home/agc/bin:/usr/pkg/bin:/usr/pkg/sbin:/usr/pkg/java/openjdk7/bin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/games:/usr/local/bin:/usr/local/sbin:/usr/X11R7/bin:/usr/pkg/bin:/usr/X11R7/bin PREFIX=/usr/pkg MAKELEVEL=0 CONFIG_SITE= PKG_SYSCONFDIR=/usr/pkg/etc HOME=/d/repo/pkgsrc/pkgtools/pkg_install/work/.home CWRAPPERS_CONFIG_DIR=/d/repo/pkgsrc/pkgtools/pkg_install/work/.cwrapper/config CPP=cpp LOCALBASE=/usr/pkg X11BASE=/usr/X11R7 PKGMANDIR=man PKGINFODIR=info PKGGNUDIR=gnu/ MAKECONF=/dev/null OBJECT_FMT=ELF USETOOLS=no BSD_INSTALL_PROGRAM=/usr/bin/install\ -c\ -s\ -o\ agc\ -g\ agc\ -m\ 755 BSD_INSTALL_SCRIPT=/usr/bin/install\ -c\ -o\ agc\ -g\ agc\ -m\ 755 BSD_INSTALL_LIB=/usr/bin/install\ -c\ -o\ agc\ -g\ agc\ -m\ 755 BSD_INSTALL_DATA=/usr/bin/install\ -c\ -o\ agc\ -g\ agc\ -m\ 644 BSD_INSTALL_MAN=/usr/bin/install\ -c\ -o\ agc\ -g\ agc\ -m\ 644 BSD_INSTALL=/usr/bin/install BSD_INSTALL_PROGRAM_DIR=/usr/bin/install\ -d\ -o\ agc\ -g\ agc\ -m\ 755 BSD_INSTALL_SCRIPT_DIR=/usr/bin/install\ -d\ -o\ agc\ -g\ agc\ -m\ 755 BSD_INSTALL_LIB_DIR=/usr/bin/install\ -d\ -o\ agc\ -g\ agc\ -m\ 755 BSD_INSTALL_DATA_DIR=/usr/bin/install\ -d\ -o\ agc\ -g\ agc\ -m\ 755 BSD_INSTALL_MAN_DIR=/usr/bin/install\ -d\ -o\ agc\ -g\ agc\ -m\ 755 BSD_INSTALL_GAME=/usr/bin/install\ -c\ -s\ -o\ agc\ -g\ agc\ -m\ 2555 BSD_INSTALL_GAME_DATA=/usr/bin/install\ -c\ -o\ agc\ -g\ agc\ -m\ 664 BSD_INSTALL_GAME_DIR=/usr/bin/install\ -d\ -o\ agc\ -g\ agc\ -m\ 775 INSTALL_INFO= MAKEINFO=/d/repo/pkgsrc/pkgtools/pkg_install/work/.tools/bin/makeinfo FLEX= BISON= PKG_CONFIG= PKG_CONFIG_LIBDIR=/d/repo/pkgsrc/pkgtools/pkg_install/work/.buildlink/lib/pkgconfig:/d/repo/pkgsrc/pkgtools/pkg_install/work/.buildlink/share/pkgconfig PKG_CONFIG_LOG=/d/repo/pkgsrc/pkgtools/pkg_install/work/.pkg-config.log PKG_CONFIG_PATH= CWRAPPERS_CONFIG_DIR=/d/repo/pkgsrc/pkgtools/pkg_install/work/.cwrapper/config BINOWN=agc BINGRP=agc GAMEOWN=agc GAMEGRP=agc MANOWN=agc MANGRP=agc SHAREOWN=agc SHAREGRP=agc DOCOWN=agc DOCGRP=agc BINMODE=755 NONBINMODE=644 BINDIR=/usr/pkg/bin INCSDIR=/usr/pkg/include LIBDIR=/usr/pkg/lib MANDIR=/usr/pkg/man STRIPFLAG=-s MANINSTALL=maninstall\ htmlinstall MKCATPAGES=no MKHTML=no /usr/bin/make -f Makefile all

(note the sed command deleting the line with zlib from Makefile.lib.in)

I traced this back to pkg_install's Makefile, and was able to reproduce the problem, deleted the offending lines (with netpgp using its own prefix-symbol-protected functions)

--- Makefile 26 Jan 2020 17:32:00 -0000 1.227
+++ Makefile 4 May 2020 18:45:09 -0000
@@ -211,9 +211,7 @@
${SETENV} ${MAKE_ENV} ${BSD_MAKE_ENV} \
${MAKE_PROGRAM} ${MAKE_FLAGS} ${BUILD_MAKE_FLAGS} \
-f ${MAKE_FILE} depend all
- cd ${WRKDIR}/netpgp && \
- ${SED} -e '/zlib/d' Makefile.lib.in >Makefile.in && \
- ${RM} -f bzlib.h zlib.h && \
+ cd ${WRKDIR}/netpgp && \
${CONFIG_SHELL} ${CONFIG_SHELL_FLAGS} ./configure && \
${SETENV} ${MAKE_ENV} ${BSD_MAKE_ENV} ${MAKE_PROGRAM} \
${MAKE_FLAGS} ${BUILD_MAKE_FLAGS} -f ${MAKE_FILE} all

which DTRT:

===> bootstrap started: Mon May 4 11:49:15 PDT 2020
===> bootstrap ended: Mon May 4 11:52:01 PDT 2020

I'll check in my fixes RSN

On Mon, 4 May 2020 at 03:47, Jonathan Perkin <jperkin%joyent.com@localhost> wrote:

* On 2020-05-04 at 00:49 BST, Alistair G. Crooks wrote:

> Module Name: pkgsrc
> Committed By: agc
> Date: Sun May 3 23:49:04 UTC 2020
>
> Modified Files:
> pkgsrc/security/netpgpverify/files: Makefile.bsd b64.c b64.h bzlib.c
> bzlib.h bzlib_private.h libverify.c rsa.c rsa.h verify.h zlib.c
> zlib.h
>
> Log Message:
> Update netpgpverify and libnetpgpverify to version 20200503
>
> ensure all exported functions use a unique prfix, so that they don't
> conflict with symbols (both data and text) in libcrypto. this works for
> statically linked binaries and libraries, rather then the version map which
> only works for dynalically-linked.

Hi Al,

Thanks for the update, however this causes bootstrap problems for me.

This log happens to be from my macOS daily build but I assume the same
issue will be seen elsewhere too:

--- pkg_add ---
clang -L/tmp/pkgsrc-macos15-trunk-x86_64/wrk/pkgtools/pkg_install/work/libarchive/.libs -L/tmp/pkgsrc-macos15-trunk-x86_64/wrk/pkgtools/pkg_install/work/libfetch -L/tmp/pkgsrc-macos15-trunk-x86_64/wrk/pkgtools/pkg_install/work/netpgp -L/tmp/pkgsrc-macos15-trunk-x86_64/wrk/pkgtools/pkg_install/work/libnbcompat -Wl,-search_paths_first -L/opt/pkg/lib -L../lib -o pkg_add main.o perform.o -linstall -lnetpgpverify -larchive -lbz2 -lz -lfetch -lbz2 -lz -lnbcompat -lnbcompat
Undefined symbols for architecture x86_64:
"_netpgpv_BZ2_bzDecompress", referenced from:
_read_all_packets in libnetpgpverify.a(libverify.o)
"_netpgpv_BZ2_bzDecompressInit", referenced from:
_read_all_packets in libnetpgpverify.a(libverify.o)
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
*** [pkg_add] Error code 1

bmake[1]: stopped in /tmp/pkgsrc-macos15-trunk-x86_64/wrk/pkgtools/pkg_install/work/pkg_install-20191008/add
1 error

Cheers,

--
Jonathan Perkin - Joyent, Inc. - www.joyent.com