Re: local builds break "devel/gobject-introspection" et al.

["John D. Baker" <jdbaker%mylinuxisp.com@localhost>]

On Wed, 3 Aug 2016, Greg Troxel wrote:

> "John D. Baker" <jdbaker%mylinuxisp.com@localhost> writes:
> 
> > I've been beating my head against the wall that is pkg/51266 trying to
> > figure out what's broken for me but apparently no-one else.
> 
> A few random suggestions:
> 
>   Run sysutils/memtestplus.  You have two machines, so seems unlikely.
>   But still, it's always good to do that every few months anyway.

Two OS-build hosts on-site.  One has all source/pkgsrc trees on local
RAIDframe and serves it to the other(s) via NFS.  A third OS-build host
is off-site and has its own local source/pkgsrc trees.

Two additional package-building test systems on-site, one amd64, one
i386.  They may run either from local disk or semi-diskless.

I'll see about running memtestplus on the on-site machines soon.  (It
has been my experience that 'memtestplus' built on i386 with GCC 4.5.x
or earlier is the only thing that works.  Built on amd64 or with GCC
4.8.x or later hangs or crashes.)

>   Do you have any kind of offloading on the NICs involved, on either nfs
>   client or server?   If so, disable that.

Sigh.  Yes, the systems all support offloading (wm(4) interfaces) and I
have all of it enabled.  I'll try disabling it soon-ish.

>   Get a disk and try this with a local filesystem.   So far that seems
>   to be the big difference between what others report and what you are
>   seeing.

My latest tests have been doing just this (run from local disk with
source trees on NFS).  As long as the host OS was installed/updated from
a TNF snapshot, the initial build and rebuild of all affected packages
worked just fine.  As soon as I updated to a local OS build, they broke.

In this state, the system is "poisoned" and updating the host OS to a
TNF snapshot build again does not restore ability to rebuild the affected
packages.

Of note is that even though "gobject-introspection" can't be rebuilt,
any package which uses the already-installed "gobject-instrospection"
(other than those I listed) will rebuild without problems.

At present, the amd64 test system's NFS install is from TNF snapshot
and started over building packages from "empty" "/usr/pkg" as described
above.  All affected packages have built and rebuilt successfully.  It
was recently updated to a later TNF snapshot and continues to perform
properly.

The amd64 test system's local-disk installation started with a TNF
snapshot and completely empty "/usr/pkg" and all packages in my usual
complement were built and the problem packages successfully rebuilt.
Subsequent update to a local OS build caused the problem packages to
fail.  After a further update to a later TNF snapshot, the problem
packages continue to fail being rebuilt.

The i386 test system's local-disk installation started with a local
OS build and while "gobject-introspection" built and installed successfully
the first time (from empty "/usr/pkg"), the other problem packages
failed and "gobject-introspection" itself could not be rebuilt.

Updating the i386 test system to a TNF snapshot did not improve the
situation.  Deleteing all packages (as described above) and starting
over, all problem packages then built successfully.  Following update
to a later TNF snapshot, they all rebuild successfully.

The i386 test system's NFS installation is currently a local OS build
and has not been used for testing in this most recent round.

>   Make sure you run postinstall fix, to remove stale header files.

I do this ('etcupdate', actally, which then invokes 'postinstall')
after every update.  If a "fix" is recommended, I do it.

>   you said 'everything rebuilt from scratch'.  Do you mean that after
>   you installed the OS, you marked every package rebuild=yes and ran
>   pkg_rr, or that you deleted all packages down to an empty /usr/pkg,
>   and built fresh?

I deleted all packages (except "security/sudo") and then manually pruned
out leftovers from "/usr/pkg" that were not related to "sudo".  I then
started over--first building "pkg_rolling-replace" and using that to
update "sudo".  Then, I started building my desired leaf packages which
eventually built the problem packages as dependencies.

As for the OS builds, I deleted all the target directories and did
a non-update build.  My typical 'build.sh' command line:

  ./build.sh -U -m amd64 -T /d0/build/current/tools/amd64 -O /d0/build/current/obj/amd64 -D /d0/build/current/DEST/amd64 -R /d0/build/current/REL -X ../xsrc -x -j 9 tools release kernel.gdb=KEPLER kernel.gdb=TESLA releasekernel=KEPLER releasekernel=TESLA iso-image

See below about MKDEBUG*.

>   Are you using ccache?  (If so, try without.)

No, I am not using ccache.

>   Enable debugging in pkgsrc, so that you can run the offending programs
>   under gdb and figure out what is actually wrong.  This is hard, but I
>   think it's probably going to be necessary.
> 
> The following is adpated from experience under 5 and 6, where we built
> the entire base system and packages with -g and unstripped. Yes, it was
> all twice as big, but it was really nice to just run gdb on whatever
> ailed us and have it work.
> 
> I think putting this in mk.conf should build with debugging:
> 
> CFLAGS+=		-ggdb
> CXXFLAGS+=		-ggdb
> INSTALL_UNSTRIPPED=	yes

I've been doing this for the problem packages and their immediate
dependencies.  I was not aware of the "-ggdb" form and have only been
adding "-g".  What results I've gleaned so far are recorded in pkg/51266.

I take it you are suggesting I do this globally?

> I think (but am less sure) putting this in mk.conf (and maybe pointing
> build.sh to it) will build a base system with debug libs.
> 
> MKDEBUG=		yes
> MKDEBUGLIB=		yes
> DBG=			-g

I have been building with MKDEBUG{,_LIB}=yes as a matter of course for
a while now (at least a year if not more).  I usually pass them on the
'build.sh' command line.  I only just now stopped to try and make my
build process look more like the TNF builds, which don't build the debug
sets.

I take it that passing "DBG=-g" (or setting in mk.conf) will cause the
standard kernels (GENERIC, et al.) to be built with debugging info and
leave a "netbsd.gdb" in their build directories?

-- 
|/"\ John D. Baker, KN5UKS               NetBSD     Darwin/MacOS X
|\ / jdbaker[snail]mylinuxisp[flyspeck]com    OpenBSD            FreeBSD
| X  No HTML/proprietary data in email.   BSD just sits there and works!
|/ \ GPGkeyID:  D703 4A7E 479F 63F8 D3F4  BD99 9572 8F23 E4AD 1645