Re: pbulk with manual steps

[Havard Eidnes <he%NetBSD.org@localhost>]

> On Sun, Jan 03, 2016 at 09:15:48AM +0100, Havard Eidnes wrote:
>> And ... as demonstrated, it's no longer a good idea to use the
>> pkg_install installed in /usr/pbulk/sbin as the "pbulk pkg_install",
>> such as directed by the pbulk.conf example config file, because those
>> programs default to operating on the binaries in /usr/pbulk and the
>> pkg db in /usr/pbulk/var/db/pkg, and pbulk apparently doesn't
>> (anymore?) explicitly specify -K and -p options to direct them to
>> operate on /usr/pkg and /var/db/pkg instead.
>
> They explicitly pass the database. They do not override the prefix as
> the prefix is reocrded in the package.

Then I don't understand the problem I've been seeing.
"Something" changed between 2015Q3 and 2015Q4 which caused my
earlier in-use setup to no longer work.

> I still have not seen whatever pbulk.conf you are using, so I
> can only guess about your problems.

A copy follows attached below (after I re-pointed the pkg_install
binaries away from /usr/pbulk/sbin to /usr/sbin to make the setup
work again).

>> >> I see that when mk/pbulk/pbulk.sh is invoked with no arguments, it
>> >> will create a bootstrap kit, and append a setting of bootstrapkit=
>> >> to pbulk.conf.  Whether that's actually used is for me as an
>> >> operator difficult to tell from the log file produced by bulkbuild.
>> >
>> > It is used if it is referenced by pbulk.conf. I don't remember right now
>> > that version of pkg_install is in netbsd-6 to know if a bootstrap kit is
>> > the best option even on NetBSD for that branch. It is currently not
>> > needed or useful for netbsd-7 and -current.
>>
>> My point is that if the distinction between "are you using a bootstrap
>> kit" and "are you *not* using a bootstrap kit" is significant to guide
>> the operator, its usage ought to be prominently noted in the log file.
>> It currently isn't.
>
> It is nowhere as important as you try to make it sound. If you need to
> bootstrap, it is pretty evident by the dependency tree and the expected
> build errors.

I don't think you understand quite what I'm saying.  "Operator
friendliness" ought to be an important factor.

If you need to ask (and apparently it *is* a factor in all of
this), I need to be able to answer, and the lack of logging makes
it difficult for me to answer.

Regards,

- Håvard

# $NetBSD: pbulk.conf,v 1.19 2013/01/07 14:15:35 jperkin Exp $

# Version of the configuration file. This is bumped whenever the default
# config changes to notify the administrator about updates.
#
config_version=0.54

# The URL where the build report will be made available. This is only
# used in the .txt version of the report.
#
base_url=ftp://malus.urc.uninett.no/pub/NetBSD/pkgstat/

# If yes, keep the last scan results in ${bulklog}.old and try to reuse them.
reuse_scan_results=no

# The pbulk framework can use multiple machines to build the packages.
# On a single-processor, non-distributed build, you may want to say "no"
# here.
#
master_mode=yes

#master_ip=192.168.75.10
#scan_clients="192.168.75.21 192.168.75.22 192.168.75.23 192.168.75.24"
#build_clients="192.168.75.21 192.168.75.22 192.168.75.23 192.168.75.24"

#master_port_scan=${master_ip}:2001
#master_port_build=${master_ip}:2002

# Some flags for the programs that publish the binary packages and the
# build report. If you want to disable the publishing at all, see below.
#
pkg_rsync_args="-av --delete-excluded -e ssh"
pkg_rsync_target="he@malus:~ftp/pub/NetBSD/packages/2015Q4/"
report_rsync_args="-avz --delete-excluded -e ssh"
report_rsync_target="he@malus:~ftp/pub/NetBSD/pkgstat/"
report_subject_prefix="pkgsrc"
report_recipients="pkgsrc-bulk%netbsd.org@localhost"

# An archive containing the pkgsrc binary tree after bootstrapping.
#
bootstrapkit=/usr/pkgsrc/bootstrap/bootstrap.tar.gz

# Optionally build a subset of the available packages and their dependencies.
# The file is a newline separated list of package locations (e.g. lang/perl5).
#limited_list=/limited_list

# Optionally ignore unresolvable dependencies for a full build (e.g.
# a build without limited_list set).  Default behavior is to bail out.
#
ignore_missing_dependencies=yes

# If yes, consider a package up-to-date, if the dependency list matches
# the existing binary package and the recorded RCS IDs match the pkgsrc
# tree. Otherwise, additionally require that the package is not older
# than any of the dependencies.
#
skip_age_check=no

report_graph_script_limit=512

# Account used for user-destdir builds. This account should have
# no special permissions.
#
unprivileged_user=pbulk

# If yes, run the checksum phase as unprivileged user.
# Note: requires DISTDIR to be writeable for the unprivileged user.
use_unprivileged_checksum=no

# Variables used for the optional cross-compiling of packages.
#
cross_compile=no
target_arch=vax
target_destdir=/usr/src/destdir.${target_arch}

# The directories where the various files are created.
#
bulklog=/bulklog
packages=/packages
prefix=/usr/pkg
pkgsrc=/usr/pkgsrc
pkgdb=/var/db/pkg
varbase=/var

# The following programs must NOT be inside ${prefix}
pkg_info=/usr/sbin/pkg_info
pkg_add=/usr/sbin/pkg_add
pkg_delete=/usr/sbin/pkg_delete

# The tools that are used for building the packages. If you do not want
# to publish anything at all, set rsync=: and mail=:.
#
bzip2=/usr/bin/bzip2
digest=/usr/pbulk/bin/digest
gzip="/usr/bin/gzip -nf -9"
# On non-NetBSD, this should usually point at the bmake in ${prefix}, not the
# make used to build pbulk itself.
#make=${prefix}/bin/bmake
make=/usr/pkg/bin/bmake
mail=/usr/bin/mail
neato=/usr/pbulk/bin/neato
rsync=/usr/pbulk/bin/rsync
sed=/usr/bin/sed
tar=/bin/tar

loc=${bulklog}/meta

pbuild=/usr/pbulk/bin/pbulk-build
presolve=/usr/pbulk/bin/pbulk-resolve
pscan=/usr/pbulk/bin/pbulk-scan

# When a package build fails, it is often necessary to have a look at
# the working directory or the installed files. When these options are
# set to "yes", they will be archived in the log directory.
#
keep_wrkdir=no
keep_prefix=no

pkg_up_to_date_script=/usr/pbulk/libexec/pbulk/pkg-up-to-date
pbuild_script=/usr/pbulk/libexec/pbulk/pkg-build
pbuild_start_script=/usr/pbulk/libexec/pbulk/build-client-start
pscan_prepare=/usr/pbulk/libexec/pbulk/client-clean
pscan_start_script=/usr/pbulk/libexec/pbulk/scan-client-start
report_script=/usr/pbulk/libexec/pbulk/create-report
report_html_script=/usr/pbulk/libexec/pbulk/create-report-html
report_txt_script=/usr/pbulk/libexec/pbulk/create-report-txt
report_graph_script=/usr/pbulk/libexec/pbulk/create-broken-graph
packages_script=/usr/pbulk/libexec/pbulk/compute-packages

script_phase_pre_build=/usr/pbulk/libexec/pbulk/pre-build
script_phase_build=/usr/pbulk/libexec/pbulk/build
script_phase_report=/usr/pbulk/libexec/pbulk/report
script_phase_scan=/usr/pbulk/libexec/pbulk/scan
script_phase_upload=/usr/pbulk/libexec/pbulk/upload

# Arguments are buildlog and pkgname
sync_buildlog=:
# Arguments are path to the temporary package file, the package name
# and categories.  The default handler just copies the file to ${packages}
# and creates the symlinks.
sync_package=default_sync_package
# PKG_SUFX as set in mk.conf
pkg_sufx=.tgz

#
# Overriding default settings:
master_mode=no
bootstrapkit=/mnt/packages/bootstrap.tar.gz
bulklog=/mnt/bulklog
packages=/mnt/packages
#mail=:
#rsync=:
reuse_scan_results=yes
# Don't forget to recompute dependent settings:
loc=${bulklog}/meta
# Don't forget to recompute dependent settings:
make=${prefix}/bin/bmake

# Put some limits on time consumption:
ulimit -t -S 3600
ulimit -t -H 18000
# ulimit -p 9999
ulimit -v 1024000