pkgsrc-Changes-HG archive
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index][Old Index]
[pkgsrc/trunk]: pkgsrc/doc/guide/files Moved the description of bulk builds i...
details: https://anonhg.NetBSD.org/pkgsrc/rev/69afa3b132dc
branches: trunk
changeset: 533390:69afa3b132dc
user: rillig <rillig%pkgsrc.org@localhost>
date: Tue Sep 18 08:17:21 2007 +0000
description:
Moved the description of bulk builds into their own chapter. A new
section covering the pbulk system will be added soon.
diffstat:
doc/guide/files/Makefile | 3 +-
doc/guide/files/binary.xml | 588 +-------------------------------------
doc/guide/files/bulk.xml | 678 +++++++++++++++++++++++++++++++++++++++++++
doc/guide/files/chapters.ent | 3 +-
doc/guide/files/pkgsrc.xml | 5 +-
5 files changed, 686 insertions(+), 591 deletions(-)
diffs (truncated from 1346 to 300 lines):
diff -r 272039d054ae -r 69afa3b132dc doc/guide/files/Makefile
--- a/doc/guide/files/Makefile Tue Sep 18 06:40:36 2007 +0000
+++ b/doc/guide/files/Makefile Tue Sep 18 08:17:21 2007 +0000
@@ -1,4 +1,4 @@
-# $NetBSD: Makefile,v 1.13 2006/12/15 13:22:14 martti Exp $
+# $NetBSD: Makefile,v 1.14 2007/09/18 08:17:21 rillig Exp $
WEB_PREFIX?= ${.CURDIR}/../htdocs
@@ -8,6 +8,7 @@
SRCS+= binary.xml
SRCS+= build.xml
SRCS+= buildlink.xml
+SRCS+= bulk.xml
SRCS+= creating.xml
SRCS+= components.xml
SRCS+= configuring.xml
diff -r 272039d054ae -r 69afa3b132dc doc/guide/files/binary.xml
--- a/doc/guide/files/binary.xml Tue Sep 18 06:40:36 2007 +0000
+++ b/doc/guide/files/binary.xml Tue Sep 18 08:17:21 2007 +0000
@@ -1,4 +1,4 @@
-<!-- $NetBSD: binary.xml,v 1.33 2007/08/15 06:33:43 rillig Exp $ -->
+<!-- $NetBSD: binary.xml,v 1.34 2007/09/18 08:17:21 rillig Exp $ -->
<chapter id="binary">
<title>Creating binary packages</title>
@@ -40,590 +40,4 @@
<para>See <xref linkend="build.helpful-targets"/>.</para>
</sect1>
-
- <sect1 id="bulkbuild">
- <title>Doing a bulk build of all packages</title>
-
- <para>If you want to get a full set of precompiled binary
- packages, this section describes how to get them. Beware that
- the bulk build will remove all currently installed packages from
- your system!</para>
-
- <para>Having an FTP server configured either on the
- machine doing the bulk builds or on a nearby NFS server can help
- to make the packages available to other machines that can then
- save time by installing only the binary packages. See &man.ftpd.8; for
- more information. If you use a remote NFS server's storage, be
- sure to not actually compile on NFS storage, as this slows
- things down a lot.</para>
-
- <sect2 id="binary.configuration">
- <title>Configuration</title>
-
- <sect3 id="binary.bulk.build.conf">
- <title><filename>build.conf</filename></title>
-
- <para>The <filename>build.conf</filename> file is the main
- configuration file for bulk builds. You can configure how your
- copy of pkgsrc is kept up to date, how the distfiles are
- downloaded, how the packages are built and how the report is
- generated. You can find an annotated example file in
- <filename>pkgsrc/mk/bulk/build.conf-example</filename>. To use
- it, copy <filename>build.conf-example</filename> to
- <filename>build.conf</filename> and edit it, following the
- comments in that file.</para>
- </sect3>
-
- <sect3 id="binary.mk.conf">
- <title>&mk.conf;</title>
-
- <para>You may want to set variables in &mk.conf;.
- Look at <filename>pkgsrc/mk/defaults/mk.conf</filename> for
- details of the default settings. You will want to ensure that
- <varname>ACCEPTABLE_LICENSES</varname> meet your local policy.
- As used in this example, <varname>_ACCEPTABLE=yes</varname>
- completely bypasses the license check.</para>
-
-<programlisting>
-PACKAGES?= ${_PKGSRCDIR}/packages/${MACHINE_ARCH}
-WRKOBJDIR?= /usr/tmp/pkgsrc # build here instead of in pkgsrc
-BSDSRCDIR= /usr/src
-BSDXSRCDIR= /usr/xsrc # for x11/xservers
-OBJHOSTNAME?= yes # use work.`hostname`
-FAILOVER_FETCH= yes # insist on the correct checksum
-PKG_DEVELOPER?= yes
-_ACCEPTABLE= yes
-</programlisting>
-
- <para>Some options that are especially useful for bulk builds
- can be found at the top lines of the file
- <filename>mk/bulk/bsd.bulk-pkg.mk</filename>. The most useful
- options of these are briefly described here.</para>
-
- <itemizedlist>
-
- <listitem><para>If you are on a slow machine, you may want to
- set <varname>USE_BULK_BROKEN_CHECK</varname> to
- <quote>no</quote>.</para></listitem>
-
- <listitem><para>If you are doing bulk builds from a read-only
- copy of pkgsrc, you have to set <varname>BULKFILESDIR</varname>
- to the directory where all log files are created. Otherwise the
- log files are created in the pkgsrc directory.</para></listitem>
-
- <listitem><para>Another important variable is
- <varname>BULK_PREREQ</varname>, which is a list of packages that
- should be always available while building other
- packages.</para></listitem>
-
- </itemizedlist>
-
- <para>Some other options are scattered in the pkgsrc
- infrastructure:</para>
-
- <itemizedlist>
-
- <listitem><para><varname>ALLOW_VULNERABLE_PACKAGES</varname>
- should be set to <literal>yes</literal>. The purpose of the bulk
- builds is creating binary packages, no matter if they are
- vulnerable or not. When uploading the packages to a public
- server, the vulnerable packages will be put into a directory of
- their own. Leaving this variable unset would prevent the bulk
- build system from even trying to build them, so possible
- building errors would not show up.</para></listitem>
-
- <listitem><para><varname>CHECK_FILES</varname>
- (<filename>pkgsrc/mk/check/check-files.mk</filename>) can be set to
- <quote>yes</quote> to check that the installed set of files
- matches the <filename>PLIST</filename>.</para></listitem>
-
- <listitem><para><varname>CHECK_INTERPRETER</varname>
- (<filename>pkgsrc/mk/check/check-interpreter.mk</filename>) can be set to
- <quote>yes</quote> to check that the installed
- <quote>#!</quote>-scripts will find their
- interpreter.</para></listitem>
-
- <listitem><para><varname>PKGSRC_RUN_TEST</varname> can be
- set to <quote><literal>yes</literal></quote> to run each
- package's self-test before installing it. Note that some
- packages make heavy use of <quote>good</quote> random
- numbers, so you need to assure that the machine on which you
- are doing the bulk builds is not completely idle. Otherwise
- some test programs will seem to hang, while they are just
- waiting for new random data to be
- available.</para></listitem>
-
- </itemizedlist>
-
- </sect3>
-
- <sect3 id="pre-build.local">
- <title><filename>pre-build.local</filename></title>
-
- <para>It is possible to configure the bulk build to perform
- certain site-specific tasks at the end of the pre-build
- stage. If the file
- <filename>pre-build.local</filename> exists in
- <filename>/usr/pkgsrc/mk/bulk</filename>, it will be executed
- (as a &man.sh.1; script) at the end of the usual pre-build
- stage. An example use of
- <filename>pre-build.local</filename> is to have the line:</para>
-
- <screen>echo "I do not have enough disk space to build this pig." \
- > misc/openoffice/$BROKENF</screen>
-
- <para>to prevent the system from trying to build a particular package
- which requires nearly 3 GB of disk space.</para>
- </sect3>
- </sect2>
-
- <sect2 id="other-environmental-considerations">
- <title>Other environmental considerations</title>
-
- <para>As <filename>/usr/pkg</filename> will be completely
- deleted at the start of bulk builds, make sure your login
- shell is placed somewhere else. Either drop it into
- <filename>/usr/local/bin</filename> (and adjust your login
- shell in the passwd file), or (re-)install it via
- &man.pkg.add.1; from <filename>/etc/rc.local</filename>, so
- you can login after a reboot (remember that your current
- process won't die if the package is removed, you just can't
- start any new instances of the shell any more). Also, if you
- use &os; earlier than 1.5, or you still want to use the pkgsrc
- version of ssh for some reason, be sure to install ssh before
- starting it from <filename>rc.local</filename>:</para>
-
-<programlisting>
-(cd /usr/pkgsrc/security/ssh && make bulk-install)
-if [ -f /usr/pkg/etc/rc.d/sshd ]; then
- /usr/pkg/etc/rc.d/sshd
-fi
-</programlisting>
-
- <para>Not doing so will result in you being not able to log in
- via ssh after the bulk build is finished or if the machine
- gets rebooted or crashes. You have been warned! :)</para>
- </sect2>
-
- <sect2 id="operation">
- <title>Operation</title>
-
- <para>Make sure you don't need any of the packages still
- installed.</para>
-
- <warning>
- <para>During the bulk build, <emphasis>all packages, their
- configuration files and some more files from
- <filename>/var</filename>, <filename>/home</filename> and
- possibly other locations will be removed! So don't run a bulk
- build with privileges that might harm your
- system.</emphasis></para>
- </warning>
-
- <para>Be sure to remove all other things that might
- interfere with builds, like some libs installed in
- <filename>/usr/local</filename>, etc. then become root and type:</para>
-
- <screen>
-&rprompt; <userinput>cd /usr/pkgsrc</userinput>
-&rprompt; <userinput>sh mk/bulk/build</userinput>
- </screen>
-
- <para>If for some reason your last build didn't complete (power
- failure, system panic, ...), you can continue it by
- running:</para>
-
- <screen>&rprompt; <userinput>sh mk/bulk/build restart</userinput></screen>
-
- <para>At the end of the bulk build, you will get a summary via mail,
- and find build logs in the directory specified by
- <varname>FTP</varname> in the <filename>build.conf</filename>
- file.</para>
- </sect2>
-
- <sect2 id="what-it-does">
- <title>What it does</title>
-
- <para>The bulk builds consist of three steps:</para>
-
- <variablelist>
- <varlistentry>
- <term>1. pre-build</term>
-
- <listitem>
- <para>The script updates your pkgsrc tree via (anon)cvs, then
- cleans out any broken distfiles, and removes all
- packages installed.</para>
- </listitem>
- </varlistentry>
-
- <varlistentry>
- <term>2. the bulk build</term>
-
- <listitem>
- <para>This is basically <quote>make bulk-package</quote> with
- an optimised order in which packages will be
- built. Packages that don't require other packages will
- be built first, and packages with many dependencies will
- be built later.</para>
- </listitem>
- </varlistentry>
-
- <varlistentry>
- <term>3. post-build</term>
-
- <listitem>
- <para>Generates a report that's placed in the directory
- specified in the <filename>build.conf</filename> file
- named <filename>broken.html</filename>, a short version
- of that report will also be mailed to the build's
- admin.</para>
- </listitem>
- </varlistentry>
- </variablelist>
-
- <para>During the build, a list of broken packages will be compiled
- in <filename>/usr/pkgsrc/.broken</filename> (or
- <filename>.../.broken.${MACHINE}</filename> if
- <varname>OBJMACHINE</varname> is set), individual build logs
- of broken builds can be found in the package's
- directory. These files are used by the bulk-targets to mark
- broken builds to not waste time trying to rebuild them, and
- they can be used to debug these broken package builds
- later.</para>
- </sect2>
-
- <sect2 id="disk-space-requirements">
- <title>Disk space requirements</title>
-
- <para>Currently, roughly the following requirements are valid for
- NetBSD 2.0/i386:</para>
-
- <itemizedlist>
- <listitem>
- <para>10 GB - distfiles (NFS ok)</para>
- </listitem>
-
- <listitem>
- <para>8 GB - full set of all binaries (NFS ok)</para>
- </listitem>
-
- <listitem>
- <para>5 GB - temp space for compiling (local disk recommended)</para>
Home |
Main Index |
Thread Index |
Old Index