tech-pkg archive
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index][Old Index]
pkgsrc cross-compilation
The other day I took a whack at making pkgsrc cross-compile packages
from one CPU architecture to another under NetBSD, from a host of
NetBSD/amd64 to a target of NetBSD/powerpc, following the notes in
doc/HOWTO-crosscompile-xorg. It kinda worked, and with some hacking I
was able to get it to work a little more smoothly. Please let me know
if you have any thoughts on any of my changes or my narrative below.
Summary of changes I think we'll need if we want to make
cross-compilation a first-class citizen:
- When recursively building build and bootstrap dependencies (i.e.,
not full dependencies) of a cross-compiled package, build them with
USE_CROSS_COMPILE=no TARGET_ARCH=${MACHINE_ARCH}.
- Automatically set WRKOBJDIR and PACKAGES differently if
USE_CROSS_COMPILE=yes, e.g. to ${DEFAULT_WRKOBJDIR}.${MACHINE_ARCH}.
- Rationalize the NATIVE_FOO, FOO, and TARGET_FOO variables
(MACHINE_ARCH, MACHINE_GNU_PLATFORM, &c.) so it's clear how to
interpret them (is TARGET_ARCH a valid MACHINE_ARCH value, a GNU
platform, or something like i386-linux which is neither?) and who is
supposed to set them (package makefiles? pkgsrc mk internals? user
on command line / mk.conf?).
We more or less have a scheme of (a) NATIVE_FOO is set if
USE_CROSS_COMPILE=yes to the platform on which we are building a
package, (b) FOO is set to the platform that the package is being
built for, and (c) TARGET_FOO may be set if USE_CROSS_COMPILE=no to
the platform that we're building host tools to work with (as in the
cross/ category).
- When building a host package configured for a cross-compilation
target (e.g., bmake install TARGET_ARCH=powerpc), and the package is
something like libtool which needs to know about other tools like cc
for the same target, we need some way to tell the package where the
other tools are going to be found -- maybe something like
TARGET_TOOLS_PATH.cc &c.
- When cross-compiling (e.g., bmake package USE_CROSS_COMPILE=yes
MACHINE_ARCH=powerpc), and we're building a package that wants to
compile and run its own tools (e.g., devel/ncurses), we need to
provide access to the host tools to build those -- maybe something
like NATIVE_TOOLS_PATH.cc &c.
Narrative:
The procedure to cross-compile a package is basically
cd cat/pkg && \
bmake package USE_CROSS_COMPILE=yes MACHINE_ARCH=powerpc \
CROSS_DESTDIR=/usr/obj/destdir.evbppc \
TOOLDIR=/usr/obj/tooldir.NetBSD-6.99.16-amd64
given a NetBSD tooldir and destdir populated by `build.sh tools' and
`build.sh distribution'. However, this likely won't quite work out of
the box.
Currently bootstrap and build dependencies are built for the target,
not for the host, and there are no separate $WRKOBJDIR or $PACKAGES
for host and target. This turns out to work accidentally for libtool,
but I'm pretty sure it's a non-starter for just about anything else.
To make bootstrap and build dependencies work for packages other than
libtool, I set WRKOBJDIR and PACKAGES conditionally in mk.conf,
.if !empty(USE_CROSS_COMPILE:M[yY][eE][sS])
PACKAGES= /path/to/packages.${MACHINE_ARCH}
WRKOBJDIR= /tmp/pkgsrc/work.${MACHINE_ARCH}
.else
PACKAGES= /path/to/packages
WRKOBJDIR= /tmp/pkgsrc/work
.endif
and I applied the attached patch build-host.patch to build bootstrap
and build dependencies using the host tools with USE_CROSS_COMPILE=no
rather than cross-compiling with the target tools. Full dependencies
are still cross-compiled with the target tools, of course. This
worked to build a few packages, such as net/unbound.
To get libtool, I
- copied devel/libtool-base to cross/libtool-base;
- tweaked it a bit to set PKGNAME=cross-libtool-base-${MACHINE_ARCH}
and USE_CROSSBASE=yes;
- applied the attached patch cross-libtool.patch so that if
USE_CROSS_COMPILE=yes and USE_LIBTOOL=yes, LIBTOOL will be set to
${CROSSBASE}/bin/libtool and we'll have a build dependency on
cross-libtool-base-${MACHINE_ARCH};
- cross-compiled cross/libtool-base; and finally
- manually installed it on the host using `pkg_add -m powerpc'.
This process, especially the last step, is mega-kludgerific, but
- it works accidentally because libtool is a shell script, so it runs
anywhere, and
- it appears to be currently necessary, because libtool needs to know
about the tools that we would use for USE_CROSS_COMPILE=yes, and
it's not clear to me how to get at them when USE_CROSS_COMPILE=no.
Merely building libtool with `./configure --target=powerpc--netbsd'
isn't enough -- libtool doesn't even seem to pay attention to --target
at all, and seems to have badly conflated the GNU --host and --target
options.
With this libtool, I've been able to build a few more packages, with a
tweak here and there to avoid GNU configure scripts relying on
AC_TRY_RUN, &c. There are some major dependencies that will require
more work, though: devel/ncurses, for instance, wants to compile and
run a tool of its own, and who knows what it will take to
cross-compile Perl.
Thoughts?
Index: mk/pkgformat/pkg/depends.mk
===================================================================
RCS file: /cvsroot/pkgsrc/mk/pkgformat/pkg/depends.mk,v
retrieving revision 1.2
diff -p -u -r1.2 depends.mk
--- mk/pkgformat/pkg/depends.mk 2 Jul 2012 14:53:13 -0000 1.2
+++ mk/pkgformat/pkg/depends.mk 26 Mar 2013 13:40:04 -0000
@@ -64,6 +64,7 @@ _LIST_DEPENDS_CMD.bootstrap= \
_RESOLVE_DEPENDS_CMD= \
${PKGSRC_SETENV} _PKG_DBDIR=${_PKG_DBDIR:Q} PKG_INFO=${PKG_INFO:Q} \
+ HOST_PKG_INFO=${HOST_PKG_INFO:Q} \
_DEPENDS_FILE=${_DEPENDS_FILE:Q} \
${SH} ${PKGSRCDIR}/mk/pkgformat/pkg/resolve-dependencies \
" "${BOOTSTRAP_DEPENDS:Q} \
@@ -80,8 +81,23 @@ _RESOLVE_DEPENDS_CMD= \
# build, full.
#
_DEPENDS_INSTALL_CMD= \
- pkg=`${_PKG_BEST_EXISTS} "$$pattern" || ${TRUE}`; \
+ case $$type in \
+ bootstrap|build) \
+ pkg=`${_HOST_PKG_BEST_EXISTS} "$$pattern" || ${TRUE}`;; \
+ full) \
+ pkg=`${_PKG_BEST_EXISTS} "$$pattern" || ${TRUE}`;; \
+ esac; \
case $$type in bootstrap) Type=Bootstrap;; build) Type=Build;; full)
Type=Full;; esac; \
+ case $$type in \
+ bootstrap|build) \
+ cross=no; \
+ archopt=TARGET_ARCH=${MACHINE_ARCH}; \
+ ;; \
+ full) \
+ cross=${USE_CROSS_COMPILE:Uno}; \
+ archopt=; \
+ ;; \
+ esac; \
case "$$pkg" in \
"") \
${STEP_MSG} "$$Type dependency $$pattern: NOT found"; \
@@ -89,8 +105,18 @@ _DEPENDS_INSTALL_CMD=
\
${STEP_MSG} "Verifying $$target for $$dir"; \
[ -d "$$dir" ] || ${FAIL_MSG} "[depends.mk] The directory
\`\`$$dir'' does not exist."; \
cd $$dir; \
- ${PKGSRC_SETENV} ${PKGSRC_MAKE_ENV} _PKGSRC_DEPS="
${PKGNAME}${_PKGSRC_DEPS}" PKGNAME_REQD="$$pattern" ${MAKE} ${MAKEFLAGS}
_AUTOMATIC=yes $$target; \
- pkg=`${_PKG_BEST_EXISTS} "$$pattern" || ${TRUE}`; \
+ ${PKGSRC_SETENV} ${PKGSRC_MAKE_ENV} \
+ _PKGSRC_DEPS=" ${PKGNAME}${_PKGSRC_DEPS}" \
+ PKGNAME_REQD="$$pattern" \
+ USE_CROSS_COMPILE=$$cross \
+ $$archopt \
+ ${MAKE} ${MAKEFLAGS} _AUTOMATIC=yes $$target; \
+ case $$type in \
+ bootstrap|build) \
+ pkg=`${_HOST_PKG_BEST_EXISTS} "$$pattern" || ${TRUE}`;;
\
+ full) \
+ pkg=`${_PKG_BEST_EXISTS} "$$pattern" || ${TRUE}`;; \
+ esac; \
case "$$pkg" in \
"") ${ERROR_MSG} "[depends.mk] A package matching
\`\`$$pattern'' should"; \
${ERROR_MSG} " be installed, but one cannot be
found. Perhaps there is a"; \
@@ -100,7 +126,12 @@ _DEPENDS_INSTALL_CMD=
\
${STEP_MSG} "Returning to build of ${PKGNAME}"; \
;; \
*) \
- objfmt=`${PKG_INFO} -Q OBJECT_FMT "$$pkg"`; \
+ case $$type in \
+ bootstrap|build) \
+ objfmt=`${HOST_PKG_INFO} -Q OBJECT_FMT "$$pkg"`;; \
+ full) \
+ objfmt=`${PKG_INFO} -Q OBJECT_FMT "$$pkg"`;; \
+ esac; \
case "$$objfmt" in \
"") ${WARNING_MSG} "[depends.mk] Unknown object format for
installed package $$pkg" ;; \
${OBJECT_FMT}) ;; \
Index: mk/pkgformat/pkg/pkgformat-vars.mk
===================================================================
RCS file: /cvsroot/pkgsrc/mk/pkgformat/pkg/pkgformat-vars.mk,v
retrieving revision 1.2
diff -p -u -r1.2 pkgformat-vars.mk
--- mk/pkgformat/pkg/pkgformat-vars.mk 13 Dec 2011 16:35:48 -0000 1.2
+++ mk/pkgformat/pkg/pkgformat-vars.mk 26 Mar 2013 13:40:04 -0000
@@ -23,8 +23,10 @@ PKG_DBDIR?= /var/db/pkg
#
.if ${PKG_INSTALLATION_TYPE} == "overwrite"
_PKG_DBDIR= ${_CROSS_DESTDIR}${PKG_DBDIR}
+_HOST_PKG_DBDIR= ${PKG_DBDIR}
.elif ${PKG_INSTALLATION_TYPE} == "pkgviews"
_PKG_DBDIR= ${_CROSS_DESTDIR}${DEPOTBASE}
+_HOST_PKG_DBDIR= ${DEPOTBASE}
.endif
PKG_ADD_CMD?= ${PKG_TOOLS_BIN}/pkg_add
@@ -67,12 +69,14 @@ _AUDIT_CONFIG_OPTION= IGNORE_URL
# correct package database directory.
#
PKGTOOLS_ARGS?= -K ${_PKG_DBDIR}
+HOST_PKGTOOLS_ARGS?= -K ${_HOST_PKG_DBDIR}
# Views are rooted in ${LOCALBASE}, all packages are depoted in
# ${DEPOTBASE}, and the package database directory for the default view
# is in ${PKG_DBDIR}.
#
-PKG_VIEW_ARGS?= -W ${LOCALBASE} -d ${DEPOTBASE} -k ${PKG_DBDIR}
+PKG_VIEW_ARGS?= -W ${LOCALBASE} -d ${DEPOTBASE} -k
${_CROSS_DESTDIR}${PKG_DBDIR}
+HOST_PKG_VIEW_ARGS?= -W ${LOCALBASE} -d ${DEPOTBASE} -k ${PKG_DBDIR}
PKG_ADD?= ${PKG_ADD_CMD} ${PKGTOOLS_ARGS}
PKG_ADMIN?= ${PKG_ADMIN_CMD} ${PKGTOOLS_ARGS}
@@ -82,12 +86,21 @@ PKG_INFO?= ${PKG_INFO_CMD} ${PKGTOOLS_AR
PKG_VIEW?= ${PKG_VIEW_CMD} ${PKG_VIEW_ARGS}
LINKFARM?= ${LINKFARM_CMD}
+HOST_PKG_ADD?= ${PKG_ADD_CMD} ${HOST-PKGTOOLS_ARGS}
+HOST_PKG_ADMIN?= ${PKG_ADMIN_CMD} ${HOST_PKGTOOLS_ARGS}
+HOST_PKG_CREATE?= ${PKG_CREATE_CMD} ${HOST_PKGTOOLS_ARGS}
+HOST_PKG_DELETE?= ${PKG_DELETE_CMD} ${HOST_PKGTOOLS_ARGS}
+HOST_PKG_INFO?= ${PKG_INFO_CMD} ${HOST_PKGTOOLS_ARGS}
+HOST_PKG_VIEW?= ${PKG_VIEW_CMD} ${HOST_PKG_VIEW_ARGS}
+HOST_LINKFARM?= ${LINKFARM_CMD}
+
# "${_PKG_BEST_EXISTS} pkgpattern" prints out the name of the installed
# package that best matches pkgpattern. Use this instead of
# "${PKG_INFO} -e pkgpattern" if the latter would return more than one
# package name.
#
_PKG_BEST_EXISTS?= ${PKG_INFO} -E
+_HOST_PKG_BEST_EXISTS?= ${HOST_PKG_INFO} -E
# XXX Leave this here until all uses of this have been purged from the
# XXX public parts of pkgsrc.
Index: mk/pkgformat/pkg/resolve-dependencies
===================================================================
RCS file: /cvsroot/pkgsrc/mk/pkgformat/pkg/resolve-dependencies,v
retrieving revision 1.1
diff -p -u -r1.1 resolve-dependencies
--- mk/pkgformat/pkg/resolve-dependencies 15 Oct 2011 00:23:09 -0000
1.1
+++ mk/pkgformat/pkg/resolve-dependencies 26 Mar 2013 13:40:04 -0000
@@ -30,11 +30,16 @@ error_msg() {
}
find_best() {
- ${PKG_INFO} -E "$1" || ${TRUE}
+ case $1 in
+ bootstrap|build)
+ ${HOST_PKG_INFO} -E "$2" || ${TRUE};;
+ full)
+ ${PKG_INFO} -E "$2" || ${TRUE};;
+ esac
}
${CAT} ${DEPENDS_FILE} | while read type pattern dir; do
- pkg=`find_best "$pattern"`
+ pkg=`find_best "$type" "$pattern"`
case "$pkg" in
"")
error_msg "[resolve-dependencies] A package matching
\`\`$pattern'' should"
Index: mk/tools/tools.NetBSD.mk
===================================================================
RCS file: /cvsroot/pkgsrc/mk/tools/tools.NetBSD.mk,v
retrieving revision 1.46
diff -p -u -r1.46 tools.NetBSD.mk
--- mk/tools/tools.NetBSD.mk 26 May 2012 13:09:54 -0000 1.46
+++ mk/tools/tools.NetBSD.mk 26 Mar 2013 13:40:04 -0000
@@ -140,4 +140,7 @@ TOOLS_CREATE+= ranlib
NATIVE_CC:= /usr/bin/cc
CC= ${TOOLDIR}/bin/${MACHINE_GNU_PLATFORM}-gcc
+NATIVE_LD:= /usr/bin/ld
+LD= ${TOOLDIR}/bin/${MACHINE_GNU_PLATFORM}-ld
+
.endif
Index: devel/libtool/Makefile.common
===================================================================
RCS file: /cvsroot/pkgsrc/devel/libtool/Makefile.common,v
retrieving revision 1.78
diff -p -u -r1.78 Makefile.common
--- devel/libtool/Makefile.common 31 Oct 2012 11:17:23 -0000 1.78
+++ devel/libtool/Makefile.common 26 Mar 2013 13:40:00 -0000
@@ -41,8 +41,8 @@ PKG_INSTALLATION_TYPES= overwrite pkgvie
GNU_CONFIGURE= yes
-DISTINFO_FILE= ${.CURDIR}/../libtool/distinfo
-PATCHDIR= ${.CURDIR}/../libtool/patches
+DISTINFO_FILE= ${.CURDIR}/../../devel/libtool/distinfo
+PATCHDIR= ${.CURDIR}/../../devel/libtool/patches
AUTOMAKE_OVERRIDE= NO
@@ -60,12 +60,15 @@ CONFIGURE_ENV+= lt_cv_path_LD=${LD:Q}
CONFIGURE_ENV+= lt_cv_path_SED=${SED:Q}
# The configure script doesn't seem to be working out the need for -p
-# or -B, so tell it
+# or -B, so tell it. For cross-compilation, force it to use the normal
+# NM and not the one in ${BUILDLINK_DIR}.
#
.if ${OPSYS} == "SunOS" || ${OPSYS} == "HPUX"
CONFIGURE_ENV+= lt_cv_path_NM="nm -p"
.elif ${OPSYS} == "IRIX" || ${OPSYS} == "OSF1"
CONFIGURE_ENV+= lt_cv_path_NM="nm -B"
+.else
+CONFIGURE_ENV+= lt_cv_path_NM=${NM:Q}
.endif
CONFIGURE_ENV+= RANLIB=${RANLIB:Q}
Index: mk/bsd.pkg.use.mk
===================================================================
RCS file: /cvsroot/pkgsrc/mk/bsd.pkg.use.mk,v
retrieving revision 1.51
diff -p -u -r1.51 bsd.pkg.use.mk
--- mk/bsd.pkg.use.mk 19 Mar 2012 12:34:14 -0000 1.51
+++ mk/bsd.pkg.use.mk 26 Mar 2013 13:40:04 -0000
@@ -90,15 +90,24 @@ BUILD_DEFS+= KERBEROS
# LIBTOOL is the publicly-readable variable that should be used by
# Makefiles to invoke the proper libtool.
#
+.if !empty(USE_CROSS_COMPILE:M[yY][eE][sS])
+PKG_LIBTOOL?= ${CROSSBASE}/bin/libtool
+PKG_SHLIBTOOL?= ${CROSSBASE}/bin/shlibtool
+.else
PKG_LIBTOOL?= ${LOCALBASE}/bin/libtool
PKG_SHLIBTOOL?= ${LOCALBASE}/bin/shlibtool
+.endif
_LIBTOOL?= ${PKG_LIBTOOL}
_SHLIBTOOL?= ${PKG_SHLIBTOOL}
LIBTOOL?= ${PKG_LIBTOOL}
SHLIBTOOL?= ${PKG_SHLIBTOOL}
.if defined(USE_LIBTOOL)
LIBTOOL_REQD?= 2.2.6bnb3
+.if !empty(USE_CROSS_COMPILE:M[yY][eE][sS])
+BUILD_DEPENDS+=
cross-libtool-base-${MACHINE_ARCH}>=${_OPSYS_LIBTOOL_REQD:U${LIBTOOL_REQD}}:../../cross/libtool-base
+.else
BUILD_DEPENDS+=
libtool-base>=${_OPSYS_LIBTOOL_REQD:U${LIBTOOL_REQD}}:../../devel/libtool-base
+.endif
CONFIGURE_ENV+= LIBTOOL="${LIBTOOL} ${LIBTOOL_FLAGS}"
MAKE_ENV+= LIBTOOL="${LIBTOOL} ${LIBTOOL_FLAGS}"
.endif
Index: mk/compiler/f2c.mk
===================================================================
RCS file: /cvsroot/pkgsrc/mk/compiler/f2c.mk,v
retrieving revision 1.15
diff -p -u -r1.15 f2c.mk
--- mk/compiler/f2c.mk 30 Jul 2010 07:58:59 -0000 1.15
+++ mk/compiler/f2c.mk 26 Mar 2013 13:40:04 -0000
@@ -90,7 +90,7 @@ PREPEND_PATH+= ${_F2C_DIR}/bin
# Dependencies:
BUILD_DEPENDS+= f2c>=20090411nb2:../../lang/f2c # translator
-.if empty(PKGPATH:Mdevel/libtool-base) # See below
+.if empty(PKGPATH:Mdevel/libtool-base) && empty(PKGPATH:Mcross/libtool-base) #
See below
. include "../../devel/libf2c/buildlink3.mk" # library
.endif
@@ -99,7 +99,7 @@ PKGSRC_MAKE_ENV+= F2C_DIR=${F2C_DIR:Q}
. endif
# libtool-base is special as it only needs f77 to extract linker flags etc.
-.if !empty(PKGPATH:Mdevel/libtool-base)
+.if !empty(PKGPATH:Mdevel/libtool-base) || !empty(PKGPATH:Mcross/libtool-base)
pre-configure: fake-f2c-libs
_WRAP_EXTRA_ARGS.FC+= -L${WRKDIR}/.f2c/lib -I${WRKDIR}/.f2c/include
Home |
Main Index |
Thread Index |
Old Index