Subject: Re: Packages
To: None <Hubert.Feyrer@rz.uni-regensburg.de>
From: Jarle Fredrik Greipsland <jarle@idt.unit.no>
List: current-users
Date: 01/05/1996 03:32:05
In article <9601041606.ZM3662@rrzc1a>, Hubert Feyrer <Hubert.Feyrer@rz.uni-regensburg.de> writes:
>> 1. Links in /usr/local/bin, /usr/local/man, etc. 
> Yeah. That's what EazyInstall  is for ;)
[ ... ]
> I for myself, would also like to have a concept that includes other
> architectures than just NetBSD, too. 
Yeah.  That's what "store" is for ;) ;)

(Before the flames start: No, I'm not suggesting that what I descibe
below be incorporated into NetBSD, or that the packages should be
build according to these conventions.  Please look at it as an
informational description of a software managment system only.  I hope
someone may find it useful.)

If you've bit the symlink bullet already, "store" may interest you,
either as a working system or at least as a source of ideas.  "store"
is a system for handling/maintaining third party software, and
although not a requirement we use it mostly for packages where we've
got source code available.  See
http://www.pvv.unit.no/~arnej/store/storedoc.html for a more
complete description of the system.

_Very_ brief though, an application resides in a "master store".  The
master store may have several versions of an application, and each
version is kept in a separate sub-directory.

An example, lynx:
% ls /store/store/ernie/lynx
registration      src-2.3.7-local/  summary.2.3       ver-2.3/
src-2.3/          src-2.4.2/        summary.2.3.7     ver-2.3.7/
src-2.3.7/        src-2.4.2-local/  summary.2.4.2     ver-2.4.2/

We've got sources for a few versions stashed away, and the -local
trees are just shadowing the real source directory, modulo local
changes.  The registration file describes the application, and
contains miscellaneous info, e.g. what systems each version is
available for and what "release level" we've tagged each version with
(dated, stable, release quality, beta software etc.).  For lynx the
supported platforms are:
: 2.3 / 386netbsd/dsult4/linuxaout/rs6000/sgi4i5/sun4os4/sun4os5
: 2.3.7 / hp700ux9/linuxelf/m68k4knetbsd/sun4os5
: 2.4.2 / 386netbsd/m68k4knetbsd/rs6aix4/sparc/vaxnetbsd

% ls /store/store/ernie/lynx/ver-2.4.2
bin/    diffs/  doc/    lib/    man/

Each version tree uses the conventional directory layout, and we keep
a set of any locally made diffs.  These make up our collective memory
when it comes to the "what did the guy that has now left do last year
with the previous version of this package to get it to compile on this
weird OS?"-problems.

Now, how's the various OS/machine architectures handled?  There's an
hierarchical architecture concept, starting with architecture
independence.  Files in the store that are applicable to only a subset
of architectures are tagged with a @-suffix as in

% ls /store/store/ernie/lynx/ver-2.4.2/bin
lynx@386netbsd*     lynx@rs6aix4*       lynx@sun4os4*       lynx@vaxnetbsd*
lynx@m68k4knetbsd*  lynx@sparcnetbsd*   lynx@sun4os5*

E.g. @litend and @bigend are applicable to endian sensitive data
files.  Also, files applicable to only specific network domains can
likewise be tagged with suitable @-suffixes, so that if needed we
could have /store/store/ernie/lynx/ver-2.4.2/lib/lynx have the files
lynx.cfg@domain:idt.unit.no and lynx.cfg@domain:itea.unit.no to make
lynx start with domain specific initial html documents.

The users see the applications in store through the /store linktree.
A linktree is a view into the store for a given OS/HW-combination,
network domain and a release level, e.g. 386netbsd, idt.unit.no,
newer.  Thus, this linktree will link up against all architecture
neutral files, the files marked with @386netbsd (or @litend) or
@domain:idt.unit.no and it will link up against the newest versions it
can find in the store.  Another linktree may specify sun4os4,
itea.unit.no, dated, which will choose files for the obvious
arch/domain combination and choose a more conservative version picking
scheme.

% ls -l /store/bin/lynx
lrwxr-xr-x  1 store  store  59 Jan  3 10:37 /store/bin/lynx@ -> /store/store/others/ernie/lynx/ver-2.4.2/bin/lynx@386netbsd

The store is exported read only to the world, and is NFS automounted
with some fairly simple amd maps.  In order to not have all computers
on campus pound on a single master store slave stores also exist.
They're stores without the sources, and (possibly) subsets of
architecures/domains/run levels stored in them.  Linktrees may link
against them though.  One may even specify metrics for which stores a
linktree should preferrably link against.

The store system is administered with a set of perl scripts and cron
jobs.

Now, to compile an application for a new platform one would do something
like the following:
% shadow -a lynx-c -m ernie -s dvask
Makes a compilation shadow tree for application lynx from master ernie
to compilestore dvask.
Then carefully apply any platform specific diffs, configure the app in
question to live in /store/{bin,lib,man,share} etc.  Build the
application and install it.  Note that this will install it into the
linktree.

% postinst -a lynx-c -s dvask
Pulls the recently installed application out from the link tree, moves
it to a separate directory tree, automatically tags the
architecture dependent files with the proper suffix, and generates a
tar-file consisting of these files.  Untar this file into the proper
version directory at the master store.  Run a couple of administrative
scripts to update the registration file (and some summary files as
well.).

% linkup -a lynx -s ernie -l dvask
Creates or updates the needed symlinks, so that /store/bin/lynx,
/store/man/man1/lynx.1 etc. points to the proper version of lynx.
Alternatively, one could just wait for the nightly cron jobs to
propagate the version to the stores that want it, and it would
automatically get linked up into the proper linktrees.

This recipe works fine for almost every application we've encountered.
X and GNU-applications are generally no problem at all.  For some of
these one may even skip the "install into linktree" part, and install
them directly into a separate version directory tree.  Then use
postinst to just tag the files in place, and then tar them up.

Purists may find some of the mechanisms used in store ugly and evil,
but as store should run on as many (unix) platforms as possible one
has had to mostly use commonly available ones.  As far as I know store
has been used for the following HW platforms:
  i386 based PC, Alpha, Sun3, Sun4, Motorola 88k, MIPS 2k/3k (pmax),
  MIPS 4k (misc SGI boxes), HP-PA, Apollo's 88k series, vax, RS/6000,
  and hp300.
and for the following OSes:
  386BSD, NetBSD, FreeBSD, Linux, SunOS 4 and 5, OSF/1, Ultrix, Irix 4
  and 5, SR10 on Apollos, HP-UX 7 to 10, AIX 3 and 4, and DolphinOS.

As Arne Juul pointed out some time last year on current-users, our
main "master store" can be reached at

	ftp://ftp.pvv.unit.no/store/store/ernie.  

(Note that the directory contains a few hundred subdirectories, so
doing "dir" may take some time.)  Anyone interested in taking a closer
look at some of the stuff there is welcome to do so.

					-jarle
-- 
"As far as I'm concerned,  if something is so complicated that you can't
 explain it in 10 seconds, then it's probably not worth knowing anyway"
					-- Calvin