Subject: Re: layered software on NetBSD
To: Scott Ellis <scotte@netbsd.warped.com>
From: Bill Sommerfeld <sommerfeld@orchard.east-arlington.ma.us>
List: current-users
Date: 03/20/1997 14:39:46
   One of my personal peeves is having software go into /usr/local/
   and then when you go to uninstall/updte it have to track down all
   of the cruft in /usr/local/lib etc that the package brought with
   it.

Here's my "partial" solution to that problem (I use it to capture the
set of files installed using `make install' so that I can copy the
results to other systems I run, but it could be just as easily used to
generate file lists for packaging..

					- Bill

# This is a shell archive.  Save it in a file, remove anything before
# this line, and then unpack it by entering "sh file".  Note, it may
# create directories; files and directories will be owned by you and
# have default permissions.
#
# This archive contains:
#
#	traceinstall-0.0001/README
#	traceinstall-0.0001/Makefile
#	traceinstall-0.0001/traceinstall.1
#	traceinstall-0.0001/traceinstall.pl
#
echo x - traceinstall-0.0001/README
sed 's/^X//' >traceinstall-0.0001/README << 'END-of-traceinstall-0.0001/README'
XThis is traceinstall version 0.0001
X
XTraceinstall is a tool for building binary tarballs suitable for easy
Xinstallation from a syscall trace of a `make install' of a software
Xpackage.
X
XIt is known to work on a *small* number of packages on
XNetBSD-1.2/i386; it probably works on many others, but I haven't tried
Xit yet..
X
XIt requires the following programs:
X	perl (it's a perl script)
X	NetBSD-compatible ktrace, kdump, and support for same in the
X		kernel.
X	uname
X	pax
X	gzip
X
XCaveats and other assumptions:
X
XThis is very new code.
X
XStylistic hints on the perl code are very welcome.  I use perl for
Xeffect, not art; it could probably be a lot cleaner.
X
XIt assumes that a `make install' always reinstalls the entire package;
Xthis is true for most, but not all, software distributions.
X
XIt will be very confused if the install runs multiple programs in
Xparallel such that the CALL and NAMI events from ktrace get
Xinterleaved.
X
XIt currently requires that all files be written using absolute
Xpathnames; there is the beginnings of the hooks needed for tracking
Xthe working directories of the subprocesses, but they aren't complete
Xyet.
X
XThe three or four packages I've tried so far haven't needed this;
Xemacs probably will need this as it uses tar during the installation.
X
XTracking of renames during install has not been tested.
X
XThere are undoubtedly some file-system mutating syscalls which aren't
Xtracked correctly; this should result in the script aborting.  Be
Xprepared to fix it.
X
XSend comments, bug reports, improvements, etc., to:
X	sommerfeld@orchard.medford.ma.us
END-of-traceinstall-0.0001/README
echo x - traceinstall-0.0001/Makefile
sed 's/^X//' >traceinstall-0.0001/Makefile << 'END-of-traceinstall-0.0001/Makefile'
XMAN=	traceinstall.1
XMANDIR?=/usr/local/man/cat
XBINDIR?=/usr/local/bin
X
Xbeforeinstall:
X	install ${COPY} -o ${BINOWN} -g ${BINGRP} -m ${BINMODE} \
X		${.CURDIR}/traceinstall.pl ${DESTDIR}${BINDIR}/traceinstall
X
X.include <bsd.prog.mk>
X
Xtarball:
X	(cd ..; tar czf /tmp/traceinstall-0.0001.tar.gz traceinstall-0.0001/README traceinstall-0.0001/Makefile traceinstall-0.0001/traceinstall.1 traceinstall-0.0001/traceinstall.pl)
X
Xsharball:
X	(cd ..; shar traceinstall-0.0001/README traceinstall-0.0001/Makefile traceinstall-0.0001/traceinstall.1 traceinstall-0.0001/traceinstall.pl >/tmp/traceinstall.shar)
X
END-of-traceinstall-0.0001/Makefile
echo x - traceinstall-0.0001/traceinstall.1
sed 's/^X//' >traceinstall-0.0001/traceinstall.1 << 'END-of-traceinstall-0.0001/traceinstall.1'
X.Dd October 9, 1996
X.Dt TRACEINSTALL 1
X.Os NetBSD 1.2
X.Sh NAME
X.Nm traceinstall
X.Nd build binary tarball from the installation of a package 
X.Sh SYNOPSIS
X.Nm traceinstall 
X.Op packagename
X.Sh DESCRIPTION
XMany software packages for 
X.Ux
Xare distributed in source-code form, and are built and installed using
X.Xr make 1 .
X
XThe
X.Nm traceinstall
Xprogram uses the
X.Xr ktrace 1
Xprogram to audit the execution of the
Xmake install
Xcommand in the current directory.  If the install finishes without
Xreporting an error, the output from the
X.Xr kdump 1
Xprogram is used to generate a list of the files created or written by
Xthe install; this is then fed into
X.Xr pax 1
Xto generate a tarball in 
X/tmp
Xwhich is subsequently compressed using
X.Xr gzip 1 .
X
XThe name of the tarball includes the package name provided on the
Xcommand line (or the basename of the current directory if no
Xpackagename is given), and the name of the system, release, and
Xmachine as printed by 
X.Xr uname 1 .
X
X.Sh SEE ALSO
X.Xr gzip 1 ,
X.Xr ktrace 1 ,
X.Xr kdump 1 ,
X.Xr options 9 ,
X.Xr pax 1 ,
X.Xr perl 1 ,
X.Xr uname 1
X.Sh BUGS
XProbably many.
X
END-of-traceinstall-0.0001/traceinstall.1
echo x - traceinstall-0.0001/traceinstall.pl
sed 's/^X//' >traceinstall-0.0001/traceinstall.pl << 'END-of-traceinstall-0.0001/traceinstall.pl'
X#! /usr/local/bin/perl
X#
X# Copyright (c) 1996 Bill Sommerfeld
X# All rights reserved.
X#
X# Redistribution and use in source and binary forms, with or without
X# modification, are permitted provided that the following conditions
X# are met:
X# 1. Redistributions of source code must retain the above copyright
X#    notice, this list of conditions and the following disclaimer.
X# 2. Redistributions in binary form must reproduce the above copyright
X#    notice, this list of conditions and the following disclaimer in the
X#    documentation and/or other materials provided with the distribution.
X# 3. All advertising materials mentioning features or use of this software
X#    must display the following acknowledgement:
X#      This product includes software developed by Bill Sommerfeld.
X# 4. The name of the author may not be used to endorse or promote products
X#    derived from this software without specific prior written permission
X#
X# THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR
X# IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
X# OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
X# IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT,
X# INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
X# NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
X# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
X# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
X# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF
X# THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
X
Xrequire Getopt::Long;
X
Xsub dehex {
X    local ($arg)=(@_);
X    if ($arg =~ /^0x[0-9a-f]+$/) {
X	$arg=eval $arg;
X    }
X    return $arg;
X}
X
Xsub pathsimp {
X    local ($res)=(@_);
X    # add trailing slash.
X    $res = "${res}/";
X    # remove `//' (sorry, domain/os...)
X    $res =~ s:\/\/::g;
X    # remove `.'
X    $res =~ s:\/\.\/:\/:g;
X    # remove `..'
X    # !!! NOT QUITE RIGHT.. turns /../../../ into /../
X    $res =~ s:\/[^\/]+\/\.\.\/:\/:g;
X    # remove trailing slash
X    $res =~ s:\/$::;
X
X    return $res;
X}
X
Xsub fullname {
X    local ($pid, $str, $where)=(@_);
X
X#    print "Fullname `$str' at $where\n";    
X
X    $str =~ s/^"(.*)"$/\1/;
X
X    if (($str =~ /^\//) || ($str =~ /^\?\?\?/)) {
X	return &pathsimp($str);
X    } else {
X	local ($d, $res);
X	$d = $wdir{$pid};
X	if ($d eq "") {
X	    $d = $wdir{$pid} = "???${pid}???";
X	    print "$pid: unknown wdir\n";
X	}
X	$res = &pathsimp("$d/$str");
X	return "$res";
X    }
X}
X
X$mach=`uname -m`;
X$sys=`uname -s`;
X$rel=`uname -r`;
X
Xchomp $mach;
Xchomp $sys;
Xchomp $rel;
X
X$result = &Getopt::Long::GetOptions("pkgname=s", "tracefile=s", "dir=s","out=s","notar");
X
Xif (!$result)  {
X    print STDERR "Usage: traceinstall [-pkgname package] [-tracefile tracefile]\n";
X    exit(1);
X}
X
X$pkgname=$opt_pkgname;
Xif ($pkgname eq "") {
X    $pkgname=`basename \`pwd\``;
X    chomp $pkgname;
X}
X
X$pkgfullname = "$pkgname-$sys-$rel-$mach";
X
X$pkgoutdir=$opt_dir;
X$pkgoutdir="/tmp" if ($pkgoutdir eq "");
X
X$pkgoutname=$opt_out;
Xif ($pkgoutname eq "") {
X    $pkgoutname="${pkgoutdir}/${pkgfullname}.tar";
X}
X
Xif ($opt_tracefile) {
X    $pkgtracename = $opt_tracefile;
X} else {
X    $pkgtracename="/tmp/${pkgfullname}.out";
X
X    print "\nTracing `make install' into $pkgtracename..\n\n";
X
X    $status = system("ktrace", "-f", $pkgtracename, "-t", "cn", "-i", "make", "install");
X
X    if ($status) {
X	print STDERR "\nmake install failed; not packaging!\n";
X	exit(1);
X    }
X
X    print "\nInstall succeeded!\n";
X}
X
Xprint "Parsing trace data..\n\n";
X
X#print &fullname(0, "\"/tmp/foo\"", 0);
X#print "\n";
X
X# start with "ktrace -f /tmp/zsh-install.out -t cn -i make install"
X
Xopen (TRACEOUT, "kdump -f ${pkgtracename} -n -t cn|");
X
Xwhile (<TRACEOUT>) {
X    chop;
X    ($xx, $pid, $cmd, $op, @args)=split(/ +/);
X    if (($op eq "RET")) {
X	$sysc = $args[0];
X	$retv = $args[1];
X	if (($sysc eq "open")) {
X	    $fd=&dehex($retv);
X	    $key="$pid,$fd";
X	    $openf{$key}=$openfn;
X#	    print("$pid fd $fd is $openfn\n") if ($DEBUG);
X	} elsif ((($sysc eq "fork") || ($sysc eq "vfork")) 
X		 && ($retv ne "0")) {
X	    $newpid=&dehex($retv);
X#	    print "$pid begat $newpid\n" if ($DEBUG);
X	    $wdir{$newpid}=&fullname($pid,"$wdir{$pid}",".", "fork");
X	}
X    } elsif ($op eq "CALL") {
X	$lastargs=$args[0];
X	($sysc,@argv)=split(/[(),]/, $lastargs);
X#	if ($DEBUG) {
X#	    if (($sysc eq "fork") || ($sysc eq "vfork")) {
X#		print "$pid: $lastargs\n";
X#	    }
X#	}
X	if (($sysc eq "fchdir")) {
X	    $fd = &dehex($argv[0]);
X	    $key="$pid,$fd";
X	    $newfn=$openf{$key};
X#	    print "fchdir($key) => $newfn\n" if ($DEBUG);
X	    $wdir{$pid}=$newfn;
X	}
X	$lastpid=$pid;
X    } elsif ($op eq "NAMI") {
X	if ($lastpid != $pid) {
X	    print ("mismatch: NAMI $pid vs CALL $op $lastpid\n");
X	    exit(1);
X	}
X	if (($sysc eq "open")) {
X	    $openmode=&dehex($argv[1]);
X	    $openfn = &fullname($pid,$args[0],"open");
X#	    print "open fullname is $openfn\n" if ($DEBUG);
X	    next if (($openmode & 3) == 0);
X# 	    if ($DEBUG) {
X# 		printf "%s: %s 0x%x<", $sysc, $args[0], $openmode;
X# 		print "wronly" if (($openmode&3)==1) ;
X# 		print "rdwr" if (($openmode&3)==2) ;
X# 		print ",nonblock" if ($openmode&4) ;
X# 		print ",append" if ($openmode&8) ;
X# 		print ",shlock" if ($openmode&0x10) ;
X# 		print ",exlock" if ($openmode&0x20) ;
X# 		print ",async" if ($openmode&0x40) ;
X# 		print ",fsync" if ($openmode&0x80) ;
X# 		print ",0x100?" if ($openmode&0x100) ;
X# 		print ",creat" if ($openmode&0x200) ;
X# 		print ",trunc" if ($openmode&0x400) ;
X# 		print ",excl" if ($openmode&0x800) ;
X# 		print ",fmark" if ($openmode&0x1000) ;
X# 		print ",fdefer" if ($openmode&0x2000) ;
X# 		print ",fhaslock" if ($openmode&0x4000) ;
X# 		print ",nonblock" if ($openmode&0x8000) ;
X# 		print ">\n";
X# 	    }
X	    $fileset{$openfn}="file";
X	} elsif (($sysc eq "rename") ||
X		 ($sysc eq "link")) {
X	    if ($fn1 ne "") {
X		$fn2 = &fullname($pid,$args[0],$sysc);
X#		print "$sysc: $fn1 to $fn2\n" if ($DEBUG);
X		$fileset{$fn2}=$fileset{$fn1};
X		undef $fileset{$fn1} if ($sysc eq "rename");
X		$fn1 = "";
X	    } else {
X		$fn1 = &fullname($pid,$args[0],$sysc);
X	    }
X	} elsif (($sysc eq "chdir")) {
X#	    print "$sysc: $args[0]\n" if ($DEBUG);
X	    $fn = &fullname($pid,$args[0],$sysc);
X	    $wdir{$pid}=$fn;
X	} elsif (($sysc eq "mkdir")) {
X#	    print "$sysc: $args[0]\n" if ($DEBUG);
X	    $fn = &fullname($pid,$args[0],$sysc);
X	    $fileset{$fn}="dir";
X	} elsif (($sysc eq "rmdir")) {
X#	    print "$sysc: $args[0]\n" if ($DEBUG);
X	    $fn = &fullname($pid,$args[0],$sysc);
X	    undef $fileset{$fn};
X	} elsif (($sysc eq "unlink")) {
X#	    print "$sysc: $args[0]\n" if ($DEBUG);
X	    $fn=&fullname($pid,$args[0],$sysc);
X	    undef $fileset{$fn};
X	} elsif (($sysc eq "break") ||
X		 ($sysc eq "close") ||
X		 ($sysc eq "lseek") ||
X		 ($sysc eq "access") ||
X		 ($sysc eq "stat") ||
X		 ($sysc eq "lstat") ||
X		 ($sysc eq "chmod") ||
X		 ($sysc eq "chown") ||
X		 ($sysc eq "utimes") ||
X		 ($sysc eq "execve") ||
X		 ($sysc eq "stat") ||
X		 ($sysc eq "getdirentries") ||
X		 ($sysc eq "fstatfs") ||
X		 ($sysc eq "fcntl")) {
X	    next;
X	} else {
X	    die("unhandled syscall $sysc\n");
X	}
X    }
X}
X
Xundef $fileset{"/dev/null"};
X
Xif ($opt_notar eq "") {
X    print "\nCreating output package in `${pkgoutname}' ...\n\n";
X    open (PAX, "|pax -wvd -f ${pkgoutname} -s :^/:./:") || die "Can't spawn pax";
X    select(PAX); $| = 1; select(STDOUT);
X} else {
X    open (PAX, ">/dev/null");
X}
X
Xforeach $file (sort keys %fileset) {
X
X
X    if ($fileset{$file}) {
X	if ($file =~ /\?\?\?/) {
X	    print "Junk file $file omitted from fileset..\n"; 
X	    next;
X	}
X	if ($DEBUG) {
X	    print "$file: $fileset{$file}\n";
X	}
X	print PAX "$file\n";
X    }
X} 
Xclose PAX;
X
Xif ($opt_notar eq "") {
X    print "\nCompressing package to `${pkgoutname}.gz' ...\n\n";
X
X    $status = system("gzip","-v9","${pkgoutname}");
X
X    if ($status) {
X	die "gzip failed with status $status\n";
X    }
X    print "\nPackaging complete!\n";
X}
X
END-of-traceinstall-0.0001/traceinstall.pl
exit