Subject: Re: cluster install
To: None <>
From: Jan Schaumann <>
List: tech-cluster
Date: 10/21/2003 22:28:45
Content-Type: multipart/mixed; boundary="6c2NcOVqGQ03X4Wi"
Content-Disposition: inline

Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

MLH <> wrote:
> Jan Schaumann wrote:

> > Actually, we're using rsync

> What might be the chances of letting us see/work on what you have?

Sure, why not.  If you find anything odd or obviously stupid, I'll be
happy to take suggestions. :)

The basic setup is as follows:

Main file server (doppelbock):

(doppelbock) pwd
(doppelbock) ls
NASTY_HACKS               etc-c009                  etc-c023
aftersync                 etc-c010                  etc-c024
aftersync.not             etc-c011                  etc-c025
aftersync.old             etc-c012                  etc-c026
aftersync.really.old      etc-c013                  etc-c027
beforesync.old            etc-c014                  etc-c028
bin                       etc-c015                  etc-c029
etc-c002                  etc-c016                  etc-c030
etc-c003                  etc-c017                  etc-c031
etc-c004                  etc-c018                  firstboot
etc-c005                  etc-c019                  logs
etc-c006                  etc-c020                  motd
etc-c007                  etc-c021                  rsync-excludes
etc-c008                  etc-c022
(doppelbock) cat NASTY_HACKS
/etc/rc.d/mountcritlocal paxes /var-ro over mfs mounted /var

/usr/local/node needs an up-to-date MAKEDEV
Problem: nodes mfs mount /dev/, so we can't update that version
Hack: hack /sbin/init to not use /dev/MAKEDEV and /dev/MAKEDEV.local
         but fetch it from /etc/ where we placed it (ie /usr/local/node/etc
        on doppelbock!)  In addition, /dev is created too small for our
        purposes, so we increased the size.
        See /usr/src/sbin/init/init.c around line 1405 and 1446
(doppelbock) ls etc-c002
inetd.conf rc.conf
(doppelbock) cat rsync-excludes=20
(doppelbock) ls bin
(doppelbock) and are attached -- they are used to push out an update
to all nodes.  The nodes image is in /usr/local/node on the file
server, with various shared directories located on the file server and
symlinked to from the nodes base system (users home directories are set
to /share/home/<username> in /etc/passwd et al).  Installing new
packages is thus as easy as on a single system:  simply go into
/usr/pkgsrc/category/package on doppelbock and run 'make install'.

(doppelbock) cd /usr/local/node
(doppelbock) ls -l usr/X11R6 usr/obj usr/pkg* usr/src var-ro/db/pkg
lrwxr-xr-x  1 root  wheel  12 Dec 15  2002 usr/X11R6 -> /share/X11R6
lrwxr-xr-x  1 root  wheel  10 Dec 15  2002 usr/obj -> /share/obj
lrwxr-xr-x  1 root  wheel  10 Dec 15  2002 usr/pkg -> /share/pkg
lrwxr-xr-x  1 root  wheel  13 Dec 15  2002 usr/pkgsrc -> /share/pkgsrc
lrwxr-xr-x  1 root  wheel  10 Dec 15  2002 usr/src -> /share/src
lrwxr-xr-x  1 root  wheel  13 Dec 15  2002 var-ro/db/pkg -> /share/db/pkg
(doppelbock) diff -bu etc/rc.d/mountcritlocal /etc/rc.d/mountcritlocal
--- etc/rc.d/mountcritlocal     2003-08-27 18:04:57.000000000 -0400
+++ /etc/rc.d/mountcritlocal    2002-09-08 15:33:33.000000000 -0400
@@ -20,10 +20,6 @@
        mount_critical_filesystems local
-       # XXX: local change
-       # /var is mfs mounted over a read-only /var-ro
-       cd /var-ro; pax -rwp e . /var
        #       clean up left-over files.
        #       this could include the cleanup of lock files and
        #       /var/run, etc.
(doppelbock) cd /usr/src/sbin/init
(doppelbock) cvs diff -bu
cvs server: Diffing .
Index: init.c
RCS file: /cvsroot/src/sbin/init/init.c,v
retrieving revision 1.61
diff -b -u -r1.61 init.c
--- init.c      2003/08/07 10:04:25     1.61
+++ init.c      2003/10/22 02:08:11
@@ -1401,8 +1401,8 @@
        /* Mount an mfs over /dev so we can create devices */
        switch ((pid =3D fork())) {
        case 0:
-               (void)execl(INIT_MOUNT_MFS, "mount_mfs", "-i", "192",
-                   "-s", "768", "-b", "4096", "-f", "512", "swap", "/dev",
+               (void)execl(INIT_MOUNT_MFS, "mount_mfs", "-i", "128",
+                   "-s", "10240", "-b", "4096", "-f", "512", "swap", "/dev=
@@ -1443,7 +1443,11 @@
        switch ((pid =3D fork())) {
        case 0:
                if (chdir("/dev") =3D=3D 0)
+#if 0
                        (void)execl(INIT_BSHELL, "sh", "./MAKEDEV", "init",
+                       (void)execl(INIT_BSHELL, "sh", "/etc/MAKEDEV", "all=

That's all I can think of right now.  We currently don't have a decent
automatic procedure to drop in new nodes -- the last few times I had to
reinstall a node I actually took the node, opened it up, added a CD
drive, booted off an install CD, disklabel, newfs, nfs mount from
doppelbock and pax things over.  Obviously, I, too, would be interested
in a decent automatic-install tool.

Hope you find this information useful.  If you have ideas of how to do
things more elegantly, please let me know.


Wenn ich tot bin, mir soll mal Einer mit Auferstehung oder so
kommen, ich hau ihm eine rein! (Anonym)

Content-Type: application/x-sh
Content-Disposition: attachment; filename=""
Content-Transfer-Encoding: quoted-printable

#!/bin/sh=0A#=0A# rsync the nodes with /usr/local/node=0A#set -x=0A=0ARSYNC=
"-H --delete --exclude-from=3D${MAINTDIR}/rsync-excludes"=0A=0ARSYNC_PASSWO=
td.conf rc.conf"=0A=0Ausage=3D"Usage: $0: "'=0A	[--help] [--dont] [--keep-a=
s] [--myexcludes=3D<excludes>] <hostnames|IPs>'=0A=0Adie()=0A{=0A	echo "$0:=
 Error $@"=0A	exit 1=0A}=0A=0Awhile [ $# -gt 0 ]; do=0A	case $1 in=0A	--don=
t)		DONTDOIT=3D"echo" ;;=0A	--keep-as)	KEEP_AFTERSYNC=3D"YES" ;;=0A	--myexc=
ludes=3D*)	MYEXCLUDES=3D"`echo $1 | sed -e 's|--myexcludes=3D||'`" ;;=0A	--=
help)		echo "$usage"; exit ;;=0A	-h)		echo "$usage"; exit ;;=0A	-*)		die "$=
usage" ;;=0A	*)		HOSTS=3D"$1 ${HOSTS}" ;;=0A	esac=0A	shift=0Adone=0A=0Aif [=
 "${MYEXCLUDES}" !=3D "" ]; then=0A	for exclude in ${MYEXCLUDES}; do=0A		EX=
CLUDES=3D"--exclude ${exclude} ${EXCLUDES}"=0A	done=0Afi=0A=0Afor i in ${HO=
STS}; do=0A=0A	# first things first=0A	#=0A	echo "New host: $i" >> ${MAINTD=
IR}/logs/sync.$$.log=0A=0A	if [ -e ${MAINTDIR}/beforesync ]; then=0A		${DON=
TDOIT} ${RSYNC} ${MAINTDIR}/beforesync $i:/tmp >> 	\=0A			${MAINTDIR}/logs/=
sync.$$.log 2>&1=0A		${DONTDOIT} ${RSYNC_RSH} -n $i chmod u+x /tmp/beforesy=
nc >> \=0A			${MAINTDIR}/logs/sync.$$.log 2>&1=0A		${DONTDOIT} ${RSYNC_RSH}=
 -n $i /tmp/beforesync >>		\=0A			${MAINTDIR}/logs/sync.$$.log 2>&1=0A	fi=
=0A=0A	# otherwise we can't write anything!=0A	#=0A	${DONTDOIT} ${RSYNC_RSH=
} -n $i /sbin/mount -u -o rw /=0A	${DONTDOIT} ${RSYNC_RSH} -n $i /sbin/moun=
t -u -o rw /usr=0A	${DONTDOIT} ${RSYNC_RSH} -n $i /sbin/mount -u -o rw /var=
ODESROOT}/ push@$i::slash >>	\=0A			${MAINTDIR}/logs/sync.$$.log 2>&1=0A=0A=
	for file in ${MAINTDIR}/etc-${i}/*; do=0A		${DONTDOIT} ${RSYNC} ${file} pu=
sh@$i::slash/etc/${file##*/} >>	\=0A			${MAINTDIR}/logs/sync.$$.log 2>&1=0A=
	done=0A=0A	# anything else?=0A	#=0A	if [ -e ${MAINTDIR}/aftersync ]; then=
=0A		${DONTDOIT} ${RSYNC} ${MAINTDIR}/aftersync push@$i::slash/tmp >> \=0A	=
		${MAINTDIR}/logs/sync.$$.log 2>&1=0A		${DONTDOIT} ${RSYNC_RSH} -n $i chmo=
d u+x /tmp/aftersync >>	\=0A			${MAINTDIR}/logs/sync.$$.log 2>&1=0A		${DONT=
DOIT} ${RSYNC_RSH} -n $i /tmp/aftersync >>		\=0A			${MAINTDIR}/logs/sync.$$=
.log 2>&1 &=0A=0A		if [ x"${KEEP_AFTERSYNC}" !=3D x"YES" ]; then=0A			if [ =
-f ${MAINTDIR}/aftersync.old ]; then=0A				${DONTDOIT} mv -f ${MAINTDIR}/af=
tersync.old \=0A					${MAINTDIR}/aftersync.really.old=0A			fi=0A			${DONTDO=
IT} mv ${MAINTDIR}/aftersync ${MAINTDIR}/aftersync.old=0A		fi=0A	fi=0A=0A	#=
 Update motd=0A	if [ -e ${MAINTDIR}/motd ]; then=0A		${DONTDOIT} ${RSYNC} $=
{MAINTDIR}/motd push@$i::slash/tmp >> \=0A			${MAINTDIR}/logs/sync.$$.log 2=
>&1=0A		${DONTDOIT} ${RSYNC_RSH} -n $i chmod u+x /tmp/motd >>	\=0A			${MAIN=
TDIR}/logs/sync.$$.log 2>&1=0A		${DONTDOIT} ${RSYNC_RSH} -n $i /tmp/motd >>=
		\=0A			${MAINTDIR}/logs/sync.$$.log 2>&1 &=0A	fi=0A=0A	# and back to read=
-only!=0A	#=0A	${DONTDOIT} ${RSYNC_RSH} -n $i /sbin/mount -u -o ro /var-ro=
=0A	${DONTDOIT} ${RSYNC_RSH} -n $i /sbin/mount -u -o ro /usr=0A	${DONTDOIT}=
 ${RSYNC_RSH} -n $i /sbin/mount -u -o ro /=0A=0A	# Finally, in case we made=
 any changes to /var-ro, synch the mfs=0A	# mounted /var=0A	${DONTDOIT} ${R=
SYNC_RSH} -n $i rsync -aS /var-ro/ /var=0A=0Adone=0A
Content-Type: application/x-sh
Content-Disposition: attachment; filename=""
Content-Transfer-Encoding: quoted-printable

#!/bin/sh=0A#=0A# loop through the available IPs and perform some of the us=
ual=0A# tasks=0A#=0A#set -x=0A=0A#defaults=0ABASEIP=3D"172.16.4."=0ASTART=
=3D2=0AEND=3D31=0ACMD=3D"/sbin/ping -c 1 -w 1 @IP@"=0AQUIET=3Dno=0ASYNCALL=
=3Dno=0AMAINTDIR=3D/usr/local/cluster-maint=0A=0Ausage=3D"Usage: $0: "'=0A	=
[--start=3D<num>] [--end=3D<num>] [--cmd=3D<cmd>]=0A	[-h|-help] [--ipbase=
=3D<x.x.x.>] [--quiet]=0A	[--syncall]'=0A=0Adie()=0A{=0A	echo "$0: Error: $=
@"=0A	exit 1=0A}=0A=0Awhile [ $# -gt 0 ]; do=0A        case $1 in=0A	--star=
t=3D*)	START=3D`echo $1 | sed -e 's|--start=3D||'` ;;=0A	--end=3D*)	END=3D`=
echo $1 | sed -e 's|--end=3D||'` ;;=0A	--cmd=3D*)	CMD=3D"`echo $1 | sed -e =
's|^--cmd=3D||'`" ;;=0A	--ipbase=3D*)	BASEIP=3D`echo $1 | sed -e 's|--ipbas=
e=3D||' | \=0A				awk --re-interval '/^((([0-9]{1,2})|((1[0-9]{2})|(2([0-4]=
[0-9]|5[0-5]))))\.){3}/ { print $1; }'`=0A			if [ -z ${BASEIP} ]; then=0A		=
		die "Not a valid base-ip of the format: " \=0A					"'[0-255].[0-255].[0-2=
55].'"=0A			fi=0A			;;=0A        --help)         echo "$usage"; exit ;;=0A =
       -h)             echo "$usage"; exit ;;=0A	--quiet)	QUIET=3Dyes ;;=0A=
	--syncall)	SYNCALL=3Dyes ;;=0A        -*)		echo "$usage"; exit 1 ;;=0A    =
    esac=0A        shift=0Adone=0A=0Aif [ "${SYNCALL}" =3D "yes" ]; then=0A=
	MACHINES=3D"`ls -d ${MAINTDIR}/etc-* | sed -e s/.*etc-*//`"=0A	i=3D0=0A	fo=
r m in ${MACHINES}; do=0A		i=3D$(($i+1))=0A		if [ $i -lt 10 ]; then=0A			${=
MAINTDIR}/bin/ --keep-as $m &=0A		else=0A			${MAINTDIR}/bin/ =
--keep-as $m=0A			i=3D0=0A		fi=0A	done=0Aelse=0A=0Ai=3D${START}=0Awhile [ $=
i -lt ${END} ]; do=0A        IP=3D${BASEIP}${i}=0A	CMD1=3D${CMD%@IP@*}=0A	C=
MD2=3D${CMD#*@IP@}=0A	LOCALCMD=3D"${CMD1} ${IP} ${CMD2}"=0A	if [ "${QUIET}"=
 =3D "yes" ]; then=0A		${LOCALCMD} >/dev/null=0A	else=0A		${LOCALCMD}=0A	fi=
=0A	if [ $? -gt 0 ]; then=0A		RET=3D"Failed!"=0A	else=0A		RET=3D"Success!"=
=0A	fi=0A	if [ "${QUIET}" !=3D "yes" ] || [ "${RET}" =3D "Failed!" ]; then=
=0A		echo "${LOCALCMD} : ${RET}"=0A	fi=0A        i=3D$(( $i + 1 ))=0Adone=
=0A=0Afi # syncall=0A

Content-Type: application/pgp-signature
Content-Disposition: inline

Version: GnuPG v1.2.1 (NetBSD)