Subject: Re: pkgsrc reorg proposal II (dist as vnd FS)
To: None <tech-pkg@netbsd.org>
From: John Refling <johnr@imageworks.com>
List: tech-pkg
Date: 01/26/2001 12:40:00
I was following the discussion on the massive explosion
of the quantity of small files in pkgsrc and the length
of TIME it takes to extract it.  I actually ran out of
inodes (I have syssrc, src, and pkgsrc extracted) when
extracting another copy of pkgsrc.

I had a *CRAZY* idea which may be an option for SOME
people... it works well for me:

Take the pkgsrc.tar.gz and convert it to a filesystem image
compatible with vnd:

dd if=/dev/zero of=pkgsrc.vnd bs=1m count=90
vnconfig /dev/vnd0d pkgsrc.vnd
disklabel -e -I vnd0
newfs -m 0 -o space -i 1024 /dev/rvnd0d
mount /dev/vnd0d /mnt
tar -C /mnt -zxf /mnt pkgsrc.tar.gz
umount /mnt
vnconfig -u /dev/vnd0d

[may need 0c throught instead of 0d on non-i386 systems].

The above is done once, then, at any time you can

vnconfig /dev/vnd0d pkgsrc.vnd
mount -r /dev/vnd0d /mnt

and have the entire extracted pkgsrc available INSTANTLY.

There are occasions where this is useful... ie, I create the
vnd image once on a fast machine, then bring it to my other
slower non-internetted machines.

I prefer the read-only status, since it keeps the pkgsrc
uncluttered and unmodified, and forces me to keep better
track of my local changes.  One could use the overlay or
union file system on top of this for local changes.  Or
you can write into it, carefully, w/o the -r on mount.

Of course I build in a writable directory by setting
WRKOBJDIR in /etc/mk.conf.

This is really simple, and although might not be the way to
go for power users, could get beginners up and running quickly.

It might be well to mention this option somewhere.
*Or*, even make this an alternative distribution format for
pkgsrc, syssrc, and src, etc !!*  I personally always
convert those tar files to vnd format, after downloading.


Postscript
~~~~~~~~~~
Ok... I've been testing this rather extensively and it works
well, except that it appears that filesystems created on little
endian machines cannot be used on big endian machines.  It says
"incorrect super block".  I see that newfs can take an arg
-B le|be to create different type of filesystems.

Is there an endian independent filesystem or one that can mount
a different endianness filesystem than the host?

This must also mean that a scsi hard disk initialized and loaded
with data on a sparc, cannot be physically plugged in to a i386
and successfully mounted.  Is this true?

COMMENTS?????????



Technical notes
~~~~~~~~~~~~~~~
If the tar fails due to running out of inodes on the vnd0d, you
can decrease the number following the -i in the newfs.  As this
number is decreased, more files can be created, but more of the
image is devoted to keeping track of those files.  As an example,
a 100 Mb image file turns into:

90.7 Mb useable space with -i 1024
87.4 Mb useable space with -i  768
81.9 Mb useable space with -i  512
67.7 Mb useable space with -i  245

If the tar fails due to file system full, increase the count value
in the dd, to create a larger blank starting image. Obviously, as
more and more packages are added, these numbers will need to be
adjusted.

pkgsrc.tar.gz =  6,364,609 bytes
pkgsrc.tar    = 55,582,720 bytes
pkgsrc.vnd    = 68,157,440 bytes
pkgsrc.vnd.gz =  6,521,966 bytes

So, no great loss of storage.  Would be nice to have a
*compressed* vnd file system, even if readonly!