On Thu, Dec 20, 2001 at 05:58:33PM -0800, cgd@broadcom.com wrote:
  | More seriously, I think this is a problem that should be solved, and I
  | lean towards solution 3 as well.  It's the most sane long-term, in my
  | opinion.
  | 
  | 
  | At Fri, 21 Dec 2001 01:20:14 +0000 (UTC), "Luke Mewburn" wrote:
  | >   3. Dynamically link everything against /lib
  | > 
  | > 	Provide /lib, which contains /lib/ld.elf_so and all of the
  | > 	.so* files from /usr/lib.  /usr/lib still contains the .a,
  | > 	_p.a, and _pic.a libraries.  Convert /usr/libexec/ld.elf_so
  | > 	to a symlink to /lib/ld.elf_so.
  | 
  | uh, well, if keeping consistent w/ the filenames used in the rest of
  | the system, why not /libexec/ld.elf_so?

A few (probably subjective :) reasons:

	- Why create /libexec just to contain ld.*so ?

	- SVR4 has ld.*so in /lib (which is actually /usr/lib), as
	  well as /etc/lib (for some programs in /)

	- Linux has ld.*so in /lib 

	- It's a .so file, so let's put it with the other .so files.
	  (ok, ok, that's a lame reason :)


  | > 	Dynamically link all of /bin and /sbin and the other bits of
  | > 	the tree which are currently statically linked, against /lib.
  | 
  | uh, excluding what?  there _have_ to be _some_ exclusions, like i'd
  | bloody well hope 'init' isn't dynamically linked.  8-)

My solution would not preclude creating statically linked programs;
they just won't have the benefits provided by the dlopen()-aware
dynamic libc.so.


  | That does restrict 'init' from e.g. knowing about some mechanisms of
  | getting passwords (no PAM for root for secure console access! 8-), but
  | if you don't do this I don't see how you can have sane recovery.

As to /sbin/init; there's a couple of solutions:

	- Statically link /sbin/init, for simple sane recovery.
	  My test box currently has this.
	  
	- Dynamically link /sbin/init so that dynamic features
	  can be used (e.g, ldap for secure console access, different
	  dynamically linked locales, etc ;-).
	  Provide /sbin/init.static and/or /recovery/sbin/static
	  (see below for more info about recovery options), and add
	  those paths to the list of paths (listpaths[]) that
	  sys/kern/init_main.c::start_init() tries to exec.

The former is simpler, and less intrusive whilst experimenting with
the system.  The latter still providss for recovery.


  | > 	Provide an optional recovery mechanism such as /stand/{,s}bin
  | > 	which contains statically linked versions of some (or all) of
  | > 	the programs in /{s,}bin.
  | 
  | what's the thought of how to invoke the recovery mode?  (can't invoke
  | sh for rc, time to recover?)  I think the possibility of a special
  | 'recovery kernel' is losing.  (If you're the klutz who rm'd libc.so,
  | what are the odds you've got a correct recovery kernel?  8-)
  | 
  | I think some thought has to be put into the set of programs for
  | recovery.  Maybe it's the current set of /sbin & /bin, maybe it's more
  | or less.

My test machine now has /recovery/bin and /recovery/sbin, as a full
copy of the old statically linked /bin and /sbin.  This is good for
testing, but does consume more disk space :)

My current thoughts are that a /recovery directory (either single
level, or with bin and sbin subdirectories), containing statically
linked versions of tools that are useful to help recovery, is the
best approach.  The list of tools could be the set of /sbin and /bin,
although arguably it could more like the list of tools provided in the
ramdisk install media.

Andrew Brown sent me details of a /recovery he built from
src/distrib/i386/floppies/ramdisk-big as a possible solution to this
problem.  It's a crunchgen-ed binary, consumes approximately 1.6MB,
and contains:
	[ bad144 cat cd9660 chgrp chio chmod chown chroot cp dd df
	dhclient disklabel ed fdisk ffs fsck fsck_ffs ftp gunzip
	gzcat gzip halt ifconfig init installboot kernfs less ln
	ls mbrlabel mkdir mknod more mount mount_cd9660 mount_ext2fs
	mount_ffs mount_kernfs mount_mfs mount_msdos mount_nfs
	mount_ntfs msdos mt mv newfs nfs ntfs pax ping ping6 pwd
	rcmd reboot rm route rtsol scsictl sed sh shutdown slattach
	stty swapctl sync sysinst tar test tgt tip umount wiconfig

Obviously we'd need to put more thought into this list of tools, but
this example shows how small we can make things.

In the case of a broken ld.so (or whatever), recovery is not much
different than today:

	- If you are already logged in to your system you'd just
	  use the tools in /recovery instead of /bin and /sbin. 

	- If you reboot to single user mode, you can specify
	  /recovery/sh as the shell.  If /sbin/init was dynamically
	  linked (see above), the kernel could just try
	  /recovery/init as an alternative (see above).


  | > 		- dynamic linked applications apparently run slower
  | > 		  than statically linked applications.
  | > 		  Adding support for "prebinding" (in the newer
  | > 		  binutils toolchain for some platforms) alleviates
  | > 		  most (if not all) of this.
  | 
  | s/apparently//
  | 
  | Does pre-binding do anything to touch the 'runs slower' problem?  (I
  | thought it just helped -- but did not completely address -- the
  | 'starts slower' problem.)

I'm not an expert here; I'll defer to the toolchain experts here.

Regarding the overhead of dynamic linked applications, I took a look
at the accounting information from my main machine.  The top 30 lines
are:

# sa -an | head -30
  776734  6350389.44re      967.79cp       35avio        0k   
  221246        9.29re        0.03cp        5avio        0k   rpc.rstatd
  113055     1956.75re        0.08cp        0avio        0k   sh*
   76755    60617.01re        2.11cp        0avio        0k   sh
   44794        4.05re        0.62cp        0avio        0k   sed
   42463        3.27re        0.02cp        0avio        0k   formail
   28948   870497.72re       35.56cp        4avio        0k   ssh
   21343        1.69re        0.00cp        1avio        0k   procmail*
   21120      158.18re        4.91cp        9avio        0k   procmail
   14944       37.45re        0.58cp        6avio        0k   rm
   12431    11318.53re        0.04cp        0avio        0k   cat
   12293       25.23re        8.85cp        3avio        0k   cpp
   12143        0.34re        0.00cp        1avio        0k   rpcbind*
   12137      934.51re        6.05cp        3avio        0k   make
   11035      217.30re        0.05cp        1avio        0k   cc
   10339        1.08re        0.03cp        1avio        0k   pkg_info
    8703      168.49re       58.33cp        0avio        0k   cc1
    7815       12.61re        2.28cp        1avio        0k   as
    6975        1.74re        0.02cp        0avio        0k   egrep
    6564        3.37re        0.06cp        0avio        0k   awk
    5555       24.33re        0.03cp        0avio        0k   diff
    4688        0.07re        0.13cp        0avio        0k   uname
    4335        5.53re        0.66cp        8avio        0k   ls
    3612       30.35re        1.39cp       19avio        0k   grep
    3362    46358.63re        2.12cp        1avio        0k   less
    3104        0.47re        0.02cp        3avio        0k   install
    2975        6.83re        3.83cp        2avio        0k   ld
    2838        6.97re        0.03cp        0avio        0k   collect2
    2741        0.06re        0.00cp        0avio        0k   arp
    2648    33478.41re        9.73cp       22avio        0k   vi

Of those, the following are statically linked:
	sh rm cat ls
(My shell - tcsh - has an inbuilt ls(1), which explains why ls is low
on that table)

Whilst the overhead of starting dynamic linked applications does
exist, the majority of programs that I use are dynamically linked
so I'm already "feeling this pain", and I'd argue that the vast
majority of users are in the same situation.
I.e, fixing this startup slowness of dynamic linked applications
should be investigated, even separately to this proposal!


  | From where I sit, just to be up front about it:
  | 
  | * the system needs to continue supporting creation of new
  | statically-linked binaries which have normal basic functionality
  | (i.e. passwd file and group lookups, DNS, etc.) that you'd expect.  At
  | the very least, you probably want people to be abile to run SPEC
  | statically-linked.  8-)

For the specific case of adding dynamic module support to nsswitch, 
my intention has always been to support both compiled-in and
dynamically loaded databases (e.g, "nis", "files"), etc, with
user controlled overrides in nsswitch.conf as to which linkage takes
priority.

As mentioned above, compilation of static binaries would still be
possible; said programs just would not have support for the funkier
features that the libc.so tools would provide.  (Similar to the way
that dynamic loading of multibyte regions is not available for
statically linked programs).



  | * (with the same caveat) it should remain possible to build a complete
  | system statically, or to build parts of the system statically as is
  | currently done.  (You don't like it?  Don't do it.  Oh, you wanted
  | those fancy new features?  You chose to not enable them; feel free to
  | enable them at your convenience.  8-)

Yes. :)


  | Personally, on some systems (mostly, systems where i've got root on
  | NFS), I've been using completely dynamically linked systems for years,
  | and loving it.  As long as you're careful (i have rm aliased to rm -i
  | everywhere 8-) it's a good choice for an individual user, and if good
  | recovery tools are provided I think it's a fine choice for the system
  | as a whole.

Providing good recovery tools is definitely my goal.  Other systems
like Solaris and Linux often have good ideas but the implementation
needs refinement, and it is not necessary to avoid the functionality
because of bad experience with a flawed implementation.  I believe
strongly in good recovery tools because of this.


  | Really, I don't think _I'd_ mind if we wanted to move in the direction
  | of / & /usr on same partition.  However, I suspect I'd have little
  | company there, and people with ancient, small disks would howl (for
  | good reason, I suppose; it's probably hard to replace some of them, at
  | this point)...

I personally prefer keeping / and /usr together, but I have no
intention of forcing any particular requirement down people's throats
here.  That's why I proposed the /lib stuff in the first place.


I have working versions of the third and fourth proposed solutions
(moving /usr/lib/*.so -> /lib, moving parts of /usr/lib/*.so -> /lib),
and pointers to building a reasonable /recovery system, and will
expand on that in a separate email.

Thanks for your comments and feedback,
Luke.

--
Luke Mewburn  <lukem@wasabisystems.com>  http://www.wasabisystems.com
Luke Mewburn     <lukem@netbsd.org>      http://www.netbsd.org
Wasabi Systems - NetBSD hackers for hire
NetBSD - the world's most portable UNIX-like operating system