Subject: Re: pwd_mkdb - optimisation.
To: None <netbsd-users@netbsd.org, netbsd-current@netbsd.org>
From: Stephen M Jones <smj@cirr.com>
List: netbsd-users
Date: 08/10/2001 17:16:51
Jermey Reed wrote:

> Not just passwords getting changed, but shells and other fields.
> The problem is that it takes a long time to generate the /etc/pwd.db and
> /etc/spwd.db files when there are many users.

This is true as I've found with nearly 30,000 accounts that you have an
IO issue writing both files (which are about 13mbs stock hashinfo params,
10mb tuned).  If you have a number of accounts and this is a losing 
issue for you, my recommendations are:

1. look at HASHINFO in pwd_mkdb.c and play with the ffactor and nelem
   parameters.  If you have extra memory, up your cache to accomodate
   the size of the hashtables for building.

2. 1. will reduce your hash time to about 20 seconds, if that still isn't
   good enough, try building an mfs specifically for maintaining your
   pwd.db and spwd.db file (you might want to also let 'ptemp' live on
   that space as well).  Be sure to include your own rc which run right
   after 'mountall' so your machine will boot sanely.

3. Incorporate Takahiro's changes or basic ideas into the current tree.
   This is really important to me and others.  Unfortunately, his patches
   can not be applied, but his ideas could be incorporated with minimal
   hacking.

4. fundamental db change from hash to btree?  probably not so trivial

Wojciech Puchar wrote:

> it's still 6.58s too much
> the right solution is to use database files directly and only change data
> that were changed without rebuilding. "real" files should be generated
> only in case of vipw.

Thank you, I think there is strong need for incorporating Takahiro's 
mods (or concept/ideas) into pwd_mkdb.c, chpass.c and local_passwd.c ..
without trying to be offensive, this is an gross oversight.  Another
list member queried "Never mind the passwd, how is your machine handling
that size of a user base?" (paraphrased!) Well, its handling it quite 
well .. of course, these users are never all logged in at once, but the
systems (now leenox on peecees, with the alpha being used by trusted
test users) typically handle 100-240 users on at any given time.  pop3
requests are another issue, mail too .. which is constantly coming in.

Greg Woods wrote:

> The real world doesn't work that way.  For the general purpose system
> both /etc/passwd and /etc/master.passwd must also be kept up-to-date for
> non-DB-capable programs which access their contents (awk, cut, sed, etc.).

That is true .. when I was first exposed to bsd's passwd database ideas
years ago, I understood why .. there are trade offs.  Its unfortunate 
that user accounts are stored in 'too many' ways as they are with a slew
of temporary files (pw.*a, ptmp, ptmp.orig, pwd.db.tmp, spwd.db.tmp) ..
its a 'unix' way, but its also extremely krufty .. I was hoping by 
dumping fuel on an old thread, that we could all work together to 
resolve this as its a big issue for large user base systems.  My kludge
could work for many and could be acceptable.. its what I'll use for
the interim.  I would love to have a final hack, I'm sorry that I just
can't provide one :(

SMJ