Bill Sommerfeld <sommerfeld@orchard.arlington.ma.us>  wrote:
 > I think what's going on here is that ffs_dirpref() may be screwing up
 > and always picking an initial cylinder group with few directories,
 > lots of free inodes..  but no free blocks.. so it winds up hunting all
 > over the disk for free blocks before it finds one for the directory.

wow. I think I see.

 > So, the core of ffs_dirpref() in sys/ufs/ffs/ffs_alloc.c is:
 > 
 >         for (cg = 0; cg < fs->fs_ncg; cg++)
 >                 if (fs->fs_cs(fs, cg).cs_ndir < minndir &&
 >                     fs->fs_cs(fs, cg).cs_nifree >= avgifree) {
 >                         mincg = cg;
 >                         minndir = fs->fs_cs(fs, cg).cs_ndir;
 >                 }
 > 
 > maybe it should be something more like:
 > 
 >         for (cg = 0; cg < fs->fs_ncg; cg++)
 >                 if (fs->fs_cs(fs, cg).cs_ndir < minndir &&
 > 		    fs->fs_cs(fs, cg).cs_nbfree > 0 &&		
 >                     fs->fs_cs(fs, cg).cs_nifree >= avgifree) {
 >                         mincg = cg;
 >                         minndir = fs->fs_cs(fs, cg).cs_ndir;
 >                 }
 > 
 > .. but I must admit I'm not an expert on ffs guts..

I tried out this change and it's cruising along happily now untarring
pkgsrc.tar.gz (my benchmark for causing the problem).

Now I'll leave it up to y'all to determine the correctness of the change
but it's definitely improved the mkdir(2) situation on this.