I tried to test lfs again yesterday because I'm trying to write a
summary article about ffs, lfs, and ext3fs for a web journal.

After the lfs was written for a while, it ran out of free segments,
and further writing became impossible.  I'm running current-20020412.
Here's the whole story.

I newfs_lfs'ed a 1.5GB partition and mounted it.  Then immediately,
without any rebooting or unmounting, I tried to checkout NetBSD's src
and pkgsrc from my local cvs repository.  The repository is on an ffs,
and I was checking out onto the fresh lfs.

After a few hours, cvs checked out all of src and most of pkgsrc, then
hung.  All writes to the lfs now hang, but reads are still no problem.

-----8<-----
casey:/usr/home/carton# ps lp 1154
UID  PID PPID CPU PRI NI  VSZ  RSS WCHAN    STAT TT     TIME COMMAND
405 1154 1153   0  36  4 2372 2824 lfs_rese SN   ?? 18:15.79 cvs checkout -P -d
casey:/usr/home/carton# df -k /scratch/
Filesystem  1K-blocks     Used     Avail Capacity  Mounted on
/dev/sd0d     1561195   628312    776764    44%    /scratch
casey:/usr/home/carton# echo lala > /scratch/Lulu &
[2] 1875
casey:/usr/home/carton# ps lp $!
UID  PID PPID CPU PRI NI VSZ RSS WCHAN    STAT TT    TIME COMMAND
  0 1875 1778   0  28  0 644 636 lfs_rese S    p0 0:00.02 bash --login 
-----8<-----

The processes were hung like this for at least eight hours or so, with
lfs_cleanerd thrashing the disk during all that time.  I assume the
WCHAN is lfs_reserve, which is right here:

sys/ufs/lfs/lfs_bio.c:146:          error = tsleep(&fs->lfs_avail, PCATCH | PUSER, "lfs_reserve",

If I kill any of these processes sleeping on lfs_reserve, the console
says:

  lfs_availwait: out of available space, waiting on cleaner

and the process exits (cvs needed SIGKILL before it exited, but it did
exit). until I sent SIGTERM, nothing printed on the console.

Right before I killed any of the sleeping processes, I killed the
running lfs_cleanerd and restarted another one by hand with the
debugging flag.  Here's the output.

-----8<-----
casey:/usr/home/carton# /usr/libexec/lfs_cleanerd -b -n 4 -d /scratch
lfs_cleanerd[1831]: Cleaner starting on filesystem /scratch
lfs_cleanerd[1831]: Cleaner Running  at Mon May 20 06:27:15 2002
 (39 of 1049 segments available, avail = -2056, bfree = 989603)
lfs_cleanerd[1831]: adding segment 170: contains 118656 bytes
lfs_cleanerd[1831]: Warning: invalid segment summary at 0x2a800
lfs_cleanerd[1831]: add_segment: lfs_segmapv failed for segment 170
lfs_cleanerd[1831]: add_segment failed segment 170: Cannot allocate memory
lfs_cleanerd[1831]: adding segment 187: contains 121088 bytes
lfs_cleanerd[1831]: Warning: invalid segment summary at 0x2ec00
lfs_cleanerd[1831]: add_segment: lfs_segmapv failed for segment 187
lfs_cleanerd[1831]: add_segment failed segment 187: Cannot allocate memory
lfs_cleanerd[1831]: adding segment 169: contains 129024 bytes
lfs_cleanerd[1831]: Warning: invalid segment summary at 0x2a400
lfs_cleanerd[1831]: add_segment: lfs_segmapv failed for segment 169
lfs_cleanerd[1831]: add_segment failed segment 169: Cannot allocate memory
[...]
casey:/usr/home/carton# ulimit -a
core file size (blocks)     unlimited
data seg size (kbytes)      65536
file size (blocks)          unlimited
max locked memory (kbytes)  20046
max memory size (kbytes)    60140
open files                  64
pipe size (512 bytes)       1
stack size (kbytes)         512
cpu time (seconds)          unlimited
max user processes          160
virtual memory (kbytes)     66048
-----8<-----

I get the same debug results with lfs_cleanerd options '-b -n 1' and
the same again with '-s'.

I ran fsck_lfs after unmounting, which found no problems.  I also ran
dumplfs:

-----8<-----
casey:/usr/home/carton# dumplfs /dev/rsd0d
Master Superblock at 0x8:
    magic    0x70162     version  0x2         size     1703857   
    ssize    1048576     dsize    1617840     bsize    8192      
    fsize    1024        frag     8           minfree  10        
    inopb    8           ifpb     409         nindir   2048      
    nseg     1663        sepb     341         cleansz  1         
    segtabsz 5           segmask  0x0         segshift 0         
    bmask    0x1fff      bshift   13          ffmask   0x3FF     
    ffshift  10          fbmask   0x7         fbshift  3
    sushift  0           fsbtodb  1           cksum    0x7f05    
    nclean   38          dmeta    55367       minfreeseg 83        
    roll_id  0x65845164  interleave 0         sumsize  1024      
    maxfilesize  0xfffff00000000000
  Superblock disk addresses:
     0x8        0x29800    0x53000    0x7c800    0xa6000    0xcf800   
     0xf9000    0x122800   0x14c000   0x175800  
  Checkpoint Info
    free     106791      idaddr   0x196255    ifile    1         
    uinodes  2           bfree    989478      avail    -3702     
    nfiles   106786      lastseg  0x195c00    nextseg  0x196400  
    curseg   0x196000    offset   0x196276    serial   19140     
    tstamp   Thu Jan  1 08:00:00 1970

IFILE inode
    mode  o100600	nlink 1	uid   0	gid   0	size  2195456
    atime Sun May 19 07:54:18 2002
    mtime Sun May 19 07:54:18 2002
    ctime Sun May 19 07:54:18 2002
    inum  1
    Direct Addresses
	0x19600b	0x195fbc	0xb71d2	0x195b50	0x195d28	0x196256
	0x19625e	0x1866c	0x6743	0x94bb	0x9456	0xfc3f
	0x19626e	0x0	0x0

IFILE contents
free_head 106791
free_tail 106374
clean	38	dirty	1625
bfree	989516	avail	-3088
[...]
-----8<-----

-----8<-----
casey:~# umount /scratch
casey:~# sync
casey:~# fsck_lfs /dev/rsd0d
** /dev/rsd0d
** Last Mounted on /scratch
** Phase 0 - Check Segment Summaries and Inode Free List
** Phase 1 - Check Blocks and Sizes
** Phase 2 - Check Pathnames
** Phase 3 - Check Connectivity
** Phase 4 - Check Reference Counts
** Phase 5 - Check Segment Block Accounting
106786 files, 572985 used, 0 free 

UPDATE STANDARD SUPERBLOCKS? [yn] y


***** FILE SYSTEM WAS MODIFIED *****
casey:~# 
-----8<-----


-- 
commerce could only take place under the umbrella of a temporary
framework erected by powerful individuals and their gangs.  Contracts
were personal and the gift economy took its most sadistic form ("He
made me an offer I couldn't refuse").  -- Keith Hart