As those of you following the saga of large files may recall, I gave up
on trying to get a filesystem >2T working.  Instead, I set that machine
up with a filesystem slightly under 2T.  It's holding amanda virtual
tapes, containing assorted dumps.

Then, for reasons not very relevant, I created three files (with dd,
reading from a tape device).  Two of them were 250G, one slightly
smaller (somewhere in the 220G to 230G range).  Then I deleted them.
rm deleted them all in less than second.  This ran counter to my
experience dealing with large files, to the point that it made me
suspicious.  So I ran df.  I still had a pre-rm df output on my screen,
and the difference in free space before and after was on the order of a
few tens of megabytes, nowhere near .7 terabytes.

This made me very suspicious, so I unmounted the filesystem - nothing
was using it at the time - and ran fsck.  fsck found lots of errors.
Here's just the first couple of dozen lines of output from fsck -f -n:

** /dev/rld0a (NO WRITE)
** File system is already clean
** Last Mounted on /dumps
** Phase 1 - Check Blocks and Sizes
INCORRECT BLOCK COUNT I=1736199 (536192 should be 1664)
CORRECT? no

INCORRECT BLOCK COUNT I=1736201 (349184 should be 1664)
CORRECT? no

INCORRECT BLOCK COUNT I=1736203 (418944 should be 1664)
CORRECT? no

INCORRECT BLOCK COUNT I=1736205 (29696 should be 1664)
CORRECT? no

INCORRECT BLOCK COUNT I=1736207 (2064256 should be 1664)
CORRECT? no

INCORRECT BLOCK COUNT I=1736209 (2134016 should be 36864)
CORRECT? no

INCORRECT BLOCK COUNT I=1736211 (397312 should be 1664)
CORRECT? no

There were a lot more of those, with the numbers varying.  1664 was
easily the commonest "should be" count, with 2098944 next and 1792
after that, with 10 others down in the noise:

 241 1664
  90 2098944
  35 1792
   3 4196352
   1 36864
   1 2308352
   1 4196480
   1 4196736
   1 4196992
   1 8391168
   1 8392064
   1 10489088
   1 23072256

After these I got

** Phase 2 - Check Pathnames
** Phase 3 - Check Connectivity
** Phase 4 - Check Reference Counts
** Phase 5 - Check Cyl groups
FREE BLK COUNT(S) WRONG IN SUPERBLK
SALVAGE? no

SUMMARY INFORMATION BAD
SALVAGE? no

BLK(S) MISSING IN BIT MAPS
SALVAGE? no

4825 files, 70035820 used, 44248891 free (27 frags, 5531108 blocks, 0.0% fragmentation)

Now I'm very suspicious.  It looks as though even a slightly-under-2T
filesystem doesn't really work right.

It looks as though most of the data bits are still intact - I picked
one of the inodes fsck griped about, found its path with find (with the
fs mounted RO), and did a gzip -t on the compressed portion of it, and
it hasn't found any problem yet (it's still running).  But there
certainly is something wrong somewhere, even if only with fsck (a
possibility I can't yet discount, though the odd behaviour of df seems
to indicate a problem somewhere else).

I haven't yet investigated further.  I intend to do so; I'm writing
this preliminary mail in case anyone has thoughts on the matter that
might help guide my investigation.

/~\ The ASCII				der Mouse
\ / Ribbon Campaign
 X  Against HTML	       mouse@rodents.montreal.qc.ca
/ \ Email!	     7D C8 61 52 5D E7 2D 39  4E F1 31 3E E8 B3 27 4B