More information on this issue.

I did write-distinctive-data tests.  I wrote the whole array full of
distinctive data and read it back, all through the raw device, and it
was all correct.  I then read it back again through the block device
and it was still all correct.

I then tried a compress-and-uncompress test using bzip2/bunzip2 instead
of gzip/gunzip, and it worked fine.  (The data was incompressible; I
created it by piping 16G of /dev/zero through arc4crypt.)

However, I then tried to rm -rf four directories created for another
test and got the ENOTEMPTY syndrome.  I looked at the directory as a
file and it contained fragments of a text file.  I unmounted and ran
fsck, which was happy with the filesystem.  I mounted it again, and it
contained fragments of some process's environment strings.  I umounted
it and remounted it immediately and this time it contained
random-looking binary garbage.

It is extremely unlikely that environment fragment got written to that
filesystem anywhere after I overwrote everything with distinctive data
patterns (the only way I can see it appearing in a file at all is
through a core dump, and I'm fairly sure nothing has dumped core in
that filesystem since the overwrite test).  This has me feeling almost
certain that whatever is going wrong is causing 32-bit sector numbers
to sign-extend and thus trigger the "if (bp->b_blkno < 0) {
biodone(bp); return(0); }" code Chuck Silvers quoted in connection with
PR 28291.

I'm just about to see if I can dig up kernel sources corresponding to
this kernel so I can try Charles Hannum's patch to see if that helps
any.  If not, I may try also backing out the changes Chuck Silvers
quoted the commit messages from (for this machine we don't care if LFS
is broken).

/~\ The ASCII				der Mouse
\ / Ribbon Campaign
 X  Against HTML	       mouse@rodents.montreal.qc.ca
/ \ Email!	     7D C8 61 52 5D E7 2D 39  4E F1 31 3E E8 B3 27 4B