Subject: Strange NFS-client bug
To: None <tech-kern@netbsd.org>
From: Erik Anggard <erik.anggard@packetfront.com>
List: tech-kern
Date: 07/19/2002 02:06:31
This is a multi-part message in MIME format.
--------------020702010207000100070201
Content-Type: text/plain; charset=us-ascii; format=flowed
Content-Transfer-Encoding: 7bit

Hi,

I should probably do a send-pr on this one but I would like to get some 
feedback before I do that. (Maybe there are some more things I should 
try or more data that should be included in a pr, etc.)

We have two Mac QuickSilver 2002 running 1.6 -current (4 days old) that 
we have a problem with the NFS-clients on. The really strange thing is 
that it is only one file that we have this problem with. But it doesn't 
matter if one copies the file to another place on the same NFS-server or 
even to anther NFS-server. The symptoms are that when one tries to read 
the file (using dd, hexdump, cp or what ever) over NFS on anyone of the 
Macs the reading process will just stall and after a minute or so the 
following error message appear:
    nfs server 192.168.1.246:/export: not responding
After that the process will still be stalled and the NFS-mount in 
question will be inaccessible for the whole system. After experimenting 
a bit I noticed that if one was quick enough one could overwrite the 
file with zeros (on the server-side) and the stalling process would get 
going again and the error message would not appear.

I've tried putting the file on three different NFS-servers, one running 
NetBSD 1.6 -current, one running FreeBSD 4.6 and one running Linux, and 
the problems remains the same. I've used other NFS-clients (NetBSD 1.5.2 
on i386, FreeBSD 4.6 and Linux) and they can read the file without any 
problems. This leads me to belive that the bug is in the NetBSD -current 
NFS-client (or some of its subsystems, RPC?).

The file was originally a regualar tar-gz file (2.6MB) but I truncated 
it and filled partly with zeros from the start of the file until it 
looked like this (this is a hexdump -C, it's also available in binary 
format on http://195.163.5.2/badfile.bin):

00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  
|................|
*
0002bc50  00 00 00 00 00 00 00 00  00 00 f6 89 d9 ba 96 7d  
|...............}|
0002bc60  37 60 43 70 e8 7d aa ba  f3 49 f0 77 80 b5 48 fe  
|7`Cp.}...I.w..H.|
0002bc70  fc ac fd 01 fc 66 04 f0  be 54 74 bc ba 6d d2 8e  
|.....f...Tt..m..|
0002bc80  72 dd f7 03 87 6c 3b a4  d5 e8 3f                 |r....l;...?|
0002bc8b

If I truncate it any more or fill it with any more zeros the problem 
goes away. So this leaves us with 49 bytes of file data that somehow 
cause our NFS-client to fail. I also tried moving these 49 bytes to the 
begining of the file but that also made problem go away. (I also tried 
zeroing out some bytes in different places in the middle of this block 
and sometimes the problem went away and other times not, so appearently 
all 49 bytes are not relevant to the problem).

nfsstat does not report any errors but the RPC retries count goes up 
when one tries to read the file.

I should also say that I didn't use any special agruments to mount_nfs 
(nor to nfsd on the servers).

To me this is a very strange problem, I didn't think the content of a 
file could somehow effect the NFS-protocol but obviously it does.

Anything more I should try? Is there any more information I should 
include before I do a send-pr?

(It would be interesting to see if someone could reproduce this, maybe 
on some other arch than macppc.)

/Erik




--------------020702010207000100070201
Content-Type: application/x-java-vm;
 name="dmesg"
Content-Transfer-Encoding: base64
Content-Disposition: inline;
 filename="dmesg"

TmV0QlNEIDEuNkQgKEtJV0ktJFJldmlzaW9uOiAxLjEzOCAkKSAjMDogVHVlIEp1bCAxNiAx
MDo1ODozMCBVVEMgMjAwMgogICAgZXJpYW5nQGtpd2k6L3Vzci9zcmMvc3lzL2FyY2gvbWFj
cHBjL2NvbXBpbGUvS0lXSQp0b3RhbCBtZW1vcnkgPSA1MTIgTUIKYXZhaWwgbWVtb3J5ID0g
NDYxIE1CCnVzaW5nIDMwNzIgYnVmZmVycyBjb250YWluaW5nIDI2MzE2IEtCIG9mIG1lbW9y
eQptYWluYnVzMCAocm9vdCkKY3B1MCBhdCBtYWluYnVzMDogNzQ1MCAoUmV2aXNpb24gMi4x
KSwgSUQgMCAocHJpbWFyeSkKY3B1MDogSElEMCA4NDEwYzBiYzxFTUNQLFRCRU4sRFBNLElD
RSxEQ0UsU0dFLEJUSUMsTFJTVEssRk9MRCxCSFQ+CmNwdTA6IDgwMC4wMCBNSHoKY3B1MDog
MjU2S0IgTDIgY2FjaGUKdW5pbm9ydGgwIGF0IG1haW5idXMwCnBjaTAgYXQgdW5pbm9ydGgw
IGJ1cyAwCnBjaTA6IGkvbyBzcGFjZSwgbWVtb3J5IHNwYWNlIGVuYWJsZWQKcGNoYjAgYXQg
cGNpMCBkZXYgMTEgZnVuY3Rpb24gMApwY2hiMDogQXBwbGUgQ29tcHV0ZXIgVW5pTm9ydGgg
QUdQIEludGVyZmFjZSAocmV2LiAweDAwKQpvZmIwIGF0IHBjaTAgZGV2IDE2IGZ1bmN0aW9u
IDA6IEFUSSBUZWNobm9sb2dpZXMgUmFkZW9uIDc1MDAKb2ZiMDogNjQwIHggNDgwLCA4YnBw
CndzZGlzcGxheTAgYXQgb2ZiMCBrYmRtdXggMTogY29uc29sZSAoc3RkLCB2dDEwMCBlbXVs
YXRpb24pCndzbXV4MTogY29ubmVjdGluZyB0byB3c2Rpc3BsYXkwCnVuaW5vcnRoMSBhdCBt
YWluYnVzMApwY2kxIGF0IHVuaW5vcnRoMSBidXMgMApwY2kxOiBpL28gc3BhY2UsIG1lbW9y
eSBzcGFjZSBlbmFibGVkCnBjaGIxIGF0IHBjaTEgZGV2IDExIGZ1bmN0aW9uIDAKcGNoYjE6
IEFwcGxlIENvbXB1dGVyIFVuaU5vcnRoIEhvc3QtUENJIEJyaWRnZSAocmV2LiAweDAwKQpl
eDAgYXQgcGNpMSBkZXYgMTggZnVuY3Rpb24gMDogM0NvbSAzYzkwNS1UWCAxMC8xMDAgRXRo
ZXJuZXQgKHJldi4gMHgwKQpleDA6IGludGVycnVwdGluZyBhdCBpcnEgNTIKZXgwOiBNQUMg
YWRkcmVzcyAwMDo2MDowODo3Mjo3Yzo0MQpuc3BoeTAgYXQgZXgwIHBoeSAyNDogRFA4Mzg0
MCAxMC8xMDAgbWVkaWEgaW50ZXJmYWNlLCByZXYuIDEKbnNwaHkwOiAxMGJhc2VULCAxMGJh
c2VULUZEWCwgMTAwYmFzZVRYLCAxMDBiYXNlVFgtRkRYLCBhdXRvCm9iaW8wIGF0IHBjaTEg
ZGV2IDIzIGZ1bmN0aW9uIDA6IGFkZHIgMHg4MDAwMDAwMAp6c2MwIGF0IG9iaW8wIG9mZnNl
dCAweDEzMDAwOiBpcnEgMjIsMjMKenN0dHkwIGF0IHpzYzAgY2hhbm5lbCAwCnpzdHR5MSBh
dCB6c2MwIGNoYW5uZWwgMQppMnMgYXQgb2JpbzAgb2Zmc2V0IDB4MTAwMDAgbm90IGNvbmZp
Z3VyZWQKYWRiMCBhdCBvYmlvMCBvZmZzZXQgMHgxNjAwMCBpcnEgNDc6IHBtX2FkYl9vcDog
dGltZW91dC4gY29tbWFuZCA9IDB4MAowIHRhcmdldHMKYWVkMCBhdCBhZGIwIGFkZHIgMDog
QURCIEV2ZW50IGRldmljZQphcG0wIGF0IGFkYjA6IGJhdHRlcnkgZmxhZ3MgMHgxLCAwJSBj
aGFyZ2VkCndkYzAgYXQgb2JpbzAgb2Zmc2V0IDB4MWYwMDAgaXJxIDE5OiBETUEgdHJhbnNm
ZXIKd2QwIGF0IHdkYzAgY2hhbm5lbCAwIGRyaXZlIDA6IDxTVDM0MDAxNkE+CndkMDogZHJp
dmUgc3VwcG9ydHMgMTYtc2VjdG9yIFBJTyB0cmFuc2ZlcnMsIExCQSBhZGRyZXNzaW5nCndk
MDogMzkwOTcgTUIsIDE2MzgzIGN5bCwgMTYgaGVhZCwgNjMgc2VjLCA1MTIgYnl0ZXMvc2Vj
dCB4IDgwMDcwNjg2IHNlY3RvcnMKd2QwOiBkcml2ZSBzdXBwb3J0cyBQSU8gbW9kZSA0LCBE
TUEgbW9kZSAyLCBVbHRyYS1ETUEgbW9kZSA1IChVbHRyYS8xMDApCmF0YTQgY29uZlswXSA9
IDB4YzUxOTQ2NSwgY3ljID0gMiAoMzAgbnMpLCBhY3QgPSA2ICg5MCBucyksIGluYWN0ID0g
Mwp3ZDAod2RjMDowOjApOiB1c2luZyBQSU8gbW9kZSA0LCBETUEgbW9kZSAyLCBVbHRyYS1E
TUEgbW9kZSA0IChVbHRyYS82NikgKHVzaW5nIERNQSBkYXRhIHRyYW5zZmVycykKd2RjMSBh
dCBvYmlvMCBvZmZzZXQgMHgyMDAwMCBpcnEgMjA6IERNQSB0cmFuc2ZlcgphdGFwaWJ1czAg
YXQgd2RjMSBjaGFubmVsIDA6IDIgdGFyZ2V0cwpjZDAgYXQgYXRhcGlidXMwIGRyaXZlIDA6
IDxITC1EVC1TVCBDRC1SVyBHQ0UtODI0MEIsICwgMS4wNT4gdHlwZSA1IGNkcm9tIHJlbW92
YWJsZQpjZDA6IGRyaXZlIHN1cHBvcnRzIFBJTyBtb2RlIDQsIERNQSBtb2RlIDIKY29uZlsw
XSA9IDB4MTE4MjMsIGN5YyA9IDQgKDEyMCBucyksIGFjdCA9IDMgKDc1IG5zKSwgaW5hY3Qg
PSAxCmNkMCh3ZGMxOjA6MCk6IHVzaW5nIFBJTyBtb2RlIDQsIERNQSBtb2RlIDIgKHVzaW5n
IERNQSBkYXRhIHRyYW5zZmVycykKd2RjMiBhdCBvYmlvMCBvZmZzZXQgMHgyMTAwMCBpcnEg
MjE6IERNQSB0cmFuc2ZlcgpvaGNpMCBhdCBwY2kxIGRldiAyNCBmdW5jdGlvbiAwOiBBcHBs
ZSBDb21wdXRlciBLZXlMYXJnbyBVU0IgQ29udHJvbGxlciAocmV2LiAweDAwKQpvaGNpMDog
aW50ZXJydXB0aW5nIGF0IGlycSAyNwpvaGNpMDogT0hDSSB2ZXJzaW9uIDEuMAp1c2IwIGF0
IG9oY2kwOiBVU0IgcmV2aXNpb24gMS4wCnVodWIwIGF0IHVzYjAKdWh1YjA6IEFwcGxlIENv
bXB1dGVyIE9IQ0kgcm9vdCBodWIsIGNsYXNzIDkvMCwgcmV2IDEuMDAvMS4wMCwgYWRkciAx
CnVodWIwOiAyIHBvcnRzIHdpdGggMiByZW1vdmFibGUsIHNlbGYgcG93ZXJlZApvaGNpMSBh
dCBwY2kxIGRldiAyNSBmdW5jdGlvbiAwOiBBcHBsZSBDb21wdXRlciBLZXlMYXJnbyBVU0Ig
Q29udHJvbGxlciAocmV2LiAweDAwKQpvaGNpMTogaW50ZXJydXB0aW5nIGF0IGlycSAyOApv
aGNpMTogT0hDSSB2ZXJzaW9uIDEuMAp1c2IxIGF0IG9oY2kxOiBVU0IgcmV2aXNpb24gMS4w
CnVodWIxIGF0IHVzYjEKdWh1YjE6IEFwcGxlIENvbXB1dGVyIE9IQ0kgcm9vdCBodWIsIGNs
YXNzIDkvMCwgcmV2IDEuMDAvMS4wMCwgYWRkciAxCnVodWIxOiAyIHBvcnRzIHdpdGggMiBy
ZW1vdmFibGUsIHNlbGYgcG93ZXJlZAp1bmlub3J0aDIgYXQgbWFpbmJ1czAKcGNpMiBhdCB1
bmlub3J0aDIgYnVzIDAKcGNpMjogaS9vIHNwYWNlLCBtZW1vcnkgc3BhY2UgZW5hYmxlZApw
Y2hiMiBhdCBwY2kyIGRldiAxMSBmdW5jdGlvbiAwCnBjaGIyOiBBcHBsZSBDb21wdXRlciBV
bmlOb3J0aCBIb3N0LVBDSSBCcmlkZ2UgKHJldi4gMHgwMCkKZndvaGNpMCBhdCBwY2kyIGRl
diAxNCBmdW5jdGlvbiAwOiBMdWNlbnQgVGVjaG5vbG9naWVzIHByb2R1Y3QgMHg1ODExIChy
ZXYuIDB4MDApCmZ3b2hjaTA6IGludGVycnVwdGluZyBhdCBpcnEgNDAKZndvaGNpMDogT0hD
SSAxLjAsIDAwOjAzOjkzOmZmOmZlOjZmOjgyOjJlLCA0MDBNYi9zLCAyMDQ4IG1heF9yZWMs
IDggaXNvX2N0eApnbTAgYXQgcGNpMiBkZXYgMTUgZnVuY3Rpb24gMDogRXRoZXJuZXQgYWRk
cmVzcyAwMDowMzo5Mzo2Zjo4MjoyZQpnbTA6IGludGVycnVwdGluZyBhdCBpcnEgNDEKYnJn
cGh5MCBhdCBnbTAgcGh5IDA6IEJDTTU0MjEgMTAwMEJBU0UtVCBtZWRpYSBpbnRlcmZhY2Us
IHJldi4gMQpicmdwaHkwOiAxMGJhc2VULCAxMGJhc2VULUZEWCwgMTAwYmFzZVRYLCAxMDBi
YXNlVFgtRkRYLCAxMDAwYmFzZVQsIDEwMDBiYXNlVC1GRFgsIGF1dG8KZncwIGF0IGZ3b2hj
aTA6IDAwOjAzOjkzOmZmOmZlOjZmOjgyOjJlOjBhOjAyOjIwOjAwOjAwOjAwOjAwOjAwCnVo
dWIyIGF0IHVodWIwIHBvcnQgMQp1aHViMjogTWl0c3VtaSBFbGVjdHJpYyBIdWIgaW4gQXBw
bGUgRXh0ZW5kZWQgVVNCIEtleWJvYXJkLCBjbGFzcyA5LzAsIHJldiAxLjEwLzEuMjIsIGFk
ZHIgMgp1aHViMjogMyBwb3J0cyB3aXRoIDIgcmVtb3ZhYmxlLCBidXMgcG93ZXJlZAp1aGlk
ZXYwIGF0IHVodWIyIHBvcnQgMSBjb25maWd1cmF0aW9uIDEgaW50ZXJmYWNlIDAKdWhpZGV2
MDogTWl0c3VtaSBFbGVjdHJpYyBBcHBsZSBFeHRlbmRlZCBVU0IgS2V5Ym9hcmQsIHJldiAx
LjEwLzEuMjIsIGFkZHIgMywgaWNsYXNzIDMvMQp1a2JkMCBhdCB1aGlkZXYwOiA4IG1vZGlm
aWVyIGtleXMsIDYga2V5IGNvZGVzCndza2JkMSBhdCB1a2JkMDogY29uc29sZSBrZXlib2Fy
ZCwgdXNpbmcgd3NkaXNwbGF5MAp1aGlkZXYxIGF0IHVodWIyIHBvcnQgMSBjb25maWd1cmF0
aW9uIDEgaW50ZXJmYWNlIDEKdWhpZGV2MTogTWl0c3VtaSBFbGVjdHJpYyBBcHBsZSBFeHRl
bmRlZCBVU0IgS2V5Ym9hcmQsIHJldiAxLjEwLzEuMjIsIGFkZHIgMywgaWNsYXNzIDMvMAp1
aGlkZXYxOiAzIHJlcG9ydCBpZHMKdWhpZDAgYXQgdWhpZGV2MSByZXBvcnRpZCAyOiBpbnB1
dD0xLCBvdXRwdXQ9MCwgZmVhdHVyZT0wCnVoaWQxIGF0IHVoaWRldjEgcmVwb3J0aWQgMzog
aW5wdXQ9Mywgb3V0cHV0PTAsIGZlYXR1cmU9MApib290IGRldmljZTogd2QwCnJvb3Qgb24g
d2QwYSBkdW1wcyBvbiB3ZDBiCg==
--------------020702010207000100070201--