Subject: NetBSD/Solaris NFS read/write problem
To: None <port-i386@NetBSD.ORG>
From: Kent Polk <kent@tiamat.goathill.org>
List: port-i386
Date: 05/13/1999 15:55:42
I've been running an NFS mount from a Solaris 5.7 host to a NetBSD
1.3K #2: Fri Mar 19 15:47:24 CST 1999 (beta release) NFS client
for a few months now with no problems until last week.
Reads and writes from the clinet to the NFS server began to take
longer and longer, with typical 20k writes taking in excess of a
minute to complete and reads often taking almost as long. Rebooting
either host didn't fix anything, however, the problem gradually
got better as the sysadmin began to investigate (what's new?)
Now, I've been running with this problem for about 24 hours and we
can't get it to go away this time. It appears that there are
problems with UDP sockets, but I don't really know how to investigate
this. I've included netstat -s dumps from both hosts. If you have
any suggestions about how to proceed, I'd sure appreciate them.
The netstat dumps are after rebooting both hosts and completing
one cd to the remote fs and one file copy (cp) from the remote
directory to the same remote directory using the client (a read
and a write).
Thanks
--------------
Netbsd host:
$ netstat -s
ip:
181 total packets received
0 bad header checksums
0 with size smaller than minimum
0 with data size < data length
0 with length > max ip packet size
0 with header length < data size
0 with data length < header length
0 with bad options
0 with incorrect version number
16 fragments received 0 fragments dropped (dup or out of space)
0 malformed fragments dropped
0 fragments dropped after timeout
4 packets reassembled ok
167 packets for this host
2 packets for unknown/unsupported protocol
0 packets forwarded (0 packets fast forwarded)
0 packets not forwardable
0 redirects sent
141 packets sent from this host
0 packets sent with fabricated ip header
0 output packets dropped due to no bufs, etc.
0 output packets discarded due to no route
3 output datagrams fragmented
15 fragments created
0 datagrams that can't be fragmented
icmp:
2 calls to icmp_error
0 errors not generated because old message was icmp
Output histogram:
destination unreachable: 2
0 messages with bad code fields
0 messages < minimum length
0 bad checksums
0 messages with bad length
Input histogram:
destination unreachable: 2
0 message responses generated
igmp:
0 messages received
0 messages received with too few bytes
0 messages received with bad checksum
0 membership queries received
0 membership queries received with invalid field(s)
0 membership reports received
0 membership reports received with invalid field(s)
0 membership reports received for groups to which we belong
0 membership reports sent
tcp:
16 packets sent
8 data packets (168 bytes)
0 data packets (0 bytes) retransmitted
4 ack-only packets (10 delayed)
0 URG only packets
0 window probe packets
0 window update packets
4 control packets
20 packets received
10 acks (for 170 bytes)
2 duplicate acks
0 acks for unsent data
10 packets (1188 bytes) received in-sequence
0 completely duplicate packets (0 bytes)
0 old duplicate packets
0 packets with some dup. data (0 bytes duped)
2 out-of-order packets (0 bytes)
0 packets (0 bytes) of data after window
0 window probes
0 window update packets
0 packets received after close
0 discarded for bad checksums
0 discarded for bad header offset fields
0 discarded because packet too short
2 connection requests
0 connection accepts
2 connections established (including accepts)
1 connection closed (including 0 drops)
0 embryonic connections dropped
12 segments updated rtt (of 12 attempts)
0 retransmit timeouts
0 connections dropped by rexmit timeout
0 persist timeouts (resulting in 0 dropped connections)
0 keepalive timeouts
0 keepalive probes sent
0 connections dropped by keepalive
0 correct ACK header predictions
6 correct data packet header predictions
0 PCB hash misses
0 dropped due to no socket
0 connections drained due to memory shortage
0 bad connection attempts
0 SYN cache entries added
0 hash collisions
0 completed
0 aborted (no space to build PCB)
0 timed out
0 dropped due to overflow
0 dropped due to bucket overflow
0 dropped due to RST
0 dropped due to ICMP unreachable
0 duplicate SYNs received for entries already in the cache
0 SYNs dropped (no route or no space)
udp:
147 datagrams received
0 with incomplete header
0 with bad data length field
0 with bad checksum
2 dropped due to no socket
24 broadcast/multicast datagrams dropped due to no socket
0 dropped due to full socket buffers
121 delivered
106 PCB hash misses
123 datagrams output
ddp:
0 packets with short headers
0 packets with long headers
0 packets with no checksum
0 packets too short
0 packets with bad checksum
0 packets with not enough data
0 packets forwarded
0 packets encapsulated
0 packets rcvd for unreachable dest
0 packets dropped due to no socket space
idp:
spp:
NS error statistics:
tp:
Receiving:
0 variable parameters ignored
0 invalid parameter codes
0 invalid parameter values
0 invalid dutypes
0 negotiation failures
0 invalid destination references
0 invalid suffix parameters
0 invalid length
0 invalid checksums
0 DTs out of order
0 DTs not in window
0 duplicate DTs
0 XPDs not in window
0 XPDs w/o credit to stash
0 times local credit reneged
0 concatenated TPDUs
Sending:
0 XPD marks discarded
XPD stopped data flow 0 times
0 times foreign window closed
Miscellaneous:
0 small mbufs
0 clusters
0 source quench
0 dec bits
M:L ( M mbuf chains of length L)
0: over 16
0: 1 0: 2
0: 2 0: 4
0: 3 0: 6
0: 4 0: 8
0: 5 0: 10
0: 6 0: 12
0: 7 0: 14
0: 8 0: 16
0 EOT rcvd
0 EOT sent
0 EOT indications
Connections:
0 connections used extended format
0 connections allowed transport expedited data
0 connections turned off checksumming
0 connections dropped due to retrans limit
0 tp 4 connections
0 tp 0 connections
Round trip times, listed in ticks:
Category Smoothed avg | Deviation | Deviation/Avg
~LOCAL, PDN: 0 | 0 | 0 | 0
~LOCAL,~PDN: 0 | 0 | 0 | 0
LOCAL,~PDN: 0 | 0 | 0 | 0
LOCAL, PDN: 0 | 0 | 0 | 0
Tpdus RECVD [0 valid, 0.000000 % of total (0); 0 dropped]
DT 0 AK 0 DR 0 CR 0
XPD 0 XAK 0 DC 0 CC 0 ER 0
Tpdus SENT [0 total, 0 dropped]
DT 0 AK 0 DR 0 CR 0
XPD 0 XAK 0 DC 0 CC 0 ER 0
Retransmissions:
CR 0 CC 0 DR 0
DT 0 ( 0.00%)
XPD 0 ( 0.00%)
E Timers: [ 0 ticks]
0 timers set 0 timers expired 0 timers cancelled
C Timers: [ 1458 ticks]
0 timers set 0 timers expired 0 timers cancelled
0 inactive timers cancelled
Pathological debugging activity:
0 CCs sent to zero dref
0 random DTs dropped
0 illegally large XPD TPDUs
0 faked reneging of cdt
ACK reasons:
0 not acked immediately
0 strategy==each
0 strategy==fullwindow
0 duplicate DT
0 EOTSDU
0 reordered DT
0 user rcvd
0 fcc reqd
cltp:
0 incomplete headers
0 bad data length fields
0 bad checksums
clnp:
0 total packets sent
0 total fragments sent
0 total packets received
0 with fixed part of header too small
0 with header length not reasonable
0 incorrect checksums
0 with unreasonable address lengths
0 with forgotten segmentation information
0 with an incorrect protocol identifier
0 with an incorrect version
0 dropped because the ttl has expired
0 clnp cache misses
0 clnp congestion experience bits set
0 clnp congestion experience bits received
esis:
0 esh sent, 0 esh received
0 ish sent, 0 ish received
0 rd sent, 0 rd received
0 pdus not sent due to insufficient memory
0 pdus received with bad checksum
0 pdus received with bad version number
0 pdus received with bad type field
0 short pdus received
--------------
Solaris host:
$ netstat -s
UDP
udpInDatagrams = 1545 udpInErrors = 0
udpOutDatagrams = 1548
TCP tcpRtoAlgorithm = 4 tcpRtoMin = 200
tcpRtoMax = 60000 tcpMaxConn = -1
tcpActiveOpens = 9 tcpPassiveOpens = 6
tcpAttemptFails = 0 tcpEstabResets = 1
tcpCurrEstab = 11 tcpOutSegs = 1737
tcpOutDataSegs = 310 tcpOutDataBytes = 39012
tcpRetransSegs = 2 tcpRetransBytes = 240
tcpOutAck = 1426 tcpOutAckDelayed = 47
tcpOutUrg = 0 tcpOutWinUpdate = 0
tcpOutWinProbe = 0 tcpOutControl = 21
tcpOutRsts = 3 tcpOutFastRetrans = 0
tcpInSegs = 3194
tcpInAckSegs = 290 tcpInAckBytes = 38951
tcpInDupAck = 15 tcpInAckUnsent = 0
tcpInInorderSegs = 3109 tcpInInorderBytes =4233064
tcpInUnorderSegs = 0 tcpInUnorderBytes = 0
tcpInDupSegs = 3 tcpInDupBytes = 3020
tcpInPartDupSegs = 0 tcpInPartDupBytes = 0
tcpInPastWinSegs = 0 tcpInPastWinBytes = 0
tcpInWinProbe = 0 tcpInWinUpdate = 0
tcpInClosed = 1 tcpRttNoUpdate = 2
tcpRttUpdate = 285 tcpTimRetrans = 2
tcpTimRetransDrop = 0 tcpTimKeepalive = 0
tcpTimKeepaliveProbe= 0 tcpTimKeepaliveDrop = 0
tcpListenDrop = 0 tcpListenDropQ0 = 0
tcpHalfOpenDrop = 0 tcpOutSackRetrans = 0
IP ipForwarding = 2 ipDefaultTTL = 255
ipInReceives = 3474 ipInHdrErrors = 0
ipInAddrErrors = 0 ipInCksumErrs = 0
ipForwDatagrams = 0 ipForwProhibits = 0
ipInUnknownProtos = 0 ipInDiscards = 0
ipInDelivers = 4739 ipOutRequests = 1883
ipOutDiscards = 0 ipOutNoRoutes = 0
ipReasmTimeout = 60 ipReasmReqds = 27
ipReasmOKs = 27 ipReasmFails = 0
ipReasmDuplicates = 0 ipReasmPartDups = 0
ipFragOKs = 4 ipFragFails = 0
ipFragCreates = 16 ipRoutingDiscards = 0
tcpInErrs = 0 udpNoPorts = 55
udpInCksumErrs = 0 udpInOverflows = 0
rawipInOverflows = 0
ICMP icmpInMsgs = 3 icmpInErrors = 0
icmpInCksumErrs = 0 icmpInUnknowns = 0
icmpInDestUnreachs = 3 icmpInTimeExcds = 0
icmpInParmProbs = 0 icmpInSrcQuenchs = 0
icmpInRedirects = 0 icmpInBadRedirects = 0
icmpInEchos = 0 icmpInEchoReps = 0
icmpInTimestamps = 0 icmpInTimestampReps = 0
icmpInAddrMasks = 0 icmpInAddrMaskReps = 0
icmpInFragNeeded = 0 icmpOutMsgs = 4
icmpOutDrops = 0 icmpOutErrors = 0
icmpOutDestUnreachs = 4 icmpOutTimeExcds = 0
icmpOutParmProbs = 0 icmpOutSrcQuenchs = 0
icmpOutRedirects = 0 icmpOutEchos = 0
icmpOutEchoReps = 0 icmpOutTimestamps = 0
icmpOutTimestampReps= 0 icmpOutAddrMasks = 0
icmpOutAddrMaskReps = 0 icmpOutFragNeeded = 0
icmpInOverflows = 0
IGMP:
0 messages received
0 messages received with too few bytes
0 messages received with bad checksum
0 membership queries received
0 membership queries received with invalid field(s)
0 membership reports received
0 membership reports received with invalid field(s)
0 membership reports received for groups to which we belong
0 membership reports sent