bin/49113: Multiple NFS tests randomly failing since Aug 11

[gson%gson.org@localhost (Andreas Gustafsson)]

>Number:         49113
>Category:       bin
>Synopsis:       Multiple NFS tests randomly failing since Aug 11
>Confidential:   no
>Severity:       serious
>Priority:       high
>Responsible:    bin-bug-people
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Thu Aug 14 12:00:00 +0000 2014
>Originator:     Andreas Gustafsson
>Release:        NetBSD-current, source date >= 2014-08-11
>Organization:

>Environment:
System: NetBSD
Architecture: i386
Machine: i386
>Description:

At least 15 different NFS related test cases have failed one or more
times on the TNF testbed since Aug 11, the day of the 7.0 branch.
Before that, they were passing reliably.

In each of the failing tests, the standard error stream contains a
bunch of messages like:

  t_vnops: rpcbind to nfs on server: RPC: Port mapper failure - RPC: Timed out
  t_vnops: rpcbind to nfs on server: RPC: Port mapper failure - RPC: Timed out
  t_vnops: rpcbind to nfs on server: RPC: Port mapper failure - RPC: Timed out
  t_vnops: rpcbind to nfs on server: RPC: Port mapper failure - RPC: Timed out
  t_vnops: rpcbind to nfs on server: RPC: Port mapper failure - RPC: Timed out
  t_vnops: rpcbind to nfs on server: RPC: Port mapper failure - RPC: Timed out
  t_vnops: rpcbind to nfs on server: RPC: Port mapper failure - RPC: Timed out
  t_vnops: rpcbind to nfs on server: RPC: Port mapper failure - RPC: Timed out
  t_vnops: rpcbind to nfs on server: RPC: Port mapper failure - RPC: Timed out

The first recorded failures are with CVS source date
2014.08.11.05.19.16 (no HTML test report is available as the tests
failed to run to completion) and 2014.08.11.13.22.16, HTML report here:

  
http://releng.netbsd.org/b5reports/i386/build/2014.08.11.13.22.16/test.html#fs_vfs_t_vnops_nfs_rename_reg_nodir

Determining the exact commit that triggered the problem is difficult
because it happened around or during a time when more than 1000 test
cases, including these, were failing for presumably unrelated reasons.

To illustrate, here is an ASCII table summarizing the results of the
affected tests from the last fifty i386 test runs.  Each column of
dashes and X:es represents a test run, with the most recent run at the
right.  A passing test  case is shown as a dash, and a failing one as "X":

----------------------------------XX--X--------X--  fs/vfs/t_io:nfs_extendfile
----------------------------------XX-------X------  
fs/vfs/t_io:nfs_read_after_unlink
----------------------------------XX----------X---  fs/vfs/t_io:nfs_shrinkfile
----------------------------------XX-----X--------  fs/vfs/t_ro:nfs_attrs
----------------------------------XX-----------X--  
fs/vfs/t_ro:nfs_createsymlink
----------------------------------XX----------X---  fs/vfs/t_unpriv:nfs_flags
----------------------------------XXX-------------  fs/vfs/t_unpriv:nfs_owner
----------------------------------XX-X------------  fs/vfs/t_vfsops:nfs_tsync
----------------------------------XX-----------X--  
fs/vfs/t_vnops:nfs_access_simple
----------------------------------XX----------X---  
fs/vfs/t_vnops:nfs_create_nametoolong
----------------------------------XX-------X------  
fs/vfs/t_vnops:nfs_dir_simple
----------------------------------XX---X----------  
fs/vfs/t_vnops:nfs_fcntl_lock
----------------------------------XX--------X-----  
fs/vfs/t_vnops:nfs_rename_dir
----------------------------------XX-------X------  
fs/vfs/t_vnops:nfs_rename_nametoolong
----------------------------------XXX-------------  
fs/vfs/t_vnops:nfs_rename_reg_nodir

The columns with X:es all the way through are from the time when 1000+
tests where failing on Aug 10 - 11.

>How-To-Repeat:

Run the ATF tests.  Look for failures in test cases with "nfs" in
their name.

>Fix: