NFS weirdness on EdgeRouter Lite (aka ERLite3)

[Alexander Schreiber <als%thangorodrim.ch@localhost>]

Hi,

I noticed that the Ubiquiti EdgeRouter Lite appears to be supported by
NetBSD, so I grabbed the minimal system image from
 https://cdn.netbsd.org/pub/NetBSD/NetBSD-11.0/evbmips-mips64eb/binary/gzimg/octeon.img.gz
wrote to a suitable USB stick and started testing.

It booted nicely, found both CPUs but disabled cpu1 due to UP kernel, ok.
So I installed comp.tgz, grabbed 11.0 RELEASE sources to build me
variant of OCTEON with MULTIPROCESSOR. Note: ./build.sh tools on this
machine should only be attempted with a fast USB drive (e.g. fast SSD),
because the genautomata step of the gcc build eats ~1.2G of RAM which
translates into heavy flash use - a fast USB SSD gets the build time for
tools down ~30h, vs several days with a slow one.

Booted the new MP kernel and it came up fine:
  https://dmesgd.nycbug.org/dmesgd?do=view&id=9124

Time to stress test it. I decided to do a ./build.sh distribution for
starters, with /usr/src and /usr/obj on NFS (USB not being the fastest).

Unpacked sources on NFS server, mounted NFS filesystems, started build.sh,
which eventually failed with: compiler cannot create executables .. what?

Wrote the canonical short hello.c to /usr/obj, ran gcc and it failed ...
because the just written hello.c (via vi) was 0 bytes long. WAT?

A whole series of tests ensued:

First, I rebooted into the mkrepo kernel to exclude any nonsense during
the local build process (unlikely, but just in case), uname -a:
 NetBSD octeon 11.0 NetBSD 11.0 (OCTEON) #0: Thu Jul 30 15:23:12 UTC 2026  mkrepro%mkrepro.NetBSD.org@localhost:/usr/src/sys/arch/evbmips/compile/OCTEON evbmips

Copying a short file to NFS (/etc/fstab being my canocical example for this)
resulted in a 0 byte file. Copying a larger file (/boot/netbsd) resulted
in a file of correct size and content (md5 check on both client and NFS
server agreed). Running ktrace on the small file cp command showed the
write being handed to the kernel and succeeding.

Ok, small test programm:

---------------------- snip ----------------------
#include <stdio.h>
#include <unistd.h>
#include <string.h>
#include <fcntl.h>

int main (void) {
  char message[] =  "This is a short text.\n";
  int fd;
  int result;
  size_t msg_len;
  ssize_t written;

  msg_len = strlen(message);

  printf("starting ...\n");
  fd = open("writer.output", O_CREAT|O_WRONLY, 0644);

  if (fd == -1) {
    printf("file creation failed, aborting");
    return 1;
  }

  printf("writing message ...\n");
  written = write(fd, message, msg_len);
  if (written == msg_len) {
    printf("message written ...\n");
  } else {
    printf("tried to write %d bytes, wrote %d bytes\n", msg_len, written);
    perror("write");
  }

  result = 0;

  printf("syncing ..\n");
  result = fsync(fd);
  if (result == 0) {
    printf("synced ...\n");
  } else {
    printf("sync failed ...\n");
  }

  result = close(fd);

  return(0);
}
---------------------- snip ----------------------

results:
 - NFS server NetBSD amd64 backed by ZFS, client erlite3 NetBSD:
   - 0 byte file
 - NFS server Linux amd64 backed by ext4, client erlite3 NetBSD:
   - 0 byte file
 - NFS server Linix amd64 backed by ext4, client erlite3 NetBSD:
   - 22 bytes file (as expected)
 - erlite3 NetBSD on local disk (both msdos and ffs on USB stick):
   - 22 bytes file (as expected)

Zero errors reported in all three cases.

So I tested blocksizes with dd if=/dev/zero count=X blocksize=Y of=X_Y
 - up to a blocksize of 8191 bytes: all writes eaten by the kernel,
   zero byte files created
 - blocksize of 8192 and up: files of expected size are created
 - writing 131088 blocks of size 8191 (so, ~1GB file on a 512M machine,
   to try to force a buffer cache spill): 0 byte file created

Any blocksize < 8192 on NFS writes: empty file, blocksize > 8192: file
of expected size.

I also tcpdumped the network traffic between the client and the NFS server.
NFS server was NetBSD/amd64.

NFS client erlite3, small file:
     - V3 LOOKUP -> NFS3ERR_NOENT (expected)
     - V3 ACCESS -> Allowed
     - V3 CREATE -> OK
     - the end
NFS client NetBSD sparc64, small file:
     - V3 LOOKUP -> NFS3ERR_NOENT (expected)
     - V3 ACCESS -> Allowed
     - V3 CREATE -> OK
     - V3 WRITE -> OK, all bytes written
     - the end

The erlite3 kernel just ... eats and discards the write?

By default, NetBSD mounts the NFS filesystems (I mounted in TCP mode) with
rsize=wsize=8192, umounting and mounting again with rsize=wsize=1024 had
no effect.

The only _correlation_ I found is the page size of the machine, which
happens to match the "magic" 8192 byte boundary (hw.pagesize = 8192).

Does anybody have any ideas what might be going on here? Am I just holding
it wrong in a particularly weird way?

Kind regards,
           Alex.
-- 
"Opportunity is missed by most people because it is dressed in overalls and
 looks like work."                                      -- Thomas A. Edison