Re: NFS weirdness on EdgeRouter Lite (aka ERLite3)

[Alexander Schreiber <als%thangorodrim.ch@localhost>]

On Thu, Sep 03, 2026 at 01:47:50PM +0200, Alexander Schreiber wrote:
> Hi,
> 
> I noticed that the Ubiquiti EdgeRouter Lite appears to be supported by
> NetBSD, so I grabbed the minimal system image from
>  https://cdn.netbsd.org/pub/NetBSD/NetBSD-11.0/evbmips-mips64eb/binary/gzimg/octeon.img.gz
> wrote to a suitable USB stick and started testing.
> 
> It booted nicely, found both CPUs but disabled cpu1 due to UP kernel, ok.
> So I installed comp.tgz, grabbed 11.0 RELEASE sources to build me
> variant of OCTEON with MULTIPROCESSOR. Note: ./build.sh tools on this
> machine should only be attempted with a fast USB drive (e.g. fast SSD),
> because the genautomata step of the gcc build eats ~1.2G of RAM which
> translates into heavy flash use - a fast USB SSD gets the build time for
> tools down ~30h, vs several days with a slow one.
> 
> Booted the new MP kernel and it came up fine:
>   https://dmesgd.nycbug.org/dmesgd?do=view&id=9124
> 
> Time to stress test it. I decided to do a ./build.sh distribution for
> starters, with /usr/src and /usr/obj on NFS (USB not being the fastest).
> 
> Unpacked sources on NFS server, mounted NFS filesystems, started build.sh,
> which eventually failed with: compiler cannot create executables .. what?
> 
> Wrote the canonical short hello.c to /usr/obj, ran gcc and it failed ...
> because the just written hello.c (via vi) was 0 bytes long. WAT?
> 
> A whole series of tests ensued:
> 
> First, I rebooted into the mkrepo kernel to exclude any nonsense during
> the local build process (unlikely, but just in case), uname -a:
>  NetBSD octeon 11.0 NetBSD 11.0 (OCTEON) #0: Thu Jul 30 15:23:12 UTC 2026  mkrepro%mkrepro.NetBSD.org@localhost:/usr/src/sys/arch/evbmips/compile/OCTEON evbmips
> 
> Copying a short file to NFS (/etc/fstab being my canocical example for this)
> resulted in a 0 byte file. Copying a larger file (/boot/netbsd) resulted
> in a file of correct size and content (md5 check on both client and NFS
> server agreed). Running ktrace on the small file cp command showed the
> write being handed to the kernel and succeeding.
> 
> Ok, small test programm:
> 
> ---------------------- snip ----------------------
> #include <stdio.h>
> #include <unistd.h>
> #include <string.h>
> #include <fcntl.h>
> 
> int main (void) {
>   char message[] =  "This is a short text.\n";
>   int fd;
>   int result;
>   size_t msg_len;
>   ssize_t written;
> 
>   msg_len = strlen(message);
> 
>   printf("starting ...\n");
>   fd = open("writer.output", O_CREAT|O_WRONLY, 0644);
> 
>   if (fd == -1) {
>     printf("file creation failed, aborting");
>     return 1;
>   }
> 
>   printf("writing message ...\n");
>   written = write(fd, message, msg_len);
>   if (written == msg_len) {
>     printf("message written ...\n");
>   } else {
>     printf("tried to write %d bytes, wrote %d bytes\n", msg_len, written);
>     perror("write");
>   }
> 
>   result = 0;
> 
>   printf("syncing ..\n");
>   result = fsync(fd);
>   if (result == 0) {
>     printf("synced ...\n");
>   } else {
>     printf("sync failed ...\n");
>   }
> 
>   result = close(fd);
> 
>   return(0);
> }
> ---------------------- snip ----------------------
> 
> results:
>  - NFS server NetBSD amd64 backed by ZFS, client erlite3 NetBSD:
>    - 0 byte file
>  - NFS server Linux amd64 backed by ext4, client erlite3 NetBSD:
>    - 0 byte file
>  - NFS server Linix amd64 backed by ext4, client erlite3 NetBSD:
>    - 22 bytes file (as expected)

Oops, that should have been:
  - NFS server Linux amd64 backed by ext4, client sparc64 NetBSD:
    - 22 bytes file (as expected)

(sparc64 NetBSD, running 11.0 because that's what I had to hand)

>  - erlite3 NetBSD on local disk (both msdos and ffs on USB stick):
>    - 22 bytes file (as expected)
> 
> Zero errors reported in all three cases.
> 
> So I tested blocksizes with dd if=/dev/zero count=X blocksize=Y of=X_Y
>  - up to a blocksize of 8191 bytes: all writes eaten by the kernel,
>    zero byte files created
>  - blocksize of 8192 and up: files of expected size are created
>  - writing 131088 blocks of size 8191 (so, ~1GB file on a 512M machine,
>    to try to force a buffer cache spill): 0 byte file created
> 
> Any blocksize < 8192 on NFS writes: empty file, blocksize > 8192: file
> of expected size.
> 
> I also tcpdumped the network traffic between the client and the NFS server.
> NFS server was NetBSD/amd64.
> 
> NFS client erlite3, small file:
>      - V3 LOOKUP -> NFS3ERR_NOENT (expected)
>      - V3 ACCESS -> Allowed
>      - V3 CREATE -> OK
>      - the end
> NFS client NetBSD sparc64, small file:
>      - V3 LOOKUP -> NFS3ERR_NOENT (expected)
>      - V3 ACCESS -> Allowed
>      - V3 CREATE -> OK
>      - V3 WRITE -> OK, all bytes written
>      - the end
> 
> The erlite3 kernel just ... eats and discards the write?
> 
> By default, NetBSD mounts the NFS filesystems (I mounted in TCP mode) with
> rsize=wsize=8192, umounting and mounting again with rsize=wsize=1024 had
> no effect.
> 
> The only _correlation_ I found is the page size of the machine, which
> happens to match the "magic" 8192 byte boundary (hw.pagesize = 8192).
> 
> Does anybody have any ideas what might be going on here? Am I just holding
> it wrong in a particularly weird way?
> 
> Kind regards,
>            Alex.
> -- 
> "Opportunity is missed by most people because it is dressed in overalls and
>  looks like work."                                      -- Thomas A. Edison

-- 
"Opportunity is missed by most people because it is dressed in overalls and
 looks like work."                                      -- Thomas A. Edison