aarch64 reliability issues

[John Klos <john%klos.com@localhost>]

Hi, all,

I'm running a decent number of aarch64 systems, several in production.

The lower memory systems (512 megs, 1 gig) haven't had any issues with 
running for ages while being quite busy.

Higher memory systems (4, 8, 16 gigs) have had issues. The most 
problematic issue is that when left open to public ssh, two have become 
unresponsive to ssh even when ICMP was still showing the systems were up. 
This wasn't an issue of ssh rate limiting.

This has happened when the system has been otherwise idle (although 
possibly when running daily / weekly scripts).

I was logged in to one when this happened recently. It was also running 
pkg_rolling-replace as well. I hit the up arrow to bring up a previous 
command, hit return, and got this:

...
   Compiling scoped-tls v1.0.1
   Compiling typed-arena v2.0.2
   Compiling hir-ty v0.0.0 (/usr/pkgsrc/lang/rust/work/rustc-1.85.1-src/src/tools/rust-analyzer/crates/hir-ty)
shamshu: {12} tail /root/rolling.log

[ 8888900.8992799] /: create/symlink failed, no inodes free
Connection to shamshu.zia.io closed by remote host.
Connection to shamshu.zia.io closed.

The machine still answered ICMP, but:

ssh shamshu.zia.io
Connection closed by 108.66.216.115 port 22

No filesystem has issues with free inodes:

Filesystem   1G-blocks      Used     Avail %Cap      iUsed     iAvail %iCap Mounted on
/dev/dk4          4605      3232      1142  73%    5346580  145672426    3% /
/dev/dk0             0         0         0  29%        512          0  100% /boot
kernfs               0         0         0 100%          0          0    0% /kern
ptyfs                0         0         0 100%          0          0    0% /dev/pts
procfs               0         0         0 100%         28       1016    2% /proc
tmpfs                0         0         0   0%          1     835418    0% /var/shm

So... any thoughts or ideas about what might be going on here?

John