Re: MicroVAX 3100/20 is apocalyptically slow even with 32MB of RAM

[Daniel Seagraves <dseagrav%lunar-tokyo.net@localhost>]

A discussion on discord resulted in a lot of testing and experimentation over the past several days, and this is the present status of the investigation into losing wall clock time / general performance degradation of NetBSD/vax.

Previously, it was observed that NetBSD 10.1 was extremely slow on a 3100/20 with 32MB of RAM and a fast (physical) SCSI disk. Further testing showed this behavior not apparent in NetBSD 4.0.1 but present in NetBSD 9.1, with the releases in between not being tested because the GENERIC kernels for them would not boot on my 3100.

Since the issue involves losing wall clock time, and thus presumably clock interrupts, it is assumed that in-kernel profiling will not produce valid results since the timing information will be skewed by the issue.

Since then, it has been determined that the issue isn’t strictly SCSI related as originally suspected. Under NetBSD 9.1, equal-size transfers made using dd from a file located on an NFS mount to /dev/null and from /dev/rsd0c to /dev/null lost approximately the same amount of wall clock time. A transfer from /dev/zero to /dev/null of the same size as the previous two tests lost approximately half as much time.

A second 3100 (having only 16MB of RAM) was similarly tested and showed the same results, so this is unlikely to be a hardware issue.

The first 3100 (the one with 32MB of RAM) was reinstalled with NetBSD 4.0.1 and the tests were repeated there. No appreciable loss of wall clock time was observed. Notably, the percentage of time spent in system during the dd from /dev/zero to /dev/null was much higher than that observed under NetBSD 9.1 - staying between 85% and 92% on 4.0.1 - leading us to believe that simply being in system state did not cause the issue.

Thus it was concluded that 4.0.1 did not exhibit the behavior, but then when I made a CVS checkout of the NetBSD 4 source tree on the NFS host (to save myself having to ssh from the VAX) and mv’d it to the usr filesystem, 70 seconds of time was lost moving the src directory and 30 seconds of time was lost moving the xsrc directory. Clearly some part of the issue is still there. A dd from /dev/zero to the SCSI disk was performed to see if SCSI *writes* were the problem, but it did not lose any time.

I started a build of tools after that, and the system has not yet lost a significant amount of wall clock time. It lost a little under 2 seconds in the first 24 hours.

I plan to finish an installation of 4.0.1 on the smaller VAX and try this same transfer of src and xsrc again to verify it exhibits the same lossage. If it does, then I want to try packing src / xsrc into a tar file and extracting it on the VAX to see if the filesystem operations are the issue. If nothing makes sense, then I will start an identical build on the smaller 3100 to see if the increased paging activity causes any divergence vs the bigger one.