Re: misc/57291

[Jim Spath <jspath55%gmail.com@localhost>]

The following reply was made to PR misc/57291; it has been noted by GNATS.

From: Jim Spath <jspath55%gmail.com@localhost>
To: gnats-bugs%netbsd.org@localhost
Cc: 
Subject: Re: misc/57291
Date: Tue, 2 May 2023 14:26:07 -0400

 I have run more tests, as promised, and reviewed the code and some history.
 > I will run more tests.
 
 David Holland noted:
 >  it's not clear that we should have tests that are so large they won't run on platforms we might reasonably want to test on.
 
 The regex exhaust test has hard-coded size limits of 9,999 runs (or
 "REGEX_MAXSIZE" factors), and there are limits of 256MB memory and a
 600 second/10 minute run-time.
 
 >        atf_tc_set_md_var(tc, "timeout", "600");
 >        atf_tc_set_md_var(tc, "require.memory", "256M");
 
 The Raspberry Pi 02W has 512MB memory, though the available space is
 closer to 400MB. My testing showed that different runs require varying
 amounts of memory, so even a 400MB system with no swap will sometimes
 run a successful test.
 
 Earlier versions of the t_exhaust.c code had a lower memory
 requirement. I found this 2011-era code archived:
 
 http://web.mit.edu/freebsd/head/contrib/netbsd-tests/lib/libc/regex/t_exhaust.c
 
 >      atf_tc_set_md_var(tc, "timeout", "600");
 > #if defined(__FreeBSD__)
 >     atf_tc_set_md_var(tc, "require.memory", "64M");
 > #else
 >     atf_tc_set_md_var(tc, "require.memory", "120M");
 > #endif
 
 Meanwhile, the FreeBSD version of this test had a "skip i386" switch
 added, for similar reasons of running out of memory. See:
 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=237450
 and
 https://mail-archive.freebsd.org/cgi/getmsg.cgi?fetch=8900+0+archive/2021/freebsd-testing/20211203.freebsd-testing
 
 My test summary (architect/memory):
 i386 3GB - 100% pass
 pi4:arm64 8GB - 100% pass
 pi3:arm64 1GB - 90% pass
 pi0:arm32 512MB - 10% pass (no swap)
       512MB - 90% pass (small swap file)
 
 In my view, intermittent failures of this test are okay given the
 variable memory requirements. I would rather see what happens when
 memory is exhausted for one process than not look. I don't think the
 test iterations/size should be altered given the history of test runs
 with these set conditions.
 
 Given the size capacity yet lack of write performance on SD cards it
 makes sense to not have a swap file or use SD storage for temp space.
 I would suggest added comments in the test code with real world
 examples of run time and space requirements for some common small
 footprint systems).
 
 I would be okay skipping this test on systems with no swap (yes, some
 valid systems might be ignored but this would prevent false negatives
 of a sort). I can't find a good C example so here is a shell
 equivalent;
 
 skip_if_noswap()
 {
         if /sbin/swapctl -l >/dev/null 2>&1
         then
             atf_skip "current platform shows no swap configured"
         fi
 }