Re: lib/56148: lib/libc/stdio/t_printf:snprintf_float test randomly fails

[Andreas Gustafsson <gson%gson.org@localhost>]

The following reply was made to PR lib/56148; it has been noted by GNATS.

From: Andreas Gustafsson <gson%gson.org@localhost>
To: gnats-bugs%netbsd.org@localhost
Cc: christos%NetBSD.org@localhost
Subject: Re: lib/56148: lib/libc/stdio/t_printf:snprintf_float test randomly fails
Date: Sun, 9 May 2021 19:37:33 +0300

 There are multiple layers of bugs here, and some of them are canceling
 each other out.
 
 First, there's the bug Christos already fixed that caused snprintf()
 to sometimes segfault when malloc() failed.
 
 Second, for that to happen, malloc() had to fail in the first place.
 The intent of the test seems to be to detect memory leaks by running
 out of memory only when there is a leak, but it's actually running out
 of memory many times in each run, often even in the first call to
 snprintf() when nothing has had a chance to leak yet.  So either the
 test is buggy in that the limits it sets for itself using setrlimit()
 are unreasonably low, or if you think the limits are reasonable, there
 is some other bug causing malloc() to fail regardless.
 
 Third, since the test is always running out of memory and is designed
 to fail when that happens, one might think it would always fail, but
 in actuality, it now never fails, at least not when compiled with the
 default optimization setting of -O2.  It does, however, fail if
 compiled with -O1.
 
 I believe this is because the test assumes that snprintf() will return
 -1 when malloc() fails, and the NetBSD implementation does behave that
 way, but this behavior is not allowed by C99 (the N1256 draft; I don't
 have a copy of the final standard), nor by POSIX; neither shows "out
 of memory" as a possible error for snprintf().  The gcc
 -fprintf-return-value optimization (which is enabled by default) then
 takes advantage of the knowledge that the call can't fail, computes
 the return value of snprintf() at compile time, and optimizes away the
 entire ATF_CHECK() call that would otherwise fail.
 
 Here, I think the bug is that snprintf() is using malloc() at all.
 For reference, there is a similar bug in glibc:
 
   https://bugzilla.redhat.com/show_bug.cgi?id=441945
 
 -- 
 Andreas Gustafsson, gson%gson.org@localhost