snprintf(3) behaviour regarding large "n"

[Steffen Nurpmeso <sdaoden%yandex.com@localhost>]

Hello NetBSD,

i recently had a hard time because NetBSD was the only operating
system that fails to pass the testsuite (S/MIME function
verification, and only this test!) of the S-nail mailer
i maintain.

Short: the problem is that i use snprintf(3) with a "size_t n"
argument of UI32_MAX (is EQ to UINT32_MAX).
Well, looking at POSIX this is even somewhat correct (but then it
should be EOVERFLOW for conformance), but it is (a) neither
documented in the manual nor (b) would a halfway sane person do it
like that -- :) --, though i'd agree that i should have used
SIZE_MAX to indicate what i had in mind ("buffer _is_ large
enough"), not at last because there is an error defined for when
_the return value_ would exceed INT_MAX (also EOVERFLOW).

The patch has not been compile tested (regarding availability of
INT_MAX), sorry (very small resources here).

--steffen

--- lib_libc_stdio_vsnprintf.c.orig	2014-09-29 14:46:55.000000000 +0200
+++ lib_libc_stdio_vsnprintf.c	2014-09-29 14:51:15.000000000 +0200
@@ -76,10 +76,8 @@ vsnprintf_l(char *str, size_t n, locale_
 	_DIAGASSERT(n == 0 || str != NULL);
 	_DIAGASSERT(fmt != NULL);
 
-	if ((int)n < 0) {
-		errno = EINVAL;
-		return -1;
-	}
+	if ((int)n < 0)
+		n = INT_MAX;
 
 	_FILEEXT_SETUP(&f, &fext);
 	f._file = -1;