lib/57828: strtoi(3), strtou(3): When both ERANGE and ENOTSUP conditions happen, ERANGE should be more important

[alx%kernel.org@localhost]

>Number:         57828
>Category:       lib
>Synopsis:       strtoi(3), strtou(3): When both ERANGE and ENOTSUP conditions happen, ERANGE should be more important
>Confidential:   no
>Severity:       non-critical
>Priority:       medium
>Responsible:    lib-bug-people
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Sun Jan 07 22:30:01 +0000 2024
>Originator:     Alejandro Colomar
>Release:        9.3
>Organization:
Linux man-pages
>Environment:
N/A
>Description:
In strtoi(3) and strtou(3), the conditions for both ERANGE and
ENOTSUP can ahppen at in the same call.  Consider the following:

    strtoi("42z", &end, 0, 3, 7, &status);

>From ERANGE, or ENOTSUP, I argue it should return ERANGE.

-  The information about ENOTSUP can be extracted from 'end', as
   (*end != '\0').  So if we report ERANGE, a caller can still
   check for trailing text easily.  However, there's no secondary
   way to check ERANGE; if it's shadowed by ENOTSUP, that vital
   information is completely lost.
-  It is usual to expect trailing text after the number, as
   when parsing some formatted text, where numbers are mixed with
   other text.  On the other hand, out-of-range values should
   usually be rejected as invalid input, and also warned about.
   If ENOTSUP shadows ERANGE, we're in a bad position.

The following two calls are currently equivalent:

    strtoi("7z", &end, 0, 3, 7, &status);
    strtoi("42z", &end, 0, 3, 7, &status);

I argue they shouldn't; there should be a way to distinguish them.


Here's a discussion in the shadow mailing list, where I originally reported this bug:
<https://lists.sr.ht/~hallyn/shadow/%3CZZoQDms6Sv6e5SPE%40debian%3E>
>How-To-Repeat:
Call strtoi(3) or strtou(3) with a string that overflows or underflows the [min, max] range, *and* contains trailing text.
>Fix:
I've implemented my own strtoi() and strtou(), using GNU C11, which report ERANGE if both ERANGE and ENOTSUP happen:


alx@debian:~/src/shadow/shadow/getlong$ grepc -tfd shadow_strtoi .
./lib/atoi/strtoi.c:intmax_t
shadow_strtoi(const char *str, char **restrict endptr, int base,
    intmax_t min, intmax_t max, int *restrict status)
{
	return strtoN(str, endptr, base, min, max, status, intmax_t);
}
alx@debian:~/src/shadow/shadow/getlong$ grepc -tfd shadow_strtou .
./lib/atoi/strtoi.c:uintmax_t
shadow_strtou(const char *str, char **restrict endptr, int base,
    uintmax_t min, uintmax_t max, int *restrict status)
{
	return strtoN(str, endptr, base, min, max, status, uintmax_t);
}
alx@debian:~/src/shadow/shadow/getlong$ grepc strtoN .
./lib/atoi/strtoi.c:#define strtoN(str, endptr, base, min, max, status, TYPE)                     \
({                                                                            \
	const char  *str_ = str;                                              \
	char        **endptr_ = endptr;                                       \
	int         base_ = base;                                             \
	TYPE        min_ = min;                                               \
	TYPE        max_ = max;                                               \
	int         *status_ = status;                                        \
                                                                              \
	int         errno_saved_, s_;                                         \
	char        *e_;                                                      \
	TYPE        n_;                                                       \
                                                                              \
	if (endptr_ == NULL)                                                  \
		endptr_ = &e_;                                                \
	if (status_ == NULL)                                                  \
		status_ = &s_;                                                \
                                                                              \
	if (base_ < 0 || base_ > 36) {                                        \
		*status_ = EINVAL;                                            \
		n_ = 0;                                                       \
                                                                              \
	} else {                                                              \
		errno_saved_ = errno;                                         \
		errno = 0;                                                    \
		n_ = strtoNmax(TYPE, str_, endptr_, base_);                   \
                                                                              \
		if (*endptr_ == str_)                                         \
			*status_ = ECANCELED;                                 \
		else if (errno == ERANGE)                                     \
			*status_ = ERANGE;                                    \
		else if (n_ < min_ || n_ > max_)                              \
			*status_ = ERANGE;                                    \
		else if (**endptr_ != '\0')                                   \
			*status_ = ENOTSUP;                                   \
		else                                                          \
			*status_ = 0;                                         \
                                                                              \
		errno = errno_saved_;                                         \
	}                                                                     \
	SATURATE(min_, max_, n_);                                             \
})
alx@debian:~/src/shadow/shadow/getlong$ grepc strtoNmax .
./lib/atoi/strtoi.c:#define strtoNmax(TYPE, ...)                                                  \
(                                                                             \
	_Generic((TYPE) 0,                                                    \
		intmax_t:  strtoimax,                                         \
		uintmax_t: strtoumax                                          \
	)(__VA_ARGS__)                                                        \
)
alx@debian:~/src/shadow/shadow/getlong$ grepc SATURATE .
./lib/atoi/strtoi.h:#define SATURATE(min, max, n)  MAX(min, MIN(max, n))