lib/50316: mergesort(nmemb=0) is undefined

[dyoung%whitecoralislands.com@localhost]

>Number:         50316
>Category:       lib
>Synopsis:       mergesort(nmemb=0) is undefined
>Confidential:   no
>Severity:       non-critical
>Priority:       high
>Responsible:    lib-bug-people
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Thu Oct 08 20:25:00 +0000 2015
>Originator:     David Young
>Release:        NetBSD 5.99.56
>Organization:
David Young
dyoung%pobox.com@localhost    Urbana, IL    (217) 721-9981
>Environment:
Architecture: i386
Machine: i386
>Description:
	mergesort(nmemb=0) calls setup(n=0) calls insertionsort(n=0).
	In insertionsort(), n is a size_t.  insertionsort() subtracts
	1 from n right away,

	        for (ai = a+size; --n >= 1; ai += size)

        and compares with 1.  Since n is unsigned, it wraps to a
        huge number instead of turning to -1, so insertionsort()
        doesn't quit the for-loop immediately, like it should.
        Instead, it calls the comparison function on a couple of
        fictional array members, leading to a segmentation fault.
>How-To-Repeat:
	Here is a simple program that demonstrates the problem. 

	#include <stdlib.h>

	int
	main(int argc, char **argv)
	{
		mergesort(NULL, 0, sizeof(void *) / 2, NULL);
		return 0;
	}
>Fix:
	Workaround: don't mergesort() an array with 0 members.

        Suggested fix: compare nmemb with 0 at the top of mergesort()
        and if equal, return 0 immediately?