lib/47617: Memory and socket leak in librpc

[tbrehm%dspace.de@localhost]

>Number:         47617
>Category:       lib
>Synopsis:       Memory and socket leak in librpc
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    lib-bug-people
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Mon Mar 04 15:10:01 +0000 2013
>Originator:     Thorsten Brehm
>Release:        NetBSD 6.0.1
>Organization:
dSPACE
>Environment:
>Description:
A memory and a socket descriptor leak is triggered in librpc whenever a new RPC 
connection is processed, where the file descriptor exceeds the FD_SETSIZE limit 
(i.e. when the process has too many open file/socket handles).

Cause of the problem is "xprt_register", which detects the error condition 
(socket > FD_SETSIZE), however it does not report the error, nor return status 
information to the caller. The calling functions, unaware that "xprt_register" 
failed, assume the socket is always registered ? so do not free related memory, 
neither do they close the new connection socket.

I'm attaching a patch fixing the issue:
* It adds a return status to "xprt_register" (bool_t instead of void).
* It adds appropriate error checks to all callers of "xprt_register", so they 
can free memory and close the unhandled socket when the issue occurs.

The patch should be clean ? I?ve been using it locally for a while. If you need 
anything else about it, let me know.

>How-To-Repeat:
Choose an arbitrary RPC server interface and keep creating RPC client 
connections until FD_SETSIZE is exceeded (and keep going). Eventually check 
memory consumption and especially socket statistics for the RPC server process, 
which will show loads of pending (leaked) sockets.
>Fix:
(I'm attaching a patch by email)