lib/46773: librpc segfaults when out of memory

[brehm%gmx.de@localhost]

>Number:         46773
>Category:       lib
>Synopsis:       librpc segfaults when out of memory
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    lib-bug-people
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Mon Aug 06 07:05:00 +0000 2012
>Originator:     Thorsten Brehm
>Release:        5.1.2
>Organization:
>Environment:
>Description:
When system is out of or low on memory, "xdrrec_create" in 
"src/lib/libc/rpc/xdrrec_create.c" bails out, prints the warning 
"xdrrec_create: out of memory", and leaves xdrs->x_private == NULL.

However, the "makefd_xprt" does not check if "xdrrec_create" was successful and 
will segfault in this case (by dereferencing "xdrs->x_private" unconditionally).

Attached patch adds appropriate checks to "xdrrec_create" callers, so they also 
bail out when "xdrrec_create" did.
>How-To-Repeat:
Allocate all available memory and trigger a call to any RPC server.
>Fix:
Index: clnt_vc.c
===================================================================
RCS file: /cvsroot/src/lib/libc/rpc/clnt_vc.c,v
retrieving revision 1.18
diff -w -u -r1.18 clnt_vc.c
--- clnt_vc.c   13 Mar 2012 21:13:44 -0000      1.18
+++ clnt_vc.c   4 Aug 2012 09:50:46 -0000
@@ -301,6 +301,8 @@
        recvsz = __rpc_get_t_size(si.si_af, si.si_proto, (int)recvsz);
        xdrrec_create(&(ct->ct_xdrs), sendsz, recvsz,
            h->cl_private, read_vc, write_vc);
+       if (ct->ct_xdrs.x_private == NULL)
+               goto fooy;
        return (h);
 
 fooy:
Index: svc_vc.c
===================================================================
RCS file: /cvsroot/src/lib/libc/rpc/svc_vc.c,v
retrieving revision 1.26
diff -w -u -r1.26 svc_vc.c
--- svc_vc.c    20 Mar 2012 17:14:50 -0000      1.26
+++ svc_vc.c    4 Aug 2012 09:50:47 -0000
@@ -260,7 +260,7 @@
 makefd_xprt(int fd, u_int sendsize, u_int recvsize)
 {
        SVCXPRT *xprt;
-       struct cf_conn *cd;
+       struct cf_conn *cd = NULL;
        const char *netid;
        struct __rpc_sockinfo si;
  
@@ -276,9 +276,11 @@
        cd->strm_stat = XPRT_IDLE;
        xdrrec_create(&(cd->xdrs), sendsize, recvsize,
            (caddr_t)(void *)xprt, read_vc, write_vc);
+       if (cd->xdrs.x_private == NULL)
+               goto out;
        xprt->xp_p1 = (caddr_t)(void *)cd;
        xprt->xp_verf.oa_base = cd->verf_body;
-       svc_vc_ops(xprt);  /* truely deals with calls */
+       svc_vc_ops(xprt);  /* truly deals with calls */
        xprt->xp_port = 0;  /* this is a connection, not a rendezvouser */
        xprt->xp_fd = fd;
        if (__rpc_fd2sockinfo(fd, &si) && __rpc_sockinfo2netid(&si, &netid))
@@ -289,6 +291,8 @@
        return xprt;
 out:
        warn("svc_tcp: makefd_xprt");
+       if (cd)
+               mem_free(cd, sizeof(struct cf_conn));
        if (xprt)
                mem_free(xprt, sizeof(SVCXPRT));
        return NULL;