aligning control message ancillary data

[Taylor R Campbell <campbell%mumble.net@localhost>]

Hello!  Last night while debugging an issue in stunnel, I found that
NetBSD (perhaps specifically NetBSD/macppc) is much more finnicky than
the other operating systems I tried concerning alignment of control
message ancillary data.  Setting up a msghdr structure with the
following idiom, in order to send a file descriptor over a Unix-domain
socket using only the facilities provided in the SUSv3 and documented
in the recvmsg(3) man page, causes bogus file descriptors to be
received on the other end:

   struct msghdr msg;
   struct cmsghdr * cm_ptr;
   struct {
     struct cmsghdr c_msghdr;
     int fd;
   } control;

   msg.msg_control = (void *) &control;
   msg.msg_controllen = sizeof (control);
   /* et cetera */

   cm_ptr = CMSG_FIRSTHDR (&msg);
   cm_ptr->cmsg_len = sizeof (control);
   /* ... */
   (*((int *) CMSG_DATA (cm_ptr))) = fd;

Instead, the correct idiom, I believe, is

   struct msghdr msg;
   char control[CMSG_SPACE (sizeof (int))];

   msg.msg_control = (void *) control;
   msg.msg_controllen = CMSG_LEN (sizeof (int));
   /* et cetera */

   cm_ptr = CMSG_FIRSTHDR (&msg);
   cm_ptr->cmsg_len = CMSG_LEN (sizeof (int));
   /* ... */
   (*((int *) CMSG_DATA (cm_ptr))) = fd;

The macros CMSG_SPACE and CMSG_LEN are described in RFC 2292, but not
in any man pages that I could easily find.  I believe that
substituting `sizeof (control)' for `CMSG_LEN (sizeof (int))' in the
initialization of the length fields will work too, provided that both
uses are changed; if one of them is not (and this was the problem in
stunnel), then sendmsg will yield EINVAL.  I wrote three small
programs[*] exhibiting the issue.  I have observed that

- on Solaris 9/sun4u, sus-test is the only one that compiles, since it
  doesn't have the CMSG_LEN or CMSG_SPACE macros;

- on NetBSD 4.0/macppc, nb-test is the only one that runs, while with
  mixed-test sendmsg yields EINVAL and with sus-test the client
  receives a bogus file descriptor; and

- on OpenBSD 4.3/i386, Mac OS X 10.4.11/i386, and Debian/etch Linux
  2.6.18-6-amd64, all three programs compile and run successfully.

Others have reported that

- on NetBSD-current/macppc, nb-test and mixed-test fail with the
  client receiving no control message, and with sus-test sendmsg
  yields EINVAL;

- on NetBSD 4.0/amd64, nb-test is the only one that runs, while with
  mixed-test sendmsg yields EINVAL and with sus-test sendmsg yields
  EFAULT; and

- on FreeBSD/i386 6.2 and 6.3 all three programs compile and run
  successfully.

So, except for Solaris which doesn't have the RFC 2292 macros, all
three programs ran on all platforms besides NetBSD to which I had
convenient access.

Am I doing something wrong in my test programs, is this a bug in
NetBSD, or should the RFC 2292 macros be documented in the man pages?
Or am I barking up all the wrong trees?  (Also, I wonder why nb-test
is failing on NetBSD-current/macppc.)

[*] nb-test.c, sus-test.c, and mixed-test.c, located in
    <http://mumble.net/~campbell/tmp/>