Re: shutdown(2)'ing a bound UDP socket

To: Roy Marples <roy%marples.name@localhost>
Subject: Re: shutdown(2)'ing a bound UDP socket
From: "Peter J. Philipp" <pjp%centroid.eu@localhost>
Date: Sun, 19 Jul 2020 21:39:29 +0200

On Thu, Jul 16, 2020 at 10:24:16AM +0100, Roy Marples wrote:
> Hi Peter

Hi Roy and tech-net,

Thank you for the patient wait.  I have finally written something and was
able to reproduce the condition tonight.  I'm going to paste the program
inline here and then talk a little below it.  Just search for "---" to
skip the code to see the commentary...

------->
#include <sys/types.h>
#include <sys/socket.h>
#include <sys/select.h>

#include <netinet/in.h>
#include <arpa/inet.h>
#include <netdb.h>

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <time.h>

#include <err.h>
#include <errno.h>

int bind_socketv6(int so, u_short port);
int bind_socketv4(int so, u_short port);

int
main(int argc, char *argv[])
{
	struct sockaddr_in sin;
	struct sockaddr_in6 sin6;
	struct timeval tv;

	int so, so6, dup, dup6, on = 1;
	int max, len, sel;

	u_short port = 65053;
	fd_set rdset;
	pid_t pid;
	socklen_t slen;

	char buf[512];


	if (argc > 1)
		port = atoi(argv[1]);


	/* make the dup's */

	dup = socket(AF_INET, SOCK_DGRAM, 0);
	dup6 = socket(AF_INET6, SOCK_DGRAM, 0);

	if (dup < 0 || dup6 < 0) {
		err(1, "socket");
	}
	
	on = 1;
	if (setsockopt(dup, SOL_SOCKET, SO_REUSEPORT, &on, sizeof(on)) < 0) {
		err(1, "setsockopt");
	}
		
	on = 1;
	if (setsockopt(dup6, SOL_SOCKET, SO_REUSEPORT, &on, sizeof(on)) < 0) {
		err(1, "setsockopt6");
	}
	
	/* shutdown on the dup's */

	if (shutdown(dup, SHUT_RD) < 0) {
		err(1, "shutdown dup");
	}

	if (shutdown(dup6, SHUT_RD) < 0) {
		err(1, "shutdown dup6");
	}

	/* bind the dup's too */
	
	if (bind_socketv4(dup, port) < 0) {
		err(1, "dup bind");
	}

	if (bind_socketv6(dup6, port) < 0) {
		err(1, "dup bind6");
	}

	/* the main sockets */

	so = socket(AF_INET, SOCK_DGRAM, 0);
	so6 = socket(AF_INET6, SOCK_DGRAM, 0);

	if (so < 0 || so6 < 0) {
		err(1, "socket");
	}
	
	on = 1;
	if (setsockopt(so, SOL_SOCKET, SO_REUSEPORT, &on, sizeof(on)) < 0) {
		err(1, "setsockopt");
	}
		
	on = 1;
	if (setsockopt(so6, SOL_SOCKET, SO_REUSEPORT, &on, sizeof(on)) < 0) {
		err(1, "setsockopt6");
	}
	
	if (bind_socketv4(so, port) < 0) {
		err(1, "bind");
	}

	if (bind_socketv6(so6, port) < 0) {
		err(1, "bind6");
	}

	/* fork a child */

	switch (pid = fork()) {
	case -1:
		err(1, "fork");
		break;
	case 0:
		close(so); close(so6);	
		for (;;) {
			/* here we can write to the dup's if we wish */
			sleep(10);
			memset(&buf, 'X', 16);
			memset(&sin, 0, sizeof(sin));
			sin.sin_family = AF_INET;
			sin.sin_port = htons(8888);
			sin.sin_addr.s_addr = inet_addr("192.168.177.2");
			sendto(dup, buf, 16, 0, (struct sockaddr*)&sin, 
				sizeof(struct sockaddr_in));
		}
		/* NOTREACHED */
		break;
	default:
		close(dup); close(dup6);
		max = so6;
		break;
	}
	
	for (;;) {
		FD_ZERO(&rdset);
		FD_SET(so, &rdset);
		FD_SET(so6, &rdset);
	
		tv.tv_sec = 5;
		tv.tv_usec = 0;

		if ((sel = select(max + 1, &rdset, NULL, NULL, &tv)) < 0) {
			fprintf(stderr, "select error: %s\n", strerror(errno));
			continue;
		}

		if (sel == 0) {
			continue;
		}
		
		if (FD_ISSET(so, &rdset)) {
			slen = sizeof(struct sockaddr_in);
			if ((len = recvfrom(so, buf, sizeof(buf), 0, 
					(struct sockaddr*)&sin, &slen)) < 0) {
				warn("recvfrom");
			}
			printf("%lu so read %d bytes\n", time(NULL), len);

			/* send something back */
			if (sendto(so, buf, len, 0, (struct sockaddr*)&sin, slen) < 0)
				warn("sendto");
			continue;
		} else if (FD_ISSET(so6, &rdset)) {
			slen = sizeof(struct sockaddr_in6);
			if ((len = recvfrom(so6, buf, sizeof(buf), 0, 
					(struct sockaddr*)&sin6, &slen)) < 0) {
				warn("recvfrom");
			}
			printf("%lu so6 read %d bytes\n", time(NULL), len);
			continue;
		}

	} /* for(); */
/* NOTREACHED */
}
	
int
bind_socketv4(int so, u_short port)
{
	struct sockaddr_in sin;

	memset(&sin, 0, sizeof(sin));
	sin.sin_family = AF_INET;
	sin.sin_port = htons(port);
	
	return (bind(so, (struct sockaddr *)&sin, sizeof(sin)));	
}

int
bind_socketv6(int so, u_short port)
{
	struct sockaddr_in6 sin6;

	memset(&sin6, 0, sizeof(sin6));
	sin6.sin6_family = AF_INET6;
	sin6.sin6_port = htons(port);
	sin6.sin6_len = sizeof(struct sockaddr_in6);
	
	return (bind(so, (struct sockaddr *)&sin6, sizeof(sin6)));	
}
<-------

So as you can see the program binds to 65053, but I used ./test-program 8888
to have it bind on port 8888, but it doesn't really matter which port.

The dup descriptors go into the child and it sends every 10 seconds a packet
to 192.168.177.2 (my OpenBSD workstation, but it can be any address).

What I have found is that with netcat from any source I can get the select
loop in the parent to display UNIX timestamp and the "so" descriptor 
read X bytes, UNTIL the child in the for(;;)/sleep(10) loop transmits.  
Then it won't receive any more on the parent.  And this is exactly what 
I'm running into on my program delphinusdnsd.

I hope you can work with this, let me know if I must do anything to accomodate
some sort of fix.  The behaviour I'm looking for is that one can write on
the shutdown SHUT_RD socket (dup) and read without it blocking on (so) parent
socket.

Best Regards,
-peter



> On 15/07/2020 13:00, Peter J. Philipp wrote:
> > Hi,
> > 
> > I'm the author of delphinusdnsd, a lightweight dns server.  I develop on
> > OpenBSD but produce ports to Linux, FreeBSD and NetBSD.  My latest code I
> > have ported to Linux and FreeBSD successfully, but NetBSD is not working.
> > 
> > What I do in my code is I bind (with SO_REUSEPORT option) two UDP descriptors
> > on the same port and shutdown(2) one of those (called dup) in the receive
> > setting (SHUT_RD).  This allows me to read off the non-shut descriptor but
> > send packets on either, it works out well on OpenBSD.  However while NetBSD
> > does allow shutting down the descriptor (unlike FreeBSD which has other
> > code to fix that problem it looks like), it does want to deliver incoming
> > queries to the shut descriptor.  I get one answer from my server on NetBSD
> > and then it blocks.  I tried patching this in kernel but it seems to be over
> > my head, I'm doing something wrong.  Basically the socket should get a
> > SS_CANTRCVMORE state, but checking for this seems to be hard, plus I don't
> > know what I'M doing in the NetBSD kernel.
> > 
> > So I'm basically left of begging someone to fix this functionality to skip
> > shutdown(2)'ed bound reading sockets and let the ones that do read receive
> > the packet.
> > 
> > Otherwise this may be my last year of supporting NetBSD unfortunately.  I
> > would like to give a donation but I'm dirt poor, as gesture I can maybe
> > afford five euros or something, but can't find more.  I have donated five USD
> > before in 2018, if it's worth any.  I'm releasing version 1.5.0 between
> > september 2020 and november 2020, and I hope to continue NetBSD support, if
> > only in -current.
> > 
> > If you need to see my code to see what I'm doing you can get it at
> > https://delphinusdns.org/download/snapshot/delphinusdnsd-snapshot.tgz and the
> > relevant lines of code are in delphinusdnsd.c (main()) and go further into
> > forward.c.  If you need a config file for the forwarding mode I can produce
> > you one on request.
> > 
> > Please CC me directly as I'm not on the tech-net%netbsd.org@localhost list.
> 
> Do you have a small test case for this that can reproduce the issue?
> 
> Roy

Follow-Ups:
- Re: shutdown(2)'ing a bound UDP socket
  - From: Erik Fair

References:
- shutdown(2)'ing a bound UDP socket
  - From: Peter J. Philipp
- Re: shutdown(2)'ing a bound UDP socket
  - From: Roy Marples

Prev by Date: Re: Please RUMPify tests in sbin/ifconfig
Next by Date: Re: shutdown(2)'ing a bound UDP socket
Previous by Thread: Re: shutdown(2)'ing a bound UDP socket
Next by Thread: Re: shutdown(2)'ing a bound UDP socket
Indexes:

Home | Main Index | Thread Index | Old Index