Question about tcp ephemeral ports

[Olivier MATZ <zer0%droids-corp.org@localhost>]

Hello,

I have a question about how an ephemeral port is chosen when doing a
connect() for a TCP client. It seems that the behaviour between Linux
and NetBSD (and FreeBSD) is different.

I would like to open many TCP clients (several thousands) in my
application. The behaviour of NetBSD seems to prevent applications to do
that. If we have only one source address, the maximum number of TCP
clients depends on the number of ephemeral ports configured in the
system.

Let's have a look to a simple example first, using the attached C source
code:

/*
 * Create two tcp servers. Try to connect many clients to the first
 * server. Once the maximum number of ephemeral ports is reached, it
 * should fail. Then try to connect the other server which is bound to
 * another port. Linux and NetBSD have a different behaviour.
 *
 * Before doing the test, change the number of ephemeral ports to a
 * small value.
 *
 * linux:
 *   cd /proc/sys/net/ipv4/
 *   echo 35000 35005 > ip_local_port_range
 *
 * netbsd:
 *   sysctl -w net.inet.ip.anonportmin=35000
 *   sysctl -w net.inet.ip.anonportmax=35005
 */

The result on linux is:

  (server-7777) success
  (server-8888) success
  (client-7777) success
  (client-7777) success
  (client-7777) success
  (client-7777) success
  (client-7777) success
  (client-7777) success
  (client-7777) bind failed: Cannot assign requested address
  (client-7777) bind failed: Cannot assign requested address
  (client-7777) bind failed: Cannot assign requested address
  (client-7777) bind failed: Cannot assign requested address
  (client-8888) success

The result on NetBSD is:

  (server-7777) success
  (server-8888) success
  (client-7777) success
  (client-7777) success
  (client-7777) success
  (client-7777) success
  (client-7777) success
  (client-7777) success
  (client-7777) bind failed: Resource temporarily unavailable
  (client-7777) bind failed: Resource temporarily unavailable
  (client-7777) bind failed: Resource temporarily unavailable
  (client-7777) bind failed: Resource temporarily unavailable
  (client-8888) bind failed: Resource temporarily unavailable

The last line shows that on Linux, the application is allowed to do the
connect() if the destination port is different. On NetBSD, the connect()
will fail as soon as the maximum number of ephemeral ports is reached.

Actually, when we do a connect(), the network stack tries to bind the
socket first if it is not already done. It seems that this is done at 2
different places:

- in tcp_usrreq.c:tcp_usrreq():

                if (inp->inp_lport == 0) {
                        error = in_pcbbind(inp, (struct mbuf *)0, l);
                        if (error)
                                break;
                }
                error = in_pcbconnect(inp, nam, l);

- in in_pcbconnect()

        if (in_nullhost(inp->inp_laddr)) {
                if (inp->inp_lport == 0) {
                        error = in_pcbbind(inp, NULL, l);
                        /*
                         * This used to ignore the return value
                         * completely, but we need to check for
                         * ephemeral port shortage.
                         * And attempts to request low ports if not root.
                         */
                        if (error != 0)
                                return (error);
                }
                inp->inp_laddr = ifaddr->sin_addr;
        }


I would like to have some expert opinions about this behaviour. Is there
a workaround? Is there a need to patch the kernel? I would answer yes
because I don't see any obvious way to have more than several thousands
of TCP clients with one source address. Also, I wonder if this behaviour
could be used to do a deny of service and prevent other applications to
do their connect().

Please, can you CC me for answers as I'm not a subscriber of the list.

Regards,
Olivier