Re: Max concurrent connections and TCP port limits

[Sad Clouds <cryintothebluesky%googlemail.com@localhost>]

On Tue, 24 Aug 2010 07:47:44 -0400
Steven Bellovin <smb%cs.columbia.edu@localhost> wrote:

> 
> On Aug 24, 2010, at 6:38 11AM, Sad Clouds wrote:
> 
> > Hi I was pondering about the absolute maximum number of concurrent
> > TCP connections a given machine can support. In the old days the
> > limit was dictated by the CPU speed and/or available memory.
> > 
> > Well, these days it seems the limit is TCP's 16-bit port numbers. If
> > you have a single NAT firewall, or a load balancer in front of many
> > machines, how can it support more than 65536 concurrent TCP
> > connections?
> > 
> > One possible (the only?) way is to bind multiple IP addresses to the
> > same physical machine, then theoretically you can have a maximum of:
> > 
> > num_IP_addr * 2^16 concurrent TCP connections
> > 
> > Any idea if NetBSD TCP stack can handle such a case, i.e. can it
> > reuse the same ephemeral port number with different IP addresses?
> > 
> When it's acting as a server, it can almost certainly handle that
> with no problem whatsoever.  For client port numbers, I believe you
> have to remember to set SO_REUSEADDR before you can bind to it.
> 
> A TCP connection is defined by the 4-tuple <local IP,local port,
> remote IP, remote port>.  You can have very many connections to port
> 80 if they're all coming from the different remote ports or remote
> hosts.  Thus, you can also increase the number of IP addresses on
> your front end box.

Well this is the scenario I was thinking:

Load balancer (LB) in front of a large cluster of servers. All client
requests go via load balancer, which creates TCP connections to
servers. The servers do heavy-duty processing and send their responses
back to the clients, via load balancer.

LB __ S1
  |__ S2
  |__ S3
  |__ S4
  ...
  |__ S1024

With SO_REUSADDR the load balancer should be able to establish
unlimited number of concurrent TCP connections with clients (provided
it has enough memory).

However the limit on the total number of concurrent TCP connections
between the load balancer and each server is 65536. If each server is
powerful enough to handle 500000 concurrent connections, the hardware
will be underused.

I can think of just two solutions:

1) Have each server bind multiple IP addresses in order to overcome
TCP's 16-bit port limitation.

2) Develop application level protocol to multiplex multiple connections
over a single TCP connection. This way load balancer creates one
connection to each server and the servers demultiplex client requests.
This is a bit like reinventing TCP.