[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index][Old Index]
Re: Max concurrent connections and TCP port limits
On Aug 24, 2010, at 9:24 31AM, Sad Clouds wrote:
> On Tue, 24 Aug 2010 07:47:44 -0400
> Steven Bellovin <smb%cs.columbia.edu@localhost> wrote:
>> On Aug 24, 2010, at 6:38 11AM, Sad Clouds wrote:
>>> Hi I was pondering about the absolute maximum number of concurrent
>>> TCP connections a given machine can support. In the old days the
>>> limit was dictated by the CPU speed and/or available memory.
>>> Well, these days it seems the limit is TCP's 16-bit port numbers. If
>>> you have a single NAT firewall, or a load balancer in front of many
>>> machines, how can it support more than 65536 concurrent TCP
>>> One possible (the only?) way is to bind multiple IP addresses to the
>>> same physical machine, then theoretically you can have a maximum of:
>>> num_IP_addr * 2^16 concurrent TCP connections
>>> Any idea if NetBSD TCP stack can handle such a case, i.e. can it
>>> reuse the same ephemeral port number with different IP addresses?
>> When it's acting as a server, it can almost certainly handle that
>> with no problem whatsoever. For client port numbers, I believe you
>> have to remember to set SO_REUSEADDR before you can bind to it.
>> A TCP connection is defined by the 4-tuple <local IP,local port,
>> remote IP, remote port>. You can have very many connections to port
>> 80 if they're all coming from the different remote ports or remote
>> hosts. Thus, you can also increase the number of IP addresses on
>> your front end box.
> Well this is the scenario I was thinking:
> Load balancer (LB) in front of a large cluster of servers. All client
> requests go via load balancer, which creates TCP connections to
> servers. The servers do heavy-duty processing and send their responses
> back to the clients, via load balancer.
> LB __ S1
> |__ S2
> |__ S3
> |__ S4
> |__ S1024
> With SO_REUSADDR the load balancer should be able to establish
> unlimited number of concurrent TCP connections with clients (provided
> it has enough memory).
> However the limit on the total number of concurrent TCP connections
> between the load balancer and each server is 65536. If each server is
> powerful enough to handle 500000 concurrent connections, the hardware
> will be underused.
> I can think of just two solutions:
> 1) Have each server bind multiple IP addresses in order to overcome
> TCP's 16-bit port limitation.
Right, and the load balancer should neither know or care that a given IP
address is on the same machine as some other IP address it's routing traffic
to. It will just work.
> 2) Develop application level protocol to multiplex multiple connections
> over a single TCP connection. This way load balancer creates one
> connection to each server and the servers demultiplex client requests.
> This is a bit like reinventing TCP.
Very ugly -- don't go there.
Also 3) virtual machines on the servers, to permit more utilization of the
hardware, and 4) multiple IP addresses on the load balancer.
I'd go with (1) -- it's the simplest, and shouldn't require any new code
--Steve Bellovin, http://www.cs.columbia.edu/~smb
Main Index |
Thread Index |