Re: DoS attack against TCP services

[Sverre Froyen <sverre%viewmark.com@localhost>]

> On 2015-02-04, at 12:32, Christos Zoulas <christos%zoulas.com@localhost> wrote:
> 
> On Feb 4,  7:44pm, 6bone%6bone.informatik.uni-leipzig.de@localhost (6bone%6bone.informatik.uni-leipzig.de@localhost) wrote:
> -- Subject: Re: DoS attack against TCP services
> 
> | Now the server has over 5000 TIME_WAIT connections.
> | 
> | netstat -a -n | grep TIME_WAIT
> | tcp        0      0  139.18.25.33.59256     198.6.1.83.53          TIME_WAIT
> | tcp        0      0  139.18.25.33.59257     77.222.50.250.53       TIME_WAIT
> | tcp        0      0  139.18.25.33.59258     193.232.128.6.53       TIME_WAIT
> | tcp        0      0  139.18.25.33.59259     78.104.145.37.53       TIME_WAIT
> | tcp        0      0  139.18.25.33.59260     192.5.6.30.53          TIME_WAIT
> | tcp        0      0  139.18.25.33.59261     192.41.162.30.53       TIME_WAIT
> | tcp        0      0  139.18.25.33.59262     192.35.51.30.53        TIME_WAIT
> | tcp        0      0  139.18.25.33.59263     192.43.172.30.53       TIME_WAIT
> | tcp        0      0  139.18.25.33.59264     202.12.27.33.53        TIME_WAIT
> | ...
> | 
> | It seems to be a result of the named. lsof shows that the connections are 
> | not owned by named. lsof doesn't show any of the TIME_WAIT connections. So 
> | stopping and restarting named doesn't delete the connections.
> | 
> | Any more things that could be interessing for a problem report?
> 
> I'd start a tcpdump to record all traffic from your local machine going to port
> 53 on the appropriate interface...
> 
> I'd also look at the open descriptors of the named process (although they
> should be closed at this time, since TIME_WAIT means closed on this side,
> and waiting for the 4 minutes to expire before killing the connection)...
> 
> Also I'd record that information every minute or so to see how many
> connections are added and how many are going away.
> 
> Perhaps there is some bug triggered in the tcp stack and somehow connections
> are not being GC’ed?

This is vaguely similar to a problem I have seen from time to time. On my servers, it is usually port 80 that gets attacked. Someone opens TCP connections to this port on the server, sends no request, and leaves the connection open indefinitely. See http://mail-index.netbsd.org/netbsd-users/2011/01/04/msg007484.html

When I test such a scenario to port 53 (using telnet), the connection shows as ESTABLISHED for 30 seconds. Then, presumably, named times-out and closes the connection. At this point netstat shows the connection as TIME_WAIT for another 10 seconds. After that it disappears.

If I disable the network connection during the 30 second period before named times out, however, I instead observe the connection in FIN_WAIT_1 mode for another 30 minutes or so.

This is on netbsd-6. I notice that your netstat output has the client and server columns in the reverse order from what I see. Could it be that in your netstat output, FIN_WAIT_1 is reported as TIME_WAIT?

Regards,
Sverre