TCP connections clogging up accf_http(9) (was: ESTABLISHED sockets with no fd open? (was: generating ECONNRESET with no syscall?))

[Timo Buhrmester <timo%math.uni-bonn.de@localhost>]

> according to netstat, there are ~190 sockets in the ESTABLISHED state
> with a local port number of 80, while the lighttpd process 
> (the only one listening on port 80) has ~no file descriptors open.
I'm working with the same servers and have analyzed the issue.

It turned out that lighttpd uses accf_http(9); the server process never
even notices incoming connections to accept until ``a complete,
syntactically valid HTTP [...] request has been buffered by the kernel
[or] the data [...] cannot be part of a complete [such] request''. Okay.

Now, unless I'm missing something, the accept filter does not seem to
have any sort of timeout mechanism - a TCP connection that has been
opened but is otherwise seeing no traffic whatsoever will be stuck
there until the remote end closes it.  If the remote end never closes
it, it's stuck for good.  Once what appears to be 193 such connections
are occupying the accept filter, establishing new connections becomes
impossible.

This is easily abused for denial of service purposes(*), as well as a
real-world problem because many web browsers tend to open multiple
connections at once, even if they're using only one of them (Safari has
been observed to open 6 concurrent connections, 5 of them unused, for a
simple PDF download).

Sometimes, those extra connections are not being closed (think a user
waiting for a download to finish, then putting their laptop to sleep
and leaving the (wifi) network.

Therefore, shouldn't there be (or am I just not seeing it?) sort of
a timeout that disposes of connections that have been sitting in the
accept filter for longer than $timespan?  Currently, we have to restart
lighttpd every so many days to cope with the slow but steady "leakage"
of connections...

Bests,
Timo Buhrmester


(*) Attached is an ugly POC exploit consisting of a shell script and
a helper program.  The idea is to repeatedly open connections with
netcat, then convincing netcat to give up the connections silently
without the other end ever noticing it by faking TCP-RST packets.