[Daniel Carosone <dan@geek.com.au>, replying to me]

>> I find myself needing to encapsulate IP in TCP, for reasons not
>> really worth going into here (the principal part is a NAT box whose
>> configuration is out of my control).
> Does it have to be TCP?  Could UDP work as well?

I'm not certain, but I'm inclined to doubt it.  One problem with that
is telling what addr/port on the NATted end to send packets to,
especially after idle.

> If you can do UDP, there's a whole range of IP tunnelling options..

I'm already using a UDP tunnel for a similar case where there is no NAT
breaking things in between.

> Failing all that, my suggestion would be to dissolve the problem by
> recreating the underlying network behaviour.  Set up tun(4) or
> similar devices at each end, and write a tunnelling daemon that opens
> *lots* of TCP streams between b-c, spreads all the tunnelled packets
> evenly across them, and doesn't try to do any more recovery than
> opening a new stream when one closes unexpectedly.

That's an interesting thought.  I'll have to try it.

[Seth Kurtzberg <seth@cql.com>]

> If he really means tunneling IP through TCP, that would be
> essentially the same as the UDP case.  If he means an IP layer that
> can be used just like any other IP layer (meaning you could have
> nested TCP) then I agree that you have a mess.

Well, when I said IP tunneled through TCP, I really meant building an
IP transport over a TCP connection in order to enable use of any of the
many protocols layered above IP, so yes, nested TCP is entirely
possible.

> I think if you get into a situation with packet
> disassembly/reassembly, you would have all the problems you described
> plus a mess at that level.  So choosing a packet size that guarantees
> that this never happens is a good idea.  I don't know what DSL does
> with packet sizes,

The DSL in question supports regular Ethernet frames (there is no PPPoE
silliness causing trouble).  I don't know whether it can do jumbograms;
I don't think the machine on one end of the DSL line can, which renders
the question of whether the line itself can moot.

> Of course, since the routing isn't necessarily the same for all
> packets, you can't even find that number definitively, but in
> practice I've found that multiple routes for packets that are part of
> a TCP stream are rare.

Actually, the routing _is_ the same for all packets.  I know everything
that appears on that route, and the only time it will change is when
doing failover in response to dead hardware, which happens seldom
enough that I'm willing to totally ignore it here.

> Probably a good thing to do is to establish a TCP connection, run it
> for several hours, and see if the error rate is zero, or close to
> zero.  If so, then the nested recovery would still be nasty, but it
> might not happen often enough to get into any endless loops or
> livelock situations.

Well, I've been using a naïve IP-in-TCP tunnel for some days now (a tun
device on either end, packets get a simple length-and-data protocol
wrapped around them).  It "works", but interactive feel is not what I
feel it should be - not nearly as good as when working between the same
two machines in non-tunnel ways.  Twice, the tunnel connection has been
broken, apparently by the NAT box dropping the connection state (which
is not a problem when the NATted end wants to send something, because
it gets an RST and reestablishes the connection, but it is when the
other end is sending, since it can't bring the underlying TCP
connection back up).  After the second time, I had the NATted machine
ping the other machine through the tunnel once a minute, which might
prevent the state loss and even if it doesn't would bring the tunnel
back up; since then, I've noticed no failures.

> Of course, I know nothing about the physical topology in question
> here.  der Mouse, can you provide those details?

My house LAN is has a routed portion and a nonrouted portion, the
latter being portions of 10.0/16.  At work, there is a machine on my
desk which has an address in a subnet of 10.10/16, behind the pesky NAT
box.  Since they're not using 10.0/16 for anything at work, I wanted to
simply make my machine appear as another machine in nonrouted space on
my house LAN.  The security issues involved in pulling a machine behind
their NAT onto my non-routed house subnet (which is *not* NATted) are
acceptable, but getting a permanent NAT mapping on the NAT box is a
non-starter.

Topology: at home, my house router has three interfaces: house routed,
house nonrouted, and DSL line.  On the other end of the DSL line is a
router box at my upstream; it shares an Ethernet with the NAT box.  My
desk machine shares an Ethernet with (the other side of) the NAT box.
Thus, between the tunnel endpoints there are only two machines, the NAT
and another router, and three hops, the DSL and two Ethernets.

> If the machine is on the same subnet, or otherwise connected so that
> packet loss is unlikely, you have one situation.

I flood-pinged (ping -f) from my house gateway to another machine on
the Ethernet in the middle (ie, outside the NAT, but on the far side of
the DSL):

7496 packets transmitted, 7446 packets received, 0.7% packet loss
round-trip min/avg/max/stddev = 17.755/83.081/488.118/60.777 ms
  96.4 packets/sec sent,  95.6 packets/sec received

A flood-ping from the box on my desk to the same box outside the NAT is
rock-solid:

12260 packets transmitted, 12260 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 1.623/2.469/223.860/5.210 ms
  415.2 packets/sec sent,  415.3 packets/sec received

I was even crazy enough to try flood-pinging through the existing
simplistic IP-in-TCP tunnel (house machine pinging work machine).  The
house gateway Source Quenched twice in about 12 seconds of
flood-pinging, and the results were:

1162 packets transmitted, 1156 packets received, 0.5% packet loss
round-trip min/avg/max/stddev = 89.381/2249.982/4286.183/1606.231 ms
  100.0 packets/sec sent,  77.6 packets/sec received

> Of course you care about both the maximum burst rate and the maximum
> steady state rate.

The long-term (days) average rate is very low - tens or at most
hundreds of bytes per second.  The maximum burst rate is "as much as it
will sustain".

I should look into UDP more; the NAT box is known to keep _some_ state
for _some_ UDP traffic, so I might be able to make it work.

/~\ The ASCII				der Mouse
\ / Ribbon Campaign
 X  Against HTML	       mouse@rodents.montreal.qc.ca
/ \ Email!	     7D C8 61 52 5D E7 2D 39  4E F1 31 3E E8 B3 27 4B