So I've been using IPsec on my wireless network for quite a while.  As
various bugs in racoon(8) have been fixed (some by me, some by
others), I've transitioned from static SAD entries to simpler IKE
modes, and now to certificates.  In the process, I've found that
racoon(8) configuration is extremely baroque and difficult to debug
(often requiring ktrace(1)).

But that's not what this mail is about.  I thought I would share some
experiences using IPsec on Mac OSX 10.2.1, which is based on
KAME/NetBSD IPsec.

Since it's based on the KAME stack and racoon(8), configuration is
much like NetBSD, and, for the most part, working NetBSD
configurations work on OSX.  (I'll give an example later.)  However,
there are some pitfalls:

1) There is an extremely bad interaction between 802.11 IBSS mode
   (`computer-to-computer networks'), power-saving, and IPsec.

   In IBSS mode, the Mac attempts to do power-saving on the wireless
   interface by flapping the interface up and down.  Unfortunately,
   every time it brings the interface up, it restarts DHCP and IPsec
   negotiation.  This causes a significant delay.

   To make matters worse, the other endpoint of the IPsec tunnel in
   this case is an AlphaStation 200, which is a bit slow.  Because the
   negotiation takes a while, the Mac will sometimes shut the
   interface off again -- presumably due to inactivity -- before it
   has completed!  This can go on a few times before it finally works.

   To mitigate this problem, when interacting with OSX, I recommend
   using `dh_group modp1024' and `exchange_mode aggressive,...' to
   speed up negotiation a little.

2) There is a serious problem -- I'm not sure whether it's with the
   stack or racoon(8) -- that sometimes tunnel SAD entries will get
   entered as transport entries if there is a similar SPD entry -- for
   example, with:

spdadd 10.3.0.2 10.3.0.0/24 any -P out ipsec esp/transport//require;
spdadd 10.3.0.0/24 10.3.0.2 any -P in ipsec esp/transport//require;
spdadd 10.3.0.2 0.0.0.0/0 any -P out ipsec esp/tunnel/10.3.0.2-10.3.0.1/require;
spdadd 0.0.0.0/0 10.3.0.2 any -P in ipsec esp/tunnel/10.3.0.1-10.3.0.2/require;

   The SAD entries for the latter two policies always appear as
   transport mode entries on the iBook.  This causes traffic to not
   pass correctly between the systems.  Worse, multiple SAD entries
   stack up on both sides when this happens.

   I did not find any solution other than to just delete the transport
   SPDs, thus causing the iBook to not be able to talk to other hosts
   on the wireless network.  (An alternative would be to special-case
   a policy on all machines on the subnet so that traffic to the iBook
   always goes through the router, thus allowing the iBook to use the
   single tunnel-mode policy -- but this is exactly what I was trying
   to avoid.)


As far as actual configuration goes, I put the following in
netstart.local on the router machine:

setkey -c <<__EOF__
spdadd 10.3.0.1 10.3.0.0/24 any -P out ipsec esp/transport//require;
spdadd 10.3.0.0/24 10.3.0.1 any -P in ipsec esp/transport//require;
__EOF__
for n in `jot 62 2`; do 
setkey -c <<__EOF__
spdadd 0.0.0.0/0 10.3.0.$n any -P out ipsec esp/tunnel/10.3.0.1-10.3.0.$n/require;
spdadd 10.3.0.$n 0.0.0.0/0 any -P in ipsec esp/tunnel/10.3.0.$n-10.3.0.1/require;
__EOF__

and the following in netstart.local on each of the other machines
(changing 10.3.0.2 as appropriate, and noting the exception for the
iBook):

setkey -c <<__EOF__
spdadd 10.3.0.2 127.0.0.1 any -P out none;
spdadd 127.0.0.1 10.3.0.2 any -P in none;
spdadd 10.3.0.2 10.3.0.2 any -P out none;
spdadd 10.3.0.2 10.3.0.2 any -P in none;
spdadd 10.3.0.2 10.3.0.0/24 any -P out ipsec esp/transport//require;
spdadd 10.3.0.0/24 10.3.0.2 any -P in ipsec esp/transport//require;
spdadd 10.3.0.2 0.0.0.0/0 any -P out ipsec esp/tunnel/10.3.0.2-10.3.0.1/require;
spdadd 0.0.0.0/0 10.3.0.2 any -P in ipsec esp/tunnel/10.3.0.1-10.3.0.2/require;
__EOF__

racoon.conf is the same on all machines (modulo the certificate and
key file names):

path backupsa "/etc/racoon/racoon.sa";
path certificate "/etc/racoon/certs";
remote anonymous
{
	exchange_mode aggressive,main;
	certificate_type x509 "nagasaki.cert" "nagasaki.key";
	my_identifier asn1dn;
	keepalive;
	lifetime time 60 min;
	proposal {
		encryption_algorithm rijndael 256;
		hash_algorithm sha1;
		dh_group modp1024;
		authentication_method rsasig;
	}
}
sainfo anonymous
{
	pfs_group modp1024;
	lifetime time 60 min;
	encryption_algorithm rijndael 256;
	authentication_algorithm hmac_sha1;
	compression_algorithm deflate;
}

And finally, you need to actually create the certificates.  A simple
cookbook for this is as follows:

mkdir /etc/racoon/certs
cd /etc/racoon/certs
cp /usr/share/examples/openssl/openssl.cnf .
vi openssl.cnf
[In the CA_default section, change `dir' to `.'.]
echo -n >index.txt
echo 01 >serial
d=ihack
openssl req -config openssl.cnf -new -x509 -keyout $d.key -out $d.cert -days 1461
ln -s $d.cert `openssl x509 -noout -hash -in $d.cert`.0
for m in eniwetok mururoa lop-nor ...; do
  openssl req -config openssl.cnf -nodes -new -newkey rsa:1024 -keyout $m.key -out $m.req -days 1461
  openssl ca -config openssl.cnf -outdir . -out $m.cert -in $m.req -cert $d.cert -keyfile $d.key -days 1461 -policy policy_anything
done
rm *.req
chmod go= *.key

Now, on each machine *other* than the one you generated all the keys
on, copy the CA certificate (the one with the xxxxxxxx.0 symlink you
made), and the certificate and private key for that machine, into
/etc/racoon/certs.

Note that the xxxxxxxx.0 name is how OpenSSL looks up CA keys.  If
it's missing, the ISAKMP-SA negotiation will simply fail with both
sides retransmitting and no log messages to tell you what the Hell is
going on -- unless you frob syslog.conf and start racoon(8) with lots
of `-d's.


On the iBook, I put the setkey(8) commands and racoon(8) startup in a
script in my home directory, and I run that from a shell.  I don't
know how to get things to start up automatically, and I don't really
care enough to figure it out.

If you need any further assistance to make it work, feel free to ask,
but be prepared for a lot of grumbling.