psk.txt contains the right IP for the other endpoint and a 256-bit key
  preceeded by 0x - is this fed in exactly to the phase 1 negotiation, or
  is it treated as a string? I've always been curious about that.

A good question, and inspection of the source should reveal the
answer.  Without looking, I'd guess it is a string.

  [psk security question]

  Is this correct, or am I oversimplifying things? Would using
  pubkeys/certificates resolve any security issues there, or is it really
  only more useful for authenticating your clients?

PSKs are used to authenticate Phase 1; a fresh DH is used to derive
the actual IPsec keys just as it would be if you used public/private
keypairs with or without certificates.   256-bit AES and 2048-bit DH
seem like reasonable choices.

Certificates are not more secure than PSKs in any meaningful way.
They are more convenient in large installations since you can have a
CA and not have to configure n^2 psks.  But, using them introduces
security risks you don't have with PSKs:

* The certificate parsing and validation code might be wrong.  This
  has been an issue in racoon in the past.

* The CA might be compromised.

On the upside, PSK distribution has to be safe.

So really it's a security/convenience tradeoff.

  Is there a way NOT to use compression? 'none' didn't work, and I really
  don't see a need to add more overhead to this link.

Perhaps just omit the configuration line.

  What do the D-H calculations actually govern? The effectiveness of the
  keys or something else? It wasn't clearly documented in racoon.conf(5)
  and Google didn't get me anything useful quickly, but I trust people on
  this mailing list.

  This is really more for educational value than anything

See RFC2401, RFC2409 and the others (but note that 2401bis is in
progress, as is IKEv2).  It's not clear this should be documented in
racoon.conf.


Note that bugs in racoon or other parts of NetBSD would compromise
security.  I'd be much more worried about that then either 2048-bit DH
or 256-bit AES failing.

-- 
        Greg Troxel <gdt@ir.bbn.com>