Right now, there are two paths into having a particular network
  connection (say, a TCP session) encrypted/authenticated with IPSEC:
  either via the setkey packet-filter SPD, or via per-socket policy set
  by the application.

(speling nit: IPSEC is the kernel option, IPsec is the protocol)

Is racoon able to negotiate per-socket policy now?  I haven't looked
in a while.  But per-socket vs. SPD is an orthogonal issue to your
main concern, I think.

  I'm looking for the ability to have network connections authenticated
  with IPSEC on a per-user basis (using certificates, kerberos and/or
  XAUTH hybrid mode) via encapsulation, both for applications and
  protocols without native support for such mechanisms in the endpoints,
  and for authenticated traversal of intermediate gateways.

I'm not sure I follow 'by encapsulation', but this makes sense.

Several years ago, I did a rough outline for per-user credentials.
Note that RFC2401 allows them, but NetBSD's implementation does not.
Basically, it involved

  users being able to push keys or hook up an agent to racoon
  racoon being able to negotiate multiple Phase 1s with a given peer
  racoon being able to present an identity to a program and have it
    map it a locally meaningful name
  storing more complex identities, including user names in SPD
  storing more complex identities, including user names in SA
  checking identities when doing SPD/SA matches
  conveying identities to programs via some setsockopt a la IP_RECVIF

If you want to send money we finish the project as proposed :-)

But seriously, what you want is the direct analog of how Kerberos
handles user identities, except that it's far more complicated due to
racoon, SPD, SA insteadof all being in process, and I think it is
entirely doable and sensible, and within the IPsec architecture.



-- 
        Greg Troxel <gdt@ir.bbn.com>