Yeah, I was originally thinking in terms of SSL, for which one does
  (AFAIK) need curl or something of the sort, then designed it out.
  Woops.

I think the solution should provide perfect forward secrecy, so that
passively tapping the net ahead of time together with the assumed
physical possession doesn't get the attacker the key.  That was why I
suggested IPsec, although I should have explained why