I have found the reason to this panic (and fixed it :-):

Wiring of a page were done at the wrong place in pmap_enter, so pages in
kernel space never got the wired bit set. This caused panics if a swapped
out process was swapped in again and the kernel stack had not yet been
unmapped.

-- Ragge