I don't know if anyone else has played with VM86 mode, but I have..

Some months ago I played around with turning on VM86 mode (using the
"sigreturn" syscall) and discovered that it was a great way to crash
NetBSD -- there are a number of assumptions in the machine-dependant
code that the user-space [cdefgs]s registers contain segment
selectors, not 8086-style segment offsets; also, the
return-to-user-space part of either the context switch or syscall
return code (I don't remember which; it may be both) also made that
assumption.. the result being that the kernel attempted to load a
segment offset which was an invalid segment selector, and the kernel
died with a kernel-mode segmentation fault.

I filed a bug report, and Charles fixed sigreturn to prevent you from
setting the VM86 bit. I haven't followed what he has been doing to the
code since then.

The "fix" may well be to use a different task gate for VM86 processes,
and include special-case code in the return-to-user-mode code which
uses that instead of popping all the segment registers if the VM86 bit
is set.

						- Bill