It seems to me that rather than jostling the stack around when
entering VM86 mode, and having to provide an extra syscall gate, this
can be done in a simpler fashion.

Namely, just allocate space for 4 values at the top of the stack when
creating a process.

The idea is something like this:

1) The frame that entered the kernel from user mode is always at the
top of the stack.

2) If there is space for the 4 segment registers already, we can
simply set them, and return to user mode in VM86 mode.  From this
point, we'll be using those 4 values on entry and exit.

3) When we switch back to protected mode, the 4 values become
meaningless, and aren't popped any more when we exit to user mode.  So
we still have the extra space, in case we switch into VM86 mode again.

This would remove a lot of the hair in your implementation.


Rather than making changes like:

-       if (p == curproc && ISPL(frame->if_cs) == SEL_UPL) {
+       if (p == curproc &&
+           (ISPL(frame->if_cs) == SEL_UPL || (frame->if_eflags & PSL_VM))) {

you should really be doing:

-       if (p == curproc && ISPL(frame->if_cs) == SEL_UPL) {
+       if (p == curproc && USERMODE(frame)) {

and then define USERMODE() (and define CLKF_USERMODE() in terms of
it).  If VM86 mode is not enabled, then USERMODE() should be optimized
to not check PSL_VM.


Again, this:

        cmpb    $0,_astpending
        je      3f
        testb   $SEL_RPL_MASK,TF_CS(%esp)
+       jnz     4f
+       testl   $PSL_VM,TF_EFLAGS(%esp)
        jz      3f
-       movb    $0,_astpending
+4:     movb    $0,_astpending
        sti
        /* Pushed T_ASTFLT into tf_trapno on entry. */
        call    _trap

should be optimized in the case of VM86 mode not being allowed.


I'm not entirely happy with the vm86_context idea, but I don't have
any better suggestions at the moment.


Could you look at the above items and submit a new diff?

Thanks.