Some issues with x86's TLS [amd64: USER_LDT (and wine)]

[Maxime Villard <max%m00nbsd.net@localhost>]

There are several design issues that prevent us from implementing USER_LDT on
amd64 easily. I don't quite know how to fix them yet; I'm just posting here some
thoughts in case someone is interested or finds an easy way of fixing all of
that.

Let's consider a 32bit lwp, from netbsd32 for example, running on amd64.

Currently, when the segment registers are set by the kernel for a user process
(in setucontext, or sigcontext, or ptrace, or wherever), the kernel makes sure
these segments point to fixed entries in the gdt or ldt. These entries are
constant and are always valid, and therefore, the "mov" instructions on our
segment registers never fault in the kernel.

Our Thread Local Storage implementation uses %fs/%gs (which I will abbreviate
fsgs from now on). When entering the kernel (syscall or trap), we save fsgs in
the lwp's trap frame; but when returning to userland we don't restore them.
Instead, we rely on the fact that whenever the kernel changes the lwp's fsgs, it
calls cpu_reload_fsgs, which initializes the registers right away for userland.
The other place where the kernel initializes fsgs is in cpu_switchto: fsgs are
reloaded from the frame of the lwp we are context-switching to.

Now, with USER_LDT, we are allowing segment registers to have different
locations, which means that we need to remove the checks in the kernel about the
fixed gdt/ldt and correctly redirect the faults on "mov" to userland. That's not
a big deal.

But inherently, my understanding is that USER_LDT is still never going to work
with our TLS. The user-set ldt is stored in the lwp's pmap, but when context-
switching the pmap is reloaded *after* cpu_switchto is called. cpu_switchto
initializes fsgs, which may point to a user ldt, which is not yet loaded in
ldtr, which causes "mov" to fault. And it therefore always faults.

On i386 it works, because we save/restore fsgs when entering/exiting the kernel;
so i386's cpu_switchto does not need to touch fsgs.

We would need to have pmap_load called before cpu_switchto, which is something
we are never going to do given that this is mi code.

I see only two ways of fixing all of that:
 - change amd64 so that it saves/restores fsgs like i386. We would need to take
   care of the MSRs depending on whether the lwp is 32bit or 64bit (which is
   mostly what FreeBSD does). Then we could remove cpu_reload_fsgs, and the
   fsgs-related code in cpu_switchto.
 - change our ldt implementation, so that the user-set ldt is stored in
   proc::p_md instead of pmap. We would then have to update cpu_switchto in
   both i386 and amd64 so that it reloads the ldt before potentially touching
   fsgs. It may not be totally correct though; the CVS log implicitly suggests
   that some programs consider the ldt as attached to the vm space.

Maxime