port-amd64/59309: optimize fpu_kern_enter in kthread_fpu_enter/exit

[campbell+netbsd%mumble.net@localhost]

>Number:         59309
>Category:       port-amd64
>Synopsis:       optimize fpu_kern_enter in kthread_fpu_enter/exit
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    port-amd64-maintainer
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Thu Apr 17 06:45:01 +0000 2025
>Originator:     Taylor R Campbell
>Release:        current
>Organization:
The x86FPU kFoundation
>Environment:
>Description:
On x86, fpu_kern_enter currently always saves FPU state, toggles CR0.TS back and forth to disable and re-enable the FPu, and then restores a safe FPU state from memory.  This is expensive, and should be unnecessary when running in a kernel thread during kthread_fpu_enter/exit.  On aarch64, this improved cgd(4) throughput by ~20%, if memory serves.

My first attempts to take advantage of kthread_fpu_enter/exit on x86 met with failure:

https://mail-index.netbsd.org/source-changes/2020/08/01/msg120050.html
https://mail-index.netbsd.org/source-changes/2023/02/25/msg143550.html

The symptom of the latter failure was described in PR kern/57258: kthread_fpu_enter/exit problem <https://gnats.NetBSD.org/57258>.

It's possible this is related to an undersized union savefpu, which should be addressed by this patch series once it lands: https://mail-index.NetBSD.org/port-amd64/2025/04/11/msg003748.html

It's possible this is related to the order in which LW_SYSTEM_FPU is set/cleared and kthread_fpu_enter/exit_md is called by kthread_fpu_enter/exit, particularly if an interrupt happens in the intervening time.

Not sure but we should figure it out!
>How-To-Repeat:
get frustrated at seeing cgd(4) threads taking too much CPU time under heavy disk I/O
>Fix:
Yes, please!