kern/58195: vmt(4) driver conflicts with open-vm-tools

[pho%cielonegro.org@localhost]

>Number:         58195
>Category:       kern
>Synopsis:       vmt(4) driver conflicts with open-vm-tools
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    kern-bug-people
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Thu Apr 25 13:15:00 +0000 2024
>Originator:     PHO
>Release:        10.0
>Organization:
TNF
>Environment:
NetBSD yukari.cielonegro.org 10.0 NetBSD 10.0 (GENERIC64) #0: Thu Mar 28 08:33:33 UTC 2024  mkrepro%mkrepro.NetBSD.org@localhost:/usr/src/sys/arch/evbarm/compile/GENERIC64 evbarm
>Description:
http://mail-index.netbsd.org/current-users/2024/04/20/msg045201.html

The vmt(4) driver causes a conflict with vmtoolsd(8) from sysutils/open-vm-tools. The driver works by establishing an RPC channel called TCLO between VMware guest and host to receive controlling messages from the host. The problem is that vmt(4) is essentially a subset of vmtoolsd(8), and they both use the same RPC channel, but TCLO is never meant to be simultaneously used by two distinct services in the same VM guest. So when vmtoolsd(8) is running while also vmt(4) is active, they continually fight for the channel, both get rejected by the confused VM host, and neither one can establish a stable communication line.
>How-To-Repeat:
Run NetBSD 10.0 on VMware and make sure the vmt(4) driver is loaded. Then install sysutils/open-vm-tools, add "vmtools=YES" to /etc/rc.conf, and run "/etc/rc.d/vmtools start".

The two services will start fighting and spam the console without achieving anything. Try shutting down the guest from VMware; since no TCLO channel is successfully established, the host forcibly terminates the VM without first asking the guest politely to shut down itself.
>Fix:
There is no obvious way to detect the conflict and automatically turn off the vmt(4) driver, because the RPC can be done from userland via hypervisor calls. It does not require any kind of privileges, so the driver cannot know that there is another service in the userland that is trying to do essentially the same thing as the driver.

So I initially thought of modularizing vmt(4) and unloading the module right before starting vmtoolsd, but I scrapped the idea because that wouldn't work when kern.securelevel is above zero. The next thing I came up with was to simply add a sysctl knob to turn it off. I'll soon post a patch to do just that.