port-mac68k/60717: mac68k: genfb_grfbus registers no mode callback, so nothing on the port can react to WSDISPLAYIO_SMODE

["raytran08%gmail.com@localhost via gnats" <gnats-admin%NetBSD.org@localhost>]

>Number:         60717
>Category:       port-mac68k
>Synopsis:       mac68k: genfb_grfbus registers no mode callback, so nothing on the port can react to WSDISPLAYIO_SMODE
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    port-mac68k-maintainer
>State:          open
>Class:          change-request
>Submitter-Id:   net
>Arrival-Date:   Fri Sep 11 23:15:01 +0000 2026
>Originator:     Ray Tran
>Release:        11.0
>Organization:
Diamond Creek Digital
>Environment:
Hardware: Apple Macintosh Centris 650
Kernel: NetBSD 11.0 and CURRENT
>Description:
genfb(4) has a hook for the one transition a framebuffer driver has to
know about.  That is WSDISPLAYIO_SMODE, the ioctl by which X, wsfb
clients and image viewers take the display into mapped mode and hand
it back.  When a genfb_mode_callback is registered, genfb calls
gmc_setmode() with the new mode at attach and on every change
(dev/wsfb/genfb.c, genfb_init() and the WSDISPLAYIO_SMODE case in
genfb_ioctl()).

That is the point at which chipset-specific state has to be handed over
and reclaimed: a hardware cursor hidden or restored, a palette saved and
put back, a CRTC or scanout register parked where a linear mmap expects
it and released afterwards, a blanking or power state changed.  x86
(arch/x86/pci/pci_machdep.c), Hyper-V (arch/x86/x86/hyperv.c) and
drmfb (external/bsd/drm2/drm/drmfb.c) all register one for these
reasons.

mac68k does not.  genfb_grfbus_attach() registers the colormap callback
("cmap_callback") and stops there, so on this port sc_modecb is always
NULL and the transition is invisible to everything below wsdisplay.
Nor can a driver make up for it later: genfb reads the callback pointer
from the device properties once, in genfb_init(), so it has to exist
before genfb attaches.  A chipset driver that attaches after genfb, or
is loaded with modload(8), has nowhere to hook.

The change registers a mode callback in genfb_grfbus_attach()
unconditionally, next to the colormap one, and forwards it through a
hook pointer that any driver on the port can claim and release at any
time.  It adds no userland interface and changes nothing for a kernel
in which nothing claims the hook.
>How-To-Repeat:
On any mac68k machine with internal video, note that genfb attaches
with "colormap callback provided" and nothing equivalent for the mode
callback, and that dev/wsfb/genfb.c's scp->sc_modecb is therefore NULL
for the life of the device.  There is no way for a driver on this port
to be told that WSDISPLAYIO_SMODE happened.
>Fix:
Applies to sys/arch/mac68k/dev/genfb_grfbus.c and
sys/arch/mac68k/include/video.h, on top of the genfb_grfbus console
pitch change in port-mac68k/60695.  It applies to NetBSD 11.0 and to
-current with no fuzz.

Tested on a Macintosh Centris 650 (DAFB, 640x480x8) under NetBSD 11.0
with an out-of-tree DAFB console driver as the consumer I am developing. 
My console driver claims the hook on load and releases it on unload, 
and receives the mode on both.
It parks and restores the scanout across X and a sixel viewer entering
and leaving mapped mode, with no change to either program.  A kernel
with the change and nothing claiming the hook behaves as before.

Applies with "patch -p1" from the top of usr/src; verified with -F0
(no fuzz) against NetBSD-current 11.99.8 (20260830003849Z) and 11.0.

--- a/sys/arch/mac68k/dev/genfb_grfbus.c
+++ b/sys/arch/mac68k/dev/genfb_grfbus.c
@@ -103,6 +103,39 @@
 
 static struct genfb_colormap_callback gfb_cmcb;
 
+void (*mac68k_genfb_mode_hook)(void *, int) = NULL;
+void *mac68k_genfb_mode_arg = NULL;
+
+static bool
+genfb_grfbus_setmode(struct genfb_softc *sc, int mode)
+{
+
+	if (mac68k_genfb_mode_hook != NULL)
+		(*mac68k_genfb_mode_hook)(mac68k_genfb_mode_arg, mode);
+	return true;
+}
+
+static struct genfb_mode_callback gfb_modecb = {
+	.gmc_setmode = genfb_grfbus_setmode,
+};
+
 static int __unused
 genfb_grfbus_is_console(paddr_t addr)
 {
@@ -207,6 +240,9 @@
 		    (uint64_t)(uintptr_t)&gfb_cmcb);
 	}
 
+	prop_dictionary_set_uint64(dict, "mode_callback",
+	    (uint64_t)(uintptr_t)&gfb_modecb);
+
 	genfb_init(&sc->sc_gen);
 
 	if (sc->sc_gen.sc_width == 0 || sc->sc_gen.sc_fbsize == 0) {
--- a/sys/arch/mac68k/include/video.h
+++ b/sys/arch/mac68k/include/video.h
@@ -56,4 +56,13 @@
 
 extern struct mac68k_video mac68k_video;
 
+extern void (*mac68k_genfb_mode_hook)(void *, int);
+extern void *mac68k_genfb_mode_arg;
+
 #endif /* _CPU_VIDEO_H_ */