Have you tried patching the configuration of wlroots to not require a newer libdrm version then building it against current native X11?