This isn't strictly necessary.

What is necessary, is to start the X server from xdm or another
priviledged process (if you want to avoid that the 666), or to setup
the login in a way that the /dev/grfN is given to the person who
logged in.

And yes, this still might be constructed as a security hole. But then you
wouldn't let people without security clearance come within one meter
of your Amiga's keyboard and mouse, would you?

Regards,
	Ignatios Souvatzis