Perhaps distcc, with the configure state running natively but the workers running crosscompilers on amd64. I've never done this, but it's documented on https://wiki.netbsd.org/tutorials/pkgsrc/cross_compile_distcc/
It works well, and I use this for all the pkgsrc bulk build architectures I run, except for Alpha where there's a distcc bug I haven't figured out.
https://hackaday.io/project/218-speed-up-pkgsrc-on-retrocomputersIf speed is really a concern, NetBSD runs on the Apple M1, but this seems to be about making good use of an RPi 3.
John