It took a couple of tries (probably leftovers in objdirs) but I've had it successfully build three times now. The last time I updated all my packages using pkg_rolling-replace and then did a make replace. I'm fairly sure this fixes my problem.