So I've done some hunting around, and it appears that the bulk of
ld.so overhead is (perhaps not surprisingly) spent in
_rtld_relocate_objects().  I'm looking at some options for speeding
this up and will send more mail about it later.