One thing to note here is that, in an a.out world, ld.so.conf is
actually used for 2 orthogonal purposes: to tell ldconfig which
directories to generate hints/cache entries for, and (indirectly,
using the global cache file) to tell ld.so which directories to look
in by default.

Part of the misunderstanding here stems from the fact that these two
bits of functionality never should have been combined in the first
place.

I would actually suggest going in the opposite direction; that is,
modifying the a.out ldconfig/ld.so to generate a cache file per
directory, and use ld.so.conf (and the ldconfig arguments) *only* to
determine which directories to build cache files in.