Re: LD_PRELOAD and mixed architectures

[Dave Vitek <dvitek%grammatech.com@localhost>]

On 6/22/2014 12:19 PM, Gary Duzan wrote:
> In Message <53A6F6D7.5040300%grammatech.com@localhost>,
>     Dave Vitek <dvitek%grammatech.com@localhost>wrote:
>
> =>Hi all,
> =>
> =>I'm on an amd64 machine that has a mixture of i386 and amd64 binaries
> =>present.  I want to build a shared object file and use LD_PRELOAD to
> =>inject it into every process in some process tree. The executables in
> =>the tree could be a mixture of amd64 and i386 executables.  Many other
> =>operating systems have a way of using LD_PRELOAD in a mixed architecture
> =>environment, but I'm not seeing a path to victory on netbsd.
> =>
> =>I had a look at the source code for ld.so (the elf one anyway).  It
> =>doesn't search search LD_LIBRARY_PATH -- it just tries to open() the
> =>filename in LD_PRELOAD relative to CWD and blows up if the architecture
> =>doesn't match the process.  Other operating systems facilitate this sort
> =>of thing in various ways:
> =>
> =>Solaris: LD_PRELOAD_32 and LD_PRELOAD_64
> =>Linux: Search LD_LIBRARY_PATH, only fail if every attempt fails. Also,
> =>"$LIB" is magical.
> =>Mac OS X: Universal binaries
> =>
> =>One (ugly) thought I had is to have the shared object file hook every
> =>function in the "exec" family and fix LD_PRELOAD so it points at the
> =>architecture of the binary about to be exec'd before delegating to the
> =>real exec.  Anyone have a less ugly alternative that doesn't require
> =>modifying system files (e.g., adjusting ld.so)?
>
>     Just a thought, but have you tried putting a 32-bit version of
> the library under /emul/netbsd32 ?
>
>                                 Gary Duzan
Looks like things loaded due to LD_PRELOAD don't search /emul.  Here is 
a short experiment:

$ echo "int main(){ return 0; }" > nop.c
$ gcc -m32 nop.c -o nop
$ LD_PRELOAD=app/lib64/libhook.so ./nop
app/lib64/libhook.so: unrecognized file format2 [2 != 1]
$ LD_PRELOAD=app/lib/libhook.so ./nop
$ ls /emul/netbsd32/
libhook.so
$ LD_PRELOAD=libhook.so ./nop
Cannot open "libhook.so"
$ gcc -m64 nop.c -o nop
$ LD_PRELOAD=libhook.so ./nop
Cannot open "libhook.so"

It occurs to me that linux emulation processes may have similar challenges.

There is an existing problem report here:
http://gnats.netbsd.org/47509

Relevant code from ld.so_elf confirms that it just does a straight up 
open() of the values in the environment variable and doesn't search 
anywhere:

// ld_preload is raw value of env variable


        if (ld_preload) {
                /*
                 * Pre-load user-specified objects after the main program
                 * but before any shared object dependencies.
                 */
                dbg(("preloading objects"));
                if (_rtld_preload(ld_preload) == -1)
                        _rtld_die();
        }
...


int
_rtld_preload(const char *preload_path)
{
        const char *path;
        char *cp, *buf;
        int status = 0;

        if (preload_path != NULL && *preload_path != '\0') {
                cp = buf = xstrdup(preload_path);
                while ((path = strsep(&cp, " :")) != NULL && status == 0) {
                        if (!_rtld_load_object(path, _RTLD_MAIN))
                                status = -1;
                        else
                                dbg((" preloaded \"%s\"", path));
                }
                xfree(buf);
        }

        return status;
}


...

Obj_Entry *
_rtld_load_object(const char *filepath, int flags)
{
        Obj_Entry *obj;
        int fd = -1;
        struct stat sb;
        size_t pathlen = strlen(filepath);

        for (obj = _rtld_objlist->next; obj != NULL; obj = obj->next)
                if (pathlen == obj->pathlen && !strcmp(obj->path, filepath))
                        break;

        /*
         * If we didn't find a match by pathname, open the file and check
         * again by device and inode.  This avoids false mismatches caused
         * by multiple links or ".." in pathnames.
         *
         * To avoid a race, we open the file and use fstat() rather than
         * using stat().
         */
        if (obj == NULL) {
                if ((fd = open(filepath, O_RDONLY)) == -1) {
                        _rtld_error("Cannot open \"%s\"", filepath);
                        return NULL;
                }