tech-toolchain archive

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index][Old Index]

gcc -O2 produces invalid object code (x86_64, netbsd-5 branch)

Hi all,

I've been debugging a strange libpam segfault on the netbsd-5 branch (amd64) and I've concluded that gcc -O2 is incorrectly compiling the pam_end() function in src/dist/openpam/lib/pam_end.c. gcc -O0 works fine.

It looks like lines 59 and 60 are simply skipped. Either that or they are reordered below line 63. If pam_end() is passed a NULL pamh, it'll segfault when it dereferences pamh at line 63.

I verified that this is a compiler problem by looking at the assembler output. The following are the first few instructions emitted by gcc -O2 -S (see end of email for full command line):

        # amd64 abi requires callee to preserve %r12
        pushq   %r12

        # set up call: _openpam_log(PAM_LOG_DEBUG, "pam_end", "enter")
        # _openpam_log() is variadic, and the amd64 abi requires the
        # caller to put the number of floating point args in %rax when
        # calling a variadic function.  there are no fp arguments in
        # this case so set it to 0.
        xorl    %eax, %eax

        # amd64 abi uses %rsi for the 2nd argument.  save pam_end()'s
        # 2nd argument (status) in %r12 since %rsi will be used to pass
        # "pam_end" to _openpam_log().
        movl    %esi, %r12d

        # amd64 abi uses %rdx for the 3rd argument.  load the register
        # with the string "enter"
        movl    $.LC0, %edx

        # put "pam_end" in the 2nd argument register
        movl    $__PRETTY_FUNCTION__.2637, %esi

        # amd64 abi requires callee to preserve %rbp
        pushq   %rbp

        # amd64 abi uses %rdi for the 1st argument.  save pam_end()'s
        # 1st argument (pamh) in %rbp so that %rdi can be used for the
        # call to _openpam_log().
        movq    %rdi, %rbp

        # put PAM_LOG_DEBUG (which equals 0) in %rdi
        xorl    %edi, %edi

        # %rbx will be used after the call to _openpam_log(), and the
        # amd64 abi requires callee to preserve it
        pushq   %rbx

        # execute _openpam_log(PAM_LOG_DEBUG, "pam_end", "enter")
        call    _openpam_log

        # copy pamh->module_data to %rbx (the variable dp).  the value
        # 176 is offsetof(pam_handle_t, module_data).  THIS RESULTS IN A
        # SEGFAULT IF pamh is NULL!
        movq    176(%rbp), %rbx

Notice that there is no check for NULL in the above assembly before pamh is dereferenced, yet the 'if (pamh == NULL)' statement is in the source code, clear as day.

Changing -O2 to -O0 produces correct object code.

I haven't tried any other version of gcc. netbsd-5's gcc is quite old, so I wouldn't be surprised if the problem has already been fixed.

gcc command used to get assembler output:
/usr/obj//usr-5/tools/bin/x86_64--netbsd-gcc -Wall -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Wno-sign-compare -Wno-traditional -Wa,--fatal-warnings -Wreturn-type -Wswitch -Wshadow -Wcast-qual -Wwrite-strings -Wextra -Wno-unused-parameter -Werror -fstack-protector -Wstack-protector --param ssp-buffer-size=1 -ggdb -O2 -DLIB_MAJ=1 -DOPENPAM_MODULES_DIR=\"/usr/lib/security\" -D_FORTIFY_SOURCE=2 -DDEBUG=1 -DOPENPAM_DEBUG=1 -nostdinc -isystem /usr/obj//usr-5/destdir/amd64/usr/include -S /usr/src/dist/openpam/lib/pam_end.c -o pam_end.s

Tools were initially built by running "BUILD-NetBSD" from the etcmanage pkgsrc package.


Home | Main Index | Thread Index | Old Index