gcc -O2 produces invalid object code (x86_64, netbsd-5 branch)

[Richard Hansen <rhansen%bbn.com@localhost>]

Hi all,

I've been debugging a strange libpam segfault on the netbsd-5 branch 
(amd64) and I've concluded that gcc -O2 is incorrectly compiling the 
pam_end() function in src/dist/openpam/lib/pam_end.c.  gcc -O0 works fine.

It looks like lines 59 and 60 are simply skipped.  Either that or they 
are reordered below line 63.  If pam_end() is passed a NULL pamh, it'll 
segfault when it dereferences pamh at line 63.

I verified that this is a compiler problem by looking at the assembler 
output.  The following are the first few instructions emitted by gcc -O2 
-S (see end of email for full command line):

pam_end:
        # amd64 abi requires callee to preserve %r12
        pushq   %r12

        # set up call: _openpam_log(PAM_LOG_DEBUG, "pam_end", "enter")
        # _openpam_log() is variadic, and the amd64 abi requires the
        # caller to put the number of floating point args in %rax when
        # calling a variadic function.  there are no fp arguments in
        # this case so set it to 0.
        xorl    %eax, %eax

        # amd64 abi uses %rsi for the 2nd argument.  save pam_end()'s
        # 2nd argument (status) in %r12 since %rsi will be used to pass
        # "pam_end" to _openpam_log().
        movl    %esi, %r12d

        # amd64 abi uses %rdx for the 3rd argument.  load the register
        # with the string "enter"
        movl    $.LC0, %edx

        # put "pam_end" in the 2nd argument register
        movl    $__PRETTY_FUNCTION__.2637, %esi

        # amd64 abi requires callee to preserve %rbp
        pushq   %rbp

        # amd64 abi uses %rdi for the 1st argument.  save pam_end()'s
        # 1st argument (pamh) in %rbp so that %rdi can be used for the
        # call to _openpam_log().
        movq    %rdi, %rbp

        # put PAM_LOG_DEBUG (which equals 0) in %rdi
        xorl    %edi, %edi

        # %rbx will be used after the call to _openpam_log(), and the
        # amd64 abi requires callee to preserve it
        pushq   %rbx

        # execute _openpam_log(PAM_LOG_DEBUG, "pam_end", "enter")
        call    _openpam_log

        # copy pamh->module_data to %rbx (the variable dp).  the value
        # 176 is offsetof(pam_handle_t, module_data).  THIS RESULTS IN A
        # SEGFAULT IF pamh is NULL!
        movq    176(%rbp), %rbx

Notice that there is no check for NULL in the above assembly before pamh 
is dereferenced, yet the 'if (pamh == NULL)' statement is in the source 
code, clear as day.

Changing -O2 to -O0 produces correct object code.

I haven't tried any other version of gcc.  netbsd-5's gcc is quite old, 
so I wouldn't be surprised if the problem has already been fixed.

gcc command used to get assembler output:
/usr/obj//usr-5/tools/bin/x86_64--netbsd-gcc -Wall -Wstrict-prototypes 
-Wmissing-prototypes -Wpointer-arith -Wno-sign-compare -Wno-traditional 
-Wa,--fatal-warnings -Wreturn-type -Wswitch -Wshadow -Wcast-qual 
-Wwrite-strings -Wextra -Wno-unused-parameter  -Werror -fstack-protector 
-Wstack-protector --param ssp-buffer-size=1 -ggdb -O2  -DLIB_MAJ=1 
-DOPENPAM_MODULES_DIR=\"/usr/lib/security\"  -D_FORTIFY_SOURCE=2 
-DDEBUG=1 -DOPENPAM_DEBUG=1 -nostdinc -isystem 
/usr/obj//usr-5/destdir/amd64/usr/include -S 
/usr/src/dist/openpam/lib/pam_end.c -o pam_end.s

Tools were initially built by running "BUILD-NetBSD" from the etcmanage 
pkgsrc package.

-Richard