GCC 5.3 vs. ssh - oh the stacks you'll follow

[matthew green <mrg%eterna.com.au@localhost>]

hi folks.


it had been reported to me from 2 people that "ssh" on amd64 with GCC 5.3
crashes inside SHA1, and we tracked it down to a misaligned stack.  i've
figured out where that misaligned stack comes from and it's actually coming
from GCC itself.  ssh has a "ssh_digest" family of functions, that are
frontends to md5/sha*/etc, and ssh_digest_final() is being miscompiled such
that it leaves the stack misaligned when calling other functions, and when
those functions rely upon the stack alignment, apps crash.

this is the x86 assembly (see the entire function at the end of this mail,
it is only about 40 instructions):

Dump of assembler code for function ssh_digest_final:
   0x00007f7ff484573a <+0>:     push   %rbx
   0x00007f7ff484573b <+1>:     sub    $0x8,%rsp

and this is the last time the stack is adjusted before the epilogue (there
are two exit points in the emitted code):

   0x00007f7ff4845799 <+95>:    add    $0x8,%rsp
   0x00007f7ff484579d <+99>:    pop    %rbx
   0x00007f7ff484579e <+100>:   retq

and:

   0x00007f7ff48457a8 <+110>:   add    $0x8,%rsp
   0x00007f7ff48457ac <+114>:   pop    %rbx
   0x00007f7ff48457ad <+115>:   retq

ssh_digest_final() is pretty simple, and simply adding this to the top of
the function makes it compile properly (as does just -O1):

	char buf[8]; memset(buf, 0, sizeof buf);

not sure what the problem here is, i will try to make a very small test
case so we can file a PR upstream.


.mrg.


Dump of assembler code for function ssh_digest_final:
   0x00007f7ff484573a <+0>:     push   %rbx
   0x00007f7ff484573b <+1>:     sub    $0x8,%rsp
   0x00007f7ff484573f <+5>:     mov    (%rdi),%ecx
   0x00007f7ff4845741 <+7>:     cmp    $0x5,%ecx
   0x00007f7ff4845744 <+10>:    ja     0x7f7ff484579f <ssh_digest_final+101>
   0x00007f7ff4845746 <+12>:    movslq %ecx,%rbx
   0x00007f7ff4845749 <+15>:    shl    $0x5,%rbx
   0x00007f7ff484574d <+19>:    mov    %rbx,%rax
   0x00007f7ff4845750 <+22>:    add    0x23ead9(%rip),%rax        # 0x7f7ff4a84230
   0x00007f7ff4845757 <+29>:    cmp    (%rax),%ecx
   0x00007f7ff4845759 <+31>:    mov    $0x0,%ebx
   0x00007f7ff484575e <+36>:    cmove  %rax,%rbx
   0x00007f7ff4845762 <+40>:    mov    %edx,0x4(%rsp)
   0x00007f7ff4845766 <+44>:    mov    $0xffffffff,%eax
   0x00007f7ff484576b <+49>:    cmp    %rax,%rdx
   0x00007f7ff484576e <+52>:    ja     0x7f7ff48457a3 <ssh_digest_final+105>
   0x00007f7ff4845770 <+54>:    cmp    0x10(%rbx),%rdx
   0x00007f7ff4845774 <+58>:    jb     0x7f7ff48457a3 <ssh_digest_final+105>
   0x00007f7ff4845776 <+60>:    lea    0x4(%rsp),%rdx
   0x00007f7ff484577b <+65>:    add    $0x8,%rdi
   0x00007f7ff484577f <+69>:    callq  0x7f7ff48129d0 <EVP_DigestFinal_ex@plt>
   0x00007f7ff4845784 <+74>:    cmp    $0x1,%eax
   0x00007f7ff4845787 <+77>:    jne    0x7f7ff48457ae <ssh_digest_final+116>
   0x00007f7ff4845789 <+79>:    mov    0x4(%rsp),%eax
   0x00007f7ff484578d <+83>:    cmp    0x10(%rbx),%rax
   0x00007f7ff4845791 <+87>:    setne  %al
   0x00007f7ff4845794 <+90>:    movzbl %al,%eax
   0x00007f7ff4845797 <+93>:    neg    %eax
   0x00007f7ff4845799 <+95>:    add    $0x8,%rsp
   0x00007f7ff484579d <+99>:    pop    %rbx
   0x00007f7ff484579e <+100>:   retq
   0x00007f7ff484579f <+101>:   xor    %ebx,%ebx
   0x00007f7ff48457a1 <+103>:   jmp    0x7f7ff4845762 <ssh_digest_final+40>
   0x00007f7ff48457a3 <+105>:   mov    $0xfffffff6,%eax
   0x00007f7ff48457a8 <+110>:   add    $0x8,%rsp
   0x00007f7ff48457ac <+114>:   pop    %rbx
   0x00007f7ff48457ad <+115>:   retq
   0x00007f7ff48457ae <+116>:   mov    $0xffffffea,%eax
   0x00007f7ff48457b3 <+121>:   jmp    0x7f7ff4845799 <ssh_digest_final+95>