lib/55897: A mess with complex arithmetics in earmv7hf

[alexander.i.mukhin%gmail.com@localhost]

>Number:         55897
>Category:       lib
>Synopsis:       A mess with complex arithmetics in earmv7hf
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    lib-bug-people
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Wed Dec 30 13:15:00 +0000 2020
>Originator:     Alexander Mukhin
>Release:        9.1_STABLE
>Organization:
>Environment:
NetBSD rpi2 9.1_STABLE NetBSD 9.1_STABLE (MY-RPI2-9-v3) #1: Mon Dec 21 12:31:59 UTC 2020  alexandr1@rpi2:/home/alexandr1/build/usr/obj/sys/arch/evbarm/compile/MY-RPI2-9-v3 evbarm
>Description:
I'm developing a Fourier transform library. While testing my software on
NetBSD/earmv7hf, I found, to my surprise, that even simplest complex 
arithmetics cause the programs to dump core or return garbage, if they
are linked against libm.

Tracking the issue down, I found that it is caused by the builtin
complex arithmetic routines, such as muldc3, taken from compiler-rt and
included in libm. These routines use the standard AAPCS calling
convention, while the compiler calls them via the VFP calling convention 
variant, utilizing VFP registers for parameter passing and result
return.

All this is because the complex arithmetic routines in compiler-rt are
defined with the COMPILER_RT_ABI macro, which expands to
__attribute__((pcs("aapcs"))) and overrides the default ABI.
>How-To-Repeat:
$ cat test.c 
#include <complex.h>
int main (void) {
        double complex z,w,u;
        z=1+I;
        w=2+I;
        u=z*w;
        return 0;
}
$ gcc test.c -lm
$ ./a.out
[1]   Bus error (core dumped) ./a.out

>Fix:
A quick and dirty fix I used for myself was to force the macro
COMPILER_RT_ABI to be undefined when compiling these routines for libm:

$ diff -u src/lib/libm/compiler_rt/Makefile.inc.orig src/lib/libm/compiler_rt/Makefile.inc
--- src/lib/libm/compiler_rt/Makefile.inc.orig  2016-02-27 19:06:56.000000000 +0000
+++ src/lib/libm/compiler_rt/Makefile.inc       2020-12-30 10:15:24.865722208 +0000
@@ -36,5 +36,6 @@
 .  else
 SRCS+= ${src}
 COPTS.${src}+= -Wno-error=missing-prototypes
+COPTS.${src}+= -U__ARM_EABI__
 .  endif
 .endfor