more on proff crash building native

[Brad Parker <brad%heeltoe.com@localhost>]

I've been digging into the groff crash doing a native build.  

I did a few experiments.  libm is composed of "non-ieee" floating point
routines written (mostly) in vax assembler.

While building netbsd groff calls hypot() which crashes.  If you replace
hypot() with a simple version written in C it works fine.

So, I wrote a simple test program which is about 3 lines.

If I link directly with the assembler routines it works fine (i.e. link
with n_cabs.o n_sqrt.o & n_infnan.o).  If I link -static with -lm it
works fine.  If I like dynamic with -lm it crashes.

Originally I thought the problem was due to the "infnan()" function
doing an illegal instruction.  But I found the real problem has to do
with the code in libm.so (__libm_dsqrt_r5) doing a rsb with the stack
wrong.  Basically the 2 double args (4 words) are still on the stack.
If I modify the stack pointer (set $sp=$sp+16) and continue it works
as expected.

I am a newbie when it comes to vax calling conventions and shared
libraries.  It *looks* like the problem is due to the vax libm assembler
code being put into a shared library (i.e. pic) and somehow not being
correct.  (i.e. if all the code were in C I bet it would "just work")

The functions I'm looking at are src/lib/libm/arch/vax/*.S

I wonder if any of those functions work when called from libm.so; I'll
make a regression test for all of them in any case.

If anyone is more knowledgeable about this and wants to help out, please
let me know.  I'll muddle through it in my spare time but I suspect the
answer is pretty simple.

(and boy, the shared library dynamic binding code is *slow* and lengthy!)

-brad