port-vax/60100: vax libm: ALTENTRY labels land on NOP padding instead of entry mask, causing SIGILL on CALLS

["pyhirricturtle%gmail.com@localhost via gnats" <gnats-admin%NetBSD.org@localhost>]

>Number:         60100
>Category:       port-vax
>Synopsis:       vax libm: ALTENTRY labels land on NOP padding instead of entry mask, causing SIGILL on CALLS
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    port-vax-maintainer
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Wed Mar 18 17:20:00 +0000 2026
>Originator:     FireTurtle
>Release:        NetBSD 10 (also present in -current trunk)
>Organization:
>Environment:
System: NetBSD/vax (SIMH MicroVAX 3900)
>Description:
In several vax libm assembly files, ALTENTRY() is placed immediately before ENTRY(). The intent is for the ALTENTRY symbol to share the following ENTRY's .word register-save mask. However, ENTRY() begins with _ALIGN_TEXT, which inserts NOP padding (0x0101...) for alignment. The ALTENTRY label ends up pointing at the NOP padding rather than the .word mask.

When the function is called via the ALTENTRY name using CALLS, the processor reads the NOPs as the entry mask (e.g., 0x0101), saves the wrong registers, and begins execution at the wrong offset. This results in SIGILL or other undefined behavior.

Affected files:
- lib/libm/arch/vax/n_cbrt.S: ALTENTRY(cbrt) before ENTRY(d_cbrt, 0x00c0)
- lib/libm/arch/vax/n_cabs.S: ALTENTRY(cabs) before ENTRY(_hypot, 0x8040)
- lib/libm/arch/vax/n_scalbn.S: ALTENTRY(_scalbnl) before ENTRY(_scalbn, 0)

For example in n_cbrt.S:

    ALTENTRY(cbrt)       /* label here */
    ENTRY(d_cbrt, 0x00c0)  /* _ALIGN_TEXT inserts NOPs, then .word 0x00c0 */

objdump -d shows the cbrt symbol points several bytes before d_cbrt -- into the NOP alignment padding -- rather than at d_cbrt's .word 0x00c0 entry mask.

This may be related to the historical port-vax/18483 (2002), which reported SIGILL in atan() -- the same class of symptom. That PR noted static linking worked but dynamic linking crashed.
>How-To-Repeat:
1. Inspect the generated binary:
   objdump -d libm.so | grep -A5 '<cbrt>'
   The cbrt symbol points into NOP padding, not at the .word entry mask.

2. Or call cbrt(8.0) from a dynamically-linked program on vax:
   #include <math.h>
   #include <stdio.h>
   int main() { printf("%f\n", cbrt(8.0)); return 0; }
   This will crash with SIGILL.
>Fix:
The simplest fix for each affected file is to move ALTENTRY after the alignment and .word mask, making it an alias that points to the same address as the ENTRY name. For example in n_cbrt.S:

    ENTRY(d_cbrt, 0x00c0)
    ALTENTRY(cbrt)        /* now points at same address as d_cbrt */

Alternatively, give each alternate name its own ENTRY() with the correct mask.

The broader question is whether the ALTENTRY macro itself should be redefined to account for _ALIGN_TEXT in ENTRY, but that is a design decision for the port maintainers.