lib/46659: feenableexcept and fedisableexcept return too many bits on x86

[Taylor R Campbell <campbell+netbsd%mumble.net@localhost>]

>Number:         46659
>Category:       lib
>Synopsis:       feenableexcept and fedisableexcept return too many bits on x86
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    lib-bug-people
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Thu Jul 05 02:25:01 +0000 2012
>Originator:     Taylor R Campbell <campbell+netbsd%mumble.net@localhost>
>Release:        NetBSD 6.99.7
>Organization:
>Environment:
>Description:

        On i386 (i387) and amd64, feenableexcept and fedisableexcept
        return negative results with bits not in FE_ALL_EXCEPT.  -1 is
        supposed to mean failure, and there shouldn't be any bits not
        in FE_ALL_EXCEPT.

>How-To-Repeat:

        Call feenableexcept or fedisableexcept on i386 or amd64.

>Fix:

        Apply the following patch.  OK to commit?

Index: lib/libm/arch/i387/fenv.c
===================================================================
RCS file: /cvsroot/src/lib/libm/arch/i387/fenv.c,v
retrieving revision 1.3
diff -p -u -r1.3 fenv.c
--- lib/libm/arch/i387/fenv.c   1 Aug 2010 06:34:38 -0000       1.3
+++ lib/libm/arch/i387/fenv.c   5 Jul 2012 01:13:14 -0000
@@ -472,7 +472,7 @@ feenableexcept(int mask)
                __ldmxcsr(mxcsr);
        }
 
-       return (~omask);
+       return (FE_ALL_EXCEPT & ~omask);
 }
 
 int
@@ -496,7 +496,7 @@ fedisableexcept(int mask)
                __ldmxcsr(mxcsr);
        }
 
-       return (~omask);
+       return (FE_ALL_EXCEPT & ~omask);
 }
 
 int
Index: lib/libm/arch/x86_64/fenv.c
===================================================================
RCS file: /cvsroot/src/lib/libm/arch/x86_64/fenv.c,v
retrieving revision 1.1
diff -p -u -r1.1 fenv.c
--- lib/libm/arch/x86_64/fenv.c 31 Jul 2010 21:47:53 -0000      1.1
+++ lib/libm/arch/x86_64/fenv.c 5 Jul 2012 01:13:14 -0000
@@ -483,7 +483,7 @@ feenableexcept(int mask)
        mxcsr &= ~(mask << _SSE_EMASK_SHIFT);
        __ldmxcsr(mxcsr);
 
-       return (~omask);
+       return (FE_ALL_EXCEPT & ~omask);
 
 }
 
@@ -505,7 +505,7 @@ fedisableexcept(int mask)
        mxcsr |= mask << _SSE_EMASK_SHIFT;
        __ldmxcsr(mxcsr);
 
-       return (~omask);
+       return (FE_ALL_EXCEPT & ~omask);
 }
 
 int