kern/51492: rasops byte order issue

[scole_mail%gmx.com@localhost]

>Number:         51492
>Category:       kern
>Synopsis:       rasops byte order issue
>Confidential:   no
>Severity:       non-critical
>Priority:       medium
>Responsible:    kern-bug-people
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Tue Sep 20 16:05:00 +0000 2016
>Originator:     scole_mail
>Release:        7.0
>Organization:
none
>Environment:
NetBSD dstar 7.0.1_PATCH NetBSD 7.0.1_PATCH (GENERIC) #0: Mon Sep  5 13:24:11 PDT 2016  scole@dstar:/home/scole/nbsd/cvs/7_0/obj/sys/arch/i386/compile/GENERIC i386
>Description:
There may be a byte order issue with rasops(9) where the host byte order is possibly different from the video byte order, see this thread

http://mail-index.netbsd.org/current-users/2016/09/14/msg030132.html

igsfb and platinumfb are two drivers that might be affected.  It was suggested to add a flag like RI_FB_IS_BE.

I don't have the hardware to test this anymore but can describe the issues I had with platinunfb.

RI_BSWAP and kernel option RASOPS_SMALL may be related to this issue
>How-To-Repeat:

>Fix:
This patch fixed my issue for platinumfb 16 bit only, but may not be useful to the overall problem

Apply patch with "cd .../src/sys/dev/rasops ; patch -s -p0 < .../patchfile"

Index: rasops15.c
===================================================================
RCS file: /cvsroot/src/sys/dev/rasops/rasops15.c,v
retrieving revision 1.20
diff -b -u -r1.20 rasops15.c
--- rasops15.c	17 Apr 2012 12:06:25 -0000	1.20
+++ rasops15.c	14 Sep 2016 20:31:52 -0000
@@ -217,17 +217,26 @@
 	bg = ri->ri_devcmap[((u_int)attr >> 16) & 0xf] & 0xffff;
 	stamp_attr = attr;
 
+	/*
+	 * XXX - someone should sanity check but think this was doing
+	 *   stamp[i] = 8|16, stamp[i+1] = 2|4 (little endian)
+	 *   stamp[i] = 4|2, stamp[i+1] = 16|8 (big endian)
+	 * where should be
+	 *   stamp[i] = 2|4, stamp[i+1] = 8|16 (little endian)
+	 *   stamp[i] = 16|8, stamp[i+1] = 4|2 (big endian)
+	 */
 	for (i = 0; i < 32; i += 2) {
 #if BYTE_ORDER == LITTLE_ENDIAN
-		stamp[i] = (i & 16 ? fg : bg);
-		stamp[i] |= ((i & 8 ? fg : bg) << 16);
-		stamp[i + 1] = (i & 4 ? fg : bg);
-		stamp[i + 1] |= ((i & 2 ? fg : bg) << 16);
+            stamp[i] = (i & 4 ? fg : bg);
+            stamp[i] |= ((i & 2 ? fg : bg) << 16);
+            stamp[i + 1] = (i & 16 ? fg : bg);
+            stamp[i + 1] |= ((i & 8 ? fg : bg) << 16);
+
 #else
-		stamp[i] = (i & 2 ? fg : bg);
-		stamp[i] |= ((i & 4 ? fg : bg) << 16);
-		stamp[i + 1] = (i & 8 ? fg : bg);
-		stamp[i + 1] |= ((i & 16 ? fg : bg) << 16);
+            stamp[i] = (i & 8 ? fg : bg);
+            stamp[i] |= ((i & 16 ? fg : bg) << 16);
+            stamp[i + 1] = (i & 2 ? fg : bg);
+            stamp[i + 1] |= ((i & 4 ? fg : bg) << 16);
 #endif
 	}
 }