>Number:         4960
>Category:       pkg
>Synopsis:       new patch to python
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    gnats-admin (GNATS administrator)
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Mon Feb  9 04:05:02 1998
>Last-Modified:
>Originator:     Jaromir Dolecek
>Organization:
	ICS MU, Brno, Czech Republic
>Release:        1.3
>Environment:
	
System: NetBSD saruman.ics.muni.cz 1.3 NetBSD 1.3 (SARUMAN) #3: Wed Jan 21 13:29:49 MET 1998 dolecek@saruman.ics.muni.cz:/usr/home/dolecek/N13/usr/src/sys/arch/i386/compile/SARUMAN i386


>Description:
	There was found new bug in Python 1.5, related to dealing
	with unsigned/signed chars.
>How-To-Repeat:
	Try to use character 0xff in Python source (legitimate char
	of some national encodings).

>Fix:
	As Guido posted:

> Alexander Voinov <avv@isida.ipa.rssi.ru> writes:
> > Two most popular Russian encodings (koi8 and cp1251) use symbol with code 0x
FF.
> > This appears intolerable both for Python 1.4 and Python 1.5 (EOF).

Martin v. Loewis replies:
> As you might now, a work-around is to use \377 in the string instead.
> I agree that this is not really convenient, and should be fixed.

Agreed.  Here's a patch for Parser/tokenizer.c.  Let me know whether
this works:

--- Parser/tokenizer.c.orig	Tue Apr 29 23:03:03 1997
+++ Parser/tokenizer.c	Mon Feb  9 13:05:34 1998
@@ -46,6 +46,14 @@
 /* Don't ever change this -- it would break the portability of Python code */
 #define TABSIZE 8
 
+/* Convert a possibly signed character to a nonnegative int */
+/* XXX This assumes characters are 8 bits wide */
+#ifdef __CHAR_UNSIGNED__
+#define Py_CHARMASK(c)		(c)
+#else
+#define Py_CHARMASK(c)		((c) & 0xff)
+#endif
+
 /* Forward */
 static struct tok_state *tok_new Py_PROTO((void));
 static int tok_nextc Py_PROTO((struct tok_state *tok));
@@ -178,7 +186,7 @@
 {
 	for (;;) {
 		if (tok->cur != tok->inp) {
-			return *tok->cur++; /* Fast path */
+			return Py_CHARMASK(*tok->cur++); /* Fast path */
 		}
 		if (tok->done != E_OK)
 			return EOF;
@@ -197,7 +205,7 @@
 				tok->buf = tok->cur;
 			tok->lineno++;
 			tok->inp = end;
-			return *tok->cur++;
+			return Py_CHARMASK(*tok->cur++);
 		}
 		if (tok->prompt != NULL) {
 			char *new = PyOS_Readline(tok->prompt);
>Audit-Trail:
>Unformatted: