pkgsrc-Changes archive
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index][Old Index]
CVS commit: pkgsrc/lang/nawk
Module Name: pkgsrc
Committed By: vins
Date: Sun Sep 17 10:32:06 UTC 2023
Modified Files:
pkgsrc/lang/nawk: DESCR Makefile
pkgsrc/lang/nawk/files: FIXES README awk.h b.c lex.c lib.c main.c
nawk.1 proto.h run.c tran.c
Log Message:
lang/nawk: downgrade to 20230909.
Partially revert previous commit, by downgrading the package to the
most recent release tag supporting ASCII encoded input files (and
processing strings as sequences of bytes).
This is needed by security/mozilla-rootcerts and likely other packages;
see https://mail-index.netbsd.org/tech-pkg/2023/09/17/msg028190.html.
This version incorporates all the changes described in the FIXES file up
to 2023-09-09, minus support for UTF-8 and comma-separated values (CSV)
input.
To generate a diff of this commit:
cvs rdiff -u -r1.3 -r1.4 pkgsrc/lang/nawk/DESCR
cvs rdiff -u -r1.45 -r1.46 pkgsrc/lang/nawk/Makefile
cvs rdiff -u -r1.4 -r1.5 pkgsrc/lang/nawk/files/FIXES \
pkgsrc/lang/nawk/files/README pkgsrc/lang/nawk/files/lex.c \
pkgsrc/lang/nawk/files/main.c
cvs rdiff -u -r1.5 -r1.6 pkgsrc/lang/nawk/files/awk.h \
pkgsrc/lang/nawk/files/b.c pkgsrc/lang/nawk/files/run.c
cvs rdiff -u -r1.6 -r1.7 pkgsrc/lang/nawk/files/lib.c \
pkgsrc/lang/nawk/files/proto.h pkgsrc/lang/nawk/files/tran.c
cvs rdiff -u -r1.3 -r1.4 pkgsrc/lang/nawk/files/nawk.1
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
Modified files:
Index: pkgsrc/lang/nawk/DESCR
diff -u pkgsrc/lang/nawk/DESCR:1.3 pkgsrc/lang/nawk/DESCR:1.4
--- pkgsrc/lang/nawk/DESCR:1.3 Tue Sep 12 19:16:52 2023
+++ pkgsrc/lang/nawk/DESCR Sun Sep 17 10:32:05 2023
@@ -1,6 +1,5 @@
The one, true implementation of the AWK pattern-directed scanning and
processing language, by one of the language's creators, Brian Kernighan.
-This is the version of awk described in The AWK Programming Language,
-Second Edition, by Al Aho, Brian Kernighan, and Peter Weinberger
-(Addison-Wesley, 2024, ISBN-13 978-0138269722, ISBN-10 0138269726).
-It is also known as new awk, or nawk.
+This is the version of awk described in "The AWK Programming Language",
+by Al Aho, Brian Kernighan, and Peter Weinberger (Addison-Wesley,
+1988, ISBN 0-201-07981-X). It is also known as new awk, or nawk.
Index: pkgsrc/lang/nawk/Makefile
diff -u pkgsrc/lang/nawk/Makefile:1.45 pkgsrc/lang/nawk/Makefile:1.46
--- pkgsrc/lang/nawk/Makefile:1.45 Tue Sep 12 19:16:52 2023
+++ pkgsrc/lang/nawk/Makefile Sun Sep 17 10:32:05 2023
@@ -1,6 +1,6 @@
-# $NetBSD: Makefile,v 1.45 2023/09/12 19:16:52 vins Exp $
+# $NetBSD: Makefile,v 1.46 2023/09/17 10:32:05 vins Exp $
-DISTNAME= nawk-20230911
+DISTNAME= nawk-20230909
CATEGORIES= lang
MASTER_SITES= # empty
DISTFILES= # empty
Index: pkgsrc/lang/nawk/files/FIXES
diff -u pkgsrc/lang/nawk/files/FIXES:1.4 pkgsrc/lang/nawk/files/FIXES:1.5
--- pkgsrc/lang/nawk/files/FIXES:1.4 Tue Sep 12 19:16:52 2023
+++ pkgsrc/lang/nawk/files/FIXES Sun Sep 17 10:32:05 2023
@@ -25,27 +25,6 @@ THIS SOFTWARE.
This file lists all bug fixes, changes, etc., made since the AWK book
was sent to the printers in August 1987.
-Sep 11, 2023:
- Added --csv option to enable processing of comma-separated
- values inputs. When --csv is enabled, fields are separated
- by commas, fields may be quoted with " double quotes, fields
- may contain embedded newlines.
-
- If no explicit separator argument is provided, split() uses
- the setting of --csv to determine how fields are split.
-
- Strings may now contain UTF-8 code points (not necessarily
- characters). Functions that operate on characters, like
- length, substr, index, match, etc., use UTF-8, so the length
- of a string of 3 emojis is 3, not 12 as it would be if bytes
- were counted.
-
- Regular expressions are processes as UTF-8.
-
- Unicode literals can be written as \u followed by one
- to eight hexadecimal digits. These may appear in strings and
- regular expressions.
-
Sep 06, 2023:
Fix edge case where FS is changed on commandline. Thanks to
Gordon Shephard and Miguel Pineiro Jr.
Index: pkgsrc/lang/nawk/files/README
diff -u pkgsrc/lang/nawk/files/README:1.4 pkgsrc/lang/nawk/files/README:1.5
--- pkgsrc/lang/nawk/files/README:1.4 Tue Sep 12 19:16:52 2023
+++ pkgsrc/lang/nawk/files/README Sun Sep 17 10:32:05 2023
@@ -1,38 +1,8 @@
# The One True Awk
This is the version of `awk` described in _The AWK Programming Language_,
-Second Edition, by Al Aho, Brian Kernighan, and Peter Weinberger
-(Addison-Wesley, 2024, ISBN-13 978-0138269722, ISBN-10 0138269726).
-
-## What's New? ##
-
-This version of Awk handles UTF-8 and comma-separated values (CSV) input.
-
-### Strings ###
-
-Functions that process strings now count Unicode code points, not bytes;
-this affects `length`, `substr`, `index`, `match`, `split`,
-`sub`, `gsub`, and others. Note that code
-points are not necessarily characters.
-
-UTF-8 sequences may appear in literal strings and regular expressions.
-Aribtrary characters may be included with `\u` followed by 1 to 8 hexadecimal digits.
-
-### Regular expressions ###
-
-Regular expressions may include UTF-8 code points, including `\u`.
-Character classes are likely to be limited to about 256 characters
-when expanded.
-
-### CSV ###
-
-The option `--csv` turns on CSV processing of input:
-fields are separated by commas, fields may be quoted with
-double-quote (`"`) characters, fields may contain embedded newlines.
-In CSV mode, `FS` is ignored.
-
-If no explicit separator argument is provided,
-field-splitting in `split` is determined by CSV mode.
+by Al Aho, Brian Kernighan, and Peter Weinberger
+(Addison-Wesley, 1988, ISBN 0-201-07981-X).
## Copyright
@@ -96,21 +66,23 @@ The program itself is created by
make
which should produce a sequence of messages roughly like this:
-
- yacc -d -b awkgram awkgram.y
- yacc: 44 shift/reduce conflicts, 85 reduce/reduce conflicts.
- cc -g -Wall -pedantic -Wcast-qual -O2 -c -o awkgram.tab.o awkgram.tab.c
- cc -g -Wall -pedantic -Wcast-qual -O2 -c -o b.o b.c
- cc -g -Wall -pedantic -Wcast-qual -O2 -c -o main.o main.c
- cc -g -Wall -pedantic -Wcast-qual -O2 -c -o parse.o parse.c
- cc -g -Wall -pedantic -Wcast-qual -O2 maketab.c -o maketab
- ./maketab awkgram.tab.h >proctab.c
- cc -g -Wall -pedantic -Wcast-qual -O2 -c -o proctab.o proctab.c
- cc -g -Wall -pedantic -Wcast-qual -O2 -c -o tran.o tran.c
- cc -g -Wall -pedantic -Wcast-qual -O2 -c -o lib.o lib.c
- cc -g -Wall -pedantic -Wcast-qual -O2 -c -o run.o run.c
- cc -g -Wall -pedantic -Wcast-qual -O2 -c -o lex.o lex.c
- cc -g -Wall -pedantic -Wcast-qual -O2 awkgram.tab.o b.o main.o parse.o proctab.o tran.o lib.o run.o lex.o -lm
+
+ yacc -d awkgram.y
+ conflicts: 43 shift/reduce, 85 reduce/reduce
+ mv y.tab.c ytab.c
+ mv y.tab.h ytab.h
+ cc -c ytab.c
+ cc -c b.c
+ cc -c main.c
+ cc -c parse.c
+ cc maketab.c -o maketab
+ ./maketab >proctab.c
+ cc -c proctab.c
+ cc -c tran.c
+ cc -c lib.c
+ cc -c run.c
+ cc -c lex.c
+ cc ytab.o b.o main.o parse.o proctab.o tran.o lib.o run.o lex.o -lm
This produces an executable `a.out`; you will eventually want to
move this to some place like `/usr/bin/awk`.
@@ -130,9 +102,14 @@ the standard developer tools.
You can also use `make CC=g++` to build with the GNU C++ compiler,
should you choose to do so.
+The version of `malloc` that comes with some systems is sometimes
+astonishly slow. If `awk` seems slow, you might try fixing that.
+More generally, turning on optimization can significantly improve
+`awk`'s speed, perhaps by 1/3 for highest levels.
+
## A Note About Releases
-We don't usually do releases.
+We don't usually do releases.
## A Note About Maintenance
@@ -143,4 +120,5 @@ is not at the top of our priority list.
#### Last Updated
-Sun Sep 3 09:26:43 EDT 2023
+Sun 23 Jan 2022 03:48:01 PM EST
+
Index: pkgsrc/lang/nawk/files/lex.c
diff -u pkgsrc/lang/nawk/files/lex.c:1.4 pkgsrc/lang/nawk/files/lex.c:1.5
--- pkgsrc/lang/nawk/files/lex.c:1.4 Tue Sep 12 19:16:52 2023
+++ pkgsrc/lang/nawk/files/lex.c Sun Sep 17 10:32:06 2023
@@ -368,8 +368,6 @@ int yylex(void)
}
}
-extern int runetochar(char *str, int c);
-
int string(void)
{
int c, n;
@@ -417,7 +415,7 @@ int string(void)
*bp++ = n;
break;
- case 'x': /* hex \x0-9a-fA-F (exactly two) */
+ case 'x': /* hex \x0-9a-fA-F + */
{
int i;
@@ -443,27 +441,6 @@ int string(void)
break;
}
- case 'u': /* utf \u0-9a-fA-F (1..8) */
- {
- int i;
-
- n = 0;
- for (i = 0; i < 8; i++) {
- c = input();
- if (!isxdigit(c) || c == 0)
- break;
- c = tolower(c);
- n *= 16;
- if (isdigit(c))
- n += (c - '0');
- else
- n += 10 + (c - 'a');
- }
- unput(c);
- bp += runetochar(bp, n);
- break;
- }
-
default:
*bp++ = c;
break;
Index: pkgsrc/lang/nawk/files/main.c
diff -u pkgsrc/lang/nawk/files/main.c:1.4 pkgsrc/lang/nawk/files/main.c:1.5
--- pkgsrc/lang/nawk/files/main.c:1.4 Tue Sep 12 19:16:52 2023
+++ pkgsrc/lang/nawk/files/main.c Sun Sep 17 10:32:06 2023
@@ -22,7 +22,7 @@ ARISING OUT OF OR IN CONNECTION WITH THE
THIS SOFTWARE.
****************************************************************/
-const char *version = "version 20230911";
+const char *version = "version 20230909";
#define DEBUG
#include <stdio.h>
@@ -49,8 +49,6 @@ static size_t maxpfile; /* max program f
static size_t npfile; /* number of filenames */
static size_t curpfile; /* current filename */
-bool CSV = false; /* true for csv input */
-
bool safe = false; /* true => "safe" mode */
static noreturn void fpecatch(int n
@@ -152,12 +150,6 @@ int main(int argc, char *argv[])
argv++;
break;
}
- if (strcmp(argv[1], "--csv") == 0) { /* turn on csv input processing */
- CSV = true;
- argc--;
- argv++;
- continue;
- }
switch (argv[1][1]) {
case 's':
if (strcmp(argv[1], "-safe") == 0)
Index: pkgsrc/lang/nawk/files/awk.h
diff -u pkgsrc/lang/nawk/files/awk.h:1.5 pkgsrc/lang/nawk/files/awk.h:1.6
--- pkgsrc/lang/nawk/files/awk.h:1.5 Tue Sep 12 19:16:52 2023
+++ pkgsrc/lang/nawk/files/awk.h Sun Sep 17 10:32:05 2023
@@ -78,8 +78,6 @@ extern char **SUBSEP;
extern Awkfloat *RSTART;
extern Awkfloat *RLENGTH;
-extern bool CSV; /* true for csv input */
-
extern char *record; /* points to $0 */
extern int lineno; /* line number in awk program */
extern int errorflag; /* 1 if error has occurred */
@@ -227,8 +225,7 @@ extern int pairstack[], paircnt;
/* structures used by regular expression matching machinery, mostly b.c: */
-#define NCHARS (1256+3) /* 256 handles 8-bit chars; 128 does 7-bit */
- /* BUG: some overflows (caught) if we use 256 */
+#define NCHARS (256+3) /* 256 handles 8-bit chars; 128 does 7-bit */
/* watch out in match(), etc. */
#define HAT (NCHARS+2) /* matches ^ in regular expr */
#define NSTATES 32
@@ -239,19 +236,12 @@ typedef struct rrow {
int i;
Node *np;
uschar *up;
- int *rp; /* rune representation of char class */
} lval; /* because Al stores a pointer in it! */
int *lfollow;
} rrow;
-typedef struct gtt { /* gototab entry */
- unsigned int ch;
- unsigned int state;
-} gtt;
-
typedef struct fa {
- gtt **gototab;
- int gototab_len;
+ unsigned int **gototab;
uschar *out;
uschar *restr;
int **posns;
Index: pkgsrc/lang/nawk/files/b.c
diff -u pkgsrc/lang/nawk/files/b.c:1.5 pkgsrc/lang/nawk/files/b.c:1.6
--- pkgsrc/lang/nawk/files/b.c:1.5 Tue Sep 12 19:16:52 2023
+++ pkgsrc/lang/nawk/files/b.c Sun Sep 17 10:32:05 2023
@@ -80,41 +80,6 @@ int patlen;
fa *fatab[NFA];
int nfatab = 0; /* entries in fatab */
-
-/* utf-8 mechanism:
-
- For most of Awk, utf-8 strings just "work", since they look like
- null-terminated sequences of 8-bit bytes.
-
- Functions like length(), index(), and substr() have to operate
- in units of utf-8 characters. The u8_* functions in run.c
- handle this.
-
- Regular expressions are more complicated, since the basic
- mechanism of the goto table used 8-bit byte indices into the
- gototab entries to compute the next state. Unicode is a lot
- bigger, so the gototab entries are now structs with a character
- and a next state, and there is a linear search of the characters
- to find the state. (Yes, this is slower, by a significant
- amount. Tough.)
-
- Throughout the RE mechanism in b.c, utf-8 characters are
- converted to their utf-32 value. This mostly shows up in
- cclenter, which expands character class ranges like a-z and now
- alpha-omega. The size of a gototab array is still about 256.
- This should be dynamic, but for now things work ok for a single
- code page of Unicode, which is the most likely case.
-
- The code changes are localized in run.c and b.c. I have added a
- handful of functions to somewhat better hide the implementation,
- but a lot more could be done.
-
- */
-
-static int get_gototab(fa*, int, int);
-static int set_gototab(fa*, int, int, int);
-extern int u8_rune(int *, const uschar *);
-
static int *
intalloc(size_t n, const char *f)
{
@@ -140,7 +105,7 @@ resizesetvec(const char *f)
static void
resize_state(fa *f, int state)
{
- gtt **p;
+ unsigned int **p;
uschar *p2;
int **p3;
int i, new_count;
@@ -150,7 +115,7 @@ resize_state(fa *f, int state)
new_count = state + 10; /* needs to be tuned */
- p = (gtt **) realloc(f->gototab, new_count * sizeof(f->gototab[0]));
+ p = (unsigned int **) realloc(f->gototab, new_count * sizeof(f->gototab[0]));
if (p == NULL)
goto out;
f->gototab = p;
@@ -166,13 +131,12 @@ resize_state(fa *f, int state)
f->posns = p3;
for (i = f->state_count; i < new_count; ++i) {
- f->gototab[i] = (gtt *) calloc(NCHARS, sizeof(**f->gototab));
+ f->gototab[i] = (unsigned int *) calloc(NCHARS, sizeof(**f->gototab));
if (f->gototab[i] == NULL)
goto out;
f->out[i] = 0;
f->posns[i] = NULL;
}
- f->gototab_len = NCHARS; /* should be variable, growable */
f->state_count = new_count;
return;
out:
@@ -267,7 +231,7 @@ int makeinit(fa *f, bool anchor)
if ((f->posns[2])[1] == f->accept)
f->out[2] = 1;
for (i = 0; i < NCHARS; i++)
- set_gototab(f, 2, 0, 0); /* f->gototab[2][i] = 0; */
+ f->gototab[2][i] = 0;
f->curstat = cgoto(f, 2, HAT);
if (anchor) {
*f->posns[2] = k-1; /* leave out position 0 */
@@ -336,13 +300,13 @@ void freetr(Node *p) /* free parse tree
/* in the parsing of regular expressions, metacharacters like . have */
/* to be seen literally; \056 is not a metacharacter. */
-int hexstr(const uschar **pp, int max) /* find and eval hex string at pp, return new p */
+int hexstr(const uschar **pp) /* find and eval hex string at pp, return new p */
{ /* only pick up one 8-bit byte (2 chars) */
const uschar *p;
int n = 0;
int i;
- for (i = 0, p = *pp; i < max && isxdigit(*p); i++, p++) {
+ for (i = 0, p = *pp; i < 2 && isxdigit(*p); i++, p++) {
if (isdigit(*p))
n = 16 * n + *p - '0';
else if (*p >= 'a' && *p <= 'f')
@@ -354,8 +318,6 @@ int hexstr(const uschar **pp, int max) /
return n;
}
-
-
#define isoctdigit(c) ((c) >= '0' && (c) <= '7') /* multiple use of arg */
int quoted(const uschar **pp) /* pick up next thing after a \\ */
@@ -364,28 +326,24 @@ int quoted(const uschar **pp) /* pick up
const uschar *p = *pp;
int c;
-/* BUG: should advance by utf-8 char even if makes no sense */
-
- if ((c = *p++) == 't') {
+ if ((c = *p++) == 't')
c = '\t';
- } else if (c == 'n') {
+ else if (c == 'n')
c = '\n';
- } else if (c == 'f') {
+ else if (c == 'f')
c = '\f';
- } else if (c == 'r') {
+ else if (c == 'r')
c = '\r';
- } else if (c == 'b') {
+ else if (c == 'b')
c = '\b';
- } else if (c == 'v') {
+ else if (c == 'v')
c = '\v';
- } else if (c == 'a') {
+ else if (c == 'a')
c = '\a';
- } else if (c == '\\') {
+ else if (c == '\\')
c = '\\';
- } else if (c == 'x') { /* 2 hex digits follow */
- c = hexstr(&p, 2); /* this adds a null if number is invalid */
- } else if (c == 'u') { /* unicode char number up to 8 hex digits */
- c = hexstr(&p, 8);
+ else if (c == 'x') { /* hexadecimal goo follows */
+ c = hexstr(&p); /* this adds a null if number is invalid */
} else if (isoctdigit(c)) { /* \d \dd \ddd */
int n = c - '0';
if (isoctdigit(*p)) {
@@ -400,67 +358,50 @@ int quoted(const uschar **pp) /* pick up
return c;
}
-int *cclenter(const char *argp) /* add a character class */
+char *cclenter(const char *argp) /* add a character class */
{
int i, c, c2;
- int n;
- const uschar *p = (const uschar *) argp;
- int *bp, *retp;
- static int *buf = NULL;
+ const uschar *op, *p = (const uschar *) argp;
+ uschar *bp;
+ static uschar *buf = NULL;
static int bufsz = 100;
- if (buf == NULL && (buf = (int *) calloc(bufsz, sizeof(int))) == NULL)
+ op = p;
+ if (buf == NULL && (buf = (uschar *) malloc(bufsz)) == NULL)
FATAL("out of space for character class [%.10s...] 1", p);
bp = buf;
- for (i = 0; *p != 0; ) {
- n = u8_rune(&c, p);
- p += n;
+ for (i = 0; (c = *p++) != 0; ) {
if (c == '\\') {
c = quoted(&p);
} else if (c == '-' && i > 0 && bp[-1] != 0) {
if (*p != 0) {
c = bp[-1];
- /* c2 = *p++; */
- n = u8_rune(&c2, p);
- p += n;
+ c2 = *p++;
if (c2 == '\\')
- c2 = quoted(&p); /* BUG: sets p, has to be u8 size */
+ c2 = quoted(&p);
if (c > c2) { /* empty; ignore */
bp--;
i--;
continue;
}
while (c < c2) {
- if (i >= bufsz) {
- bufsz *= 2;
- buf = (int *) realloc(buf, bufsz * sizeof(int));
- if (buf == NULL)
- FATAL("out of space for character class [%.10s...] 2", p);
- bp = buf + i;
- }
+ if (!adjbuf((char **) &buf, &bufsz, bp-buf+2, 100, (char **) &bp, "cclenter1"))
+ FATAL("out of space for character class [%.10s...] 2", p);
*bp++ = ++c;
i++;
}
continue;
}
}
- if (i >= bufsz) {
- bufsz *= 2;
- buf = (int *) realloc(buf, bufsz * sizeof(int));
- if (buf == NULL)
- FATAL("out of space for character class [%.10s...] 2", p);
- bp = buf + i;
- }
+ if (!adjbuf((char **) &buf, &bufsz, bp-buf+2, 100, (char **) &bp, "cclenter2"))
+ FATAL("out of space for character class [%.10s...] 3", p);
*bp++ = c;
i++;
}
*bp = 0;
- /* DPRINTF("cclenter: in = |%s|, out = |%s|\n", op, buf); BUG: can't print array of int */
- /* xfree(op); BUG: what are we freeing here? */
- retp = (int *) calloc(bp-buf+1, sizeof(int));
- for (i = 0; i < bp-buf+1; i++)
- retp[i] = buf[i];
- return retp;
+ DPRINTF("cclenter: in = |%s|, out = |%s|\n", op, buf);
+ xfree(op);
+ return (char *) tostring((char *) buf);
}
void overflo(const char *s)
@@ -583,9 +524,9 @@ void follow(Node *v) /* collects leaves
}
}
-int member(int c, int *sarg) /* is c in s? */
+int member(int c, const char *sarg) /* is c in s? */
{
- int *s = (int *) sarg;
+ const uschar *s = (const uschar *) sarg;
while (*s)
if (c == *s++)
@@ -593,41 +534,11 @@ int member(int c, int *sarg) /* is c in
return(0);
}
-static int get_gototab(fa *f, int state, int ch) /* hide gototab inplementation */
-{
- int i;
- for (i = 0; i < f->gototab_len; i++) {
- if (f->gototab[state][i].ch == 0)
- break;
- if (f->gototab[state][i].ch == ch)
- return f->gototab[state][i].state;
- }
- return 0;
-}
-
-static int set_gototab(fa *f, int state, int ch, int val) /* hide gototab inplementation */
-{
- int i;
- for (i = 0; i < f->gototab_len; i++) {
- if (f->gototab[state][i].ch == 0 || f->gototab[state][i].ch == ch) {
- f->gototab[state][i].ch = ch;
- f->gototab[state][i].state = val;
- return val;
- }
- }
- overflo(__func__);
- return val; /* not used anywhere at the moment */
-}
-
int match(fa *f, const char *p0) /* shortest match ? */
{
int s, ns;
- int n;
- int rune;
const uschar *p = (const uschar *) p0;
- /* return pmatch(f, p0); does it matter whether longest or shortest? */
-
s = f->initstat;
assert (s < f->state_count);
@@ -635,25 +546,19 @@ int match(fa *f, const char *p0) /* shor
return(1);
do {
/* assert(*p < NCHARS); */
- n = u8_rune(&rune, p);
- if ((ns = get_gototab(f, s, rune)) != 0)
+ if ((ns = f->gototab[s][*p]) != 0)
s = ns;
else
- s = cgoto(f, s, rune);
+ s = cgoto(f, s, *p);
if (f->out[s])
return(1);
- if (*p == 0)
- break;
- p += n;
- } while (1); /* was *p++ != 0 */
+ } while (*p++ != 0);
return(0);
}
int pmatch(fa *f, const char *p0) /* longest match, for sub */
{
int s, ns;
- int n;
- int rune;
const uschar *p = (const uschar *) p0;
const uschar *q;
@@ -668,11 +573,10 @@ int pmatch(fa *f, const char *p0) /* lon
if (f->out[s]) /* final state */
patlen = q-p;
/* assert(*q < NCHARS); */
- n = u8_rune(&rune, q);
- if ((ns = get_gototab(f, s, rune)) != 0)
+ if ((ns = f->gototab[s][*q]) != 0)
s = ns;
else
- s = cgoto(f, s, rune);
+ s = cgoto(f, s, *q);
assert(s < f->state_count);
@@ -684,11 +588,7 @@ int pmatch(fa *f, const char *p0) /* lon
else
goto nextin; /* no match */
}
- if (*q == 0)
- break;
- q += n;
- } while (1);
- q++; /* was *q++ */
+ } while (*q++ != 0);
if (f->out[s])
patlen = q-p-1; /* don't count $ */
if (patlen >= 0) {
@@ -697,19 +597,13 @@ int pmatch(fa *f, const char *p0) /* lon
}
nextin:
s = 2;
- if (*p == 0)
- break;
- n = u8_rune(&rune, p);
- p += n;
- } while (1); /* was *p++ */
+ } while (*p++);
return (0);
}
int nematch(fa *f, const char *p0) /* non-empty match, for sub */
{
int s, ns;
- int n;
- int rune;
const uschar *p = (const uschar *) p0;
const uschar *q;
@@ -724,11 +618,10 @@ int nematch(fa *f, const char *p0) /* no
if (f->out[s]) /* final state */
patlen = q-p;
/* assert(*q < NCHARS); */
- n = u8_rune(&rune, q);
- if ((ns = get_gototab(f, s, rune)) != 0)
+ if ((ns = f->gototab[s][*q]) != 0)
s = ns;
else
- s = cgoto(f, s, rune);
+ s = cgoto(f, s, *q);
if (s == 1) { /* no transition */
if (patlen > 0) {
patbeg = (const char *) p;
@@ -736,11 +629,7 @@ int nematch(fa *f, const char *p0) /* no
} else
goto nnextin; /* no nonempty match */
}
- if (*q == 0)
- break;
- q += n;
- } while (1);
- q++;
+ } while (*q++ != 0);
if (f->out[s])
patlen = q-p-1; /* don't count $ */
if (patlen > 0 ) {
@@ -754,35 +643,6 @@ int nematch(fa *f, const char *p0) /* no
return (0);
}
-static int getrune(FILE *fp, char **pbuf, int *pbufsize, int quantum,
- int *curpos, int *lastpos)
-{
- int c = 0;
- char *buf = *pbuf;
- static const int max_bytes = 4; // max multiple bytes in UTF-8 is 4
- int i, rune;
- uschar private_buf[max_bytes + 1];
-
- for (i = 0; i <= max_bytes; i++) {
- if (++*curpos == *lastpos) {
- if (*lastpos == *pbufsize)
- if (!adjbuf((char **) pbuf, pbufsize, *pbufsize+1, quantum, 0, "getrune"))
- FATAL("stream '%.30s...' too long", buf);
- buf[(*lastpos)++] = (c = getc(fp)) != EOF ? c : 0;
- private_buf[i] = c;
- }
- if (c == 0 || c < 128 || (c >> 6) == 4) { // 10xxxxxx starts a new character
- ungetc(c, fp);
- private_buf[i] = 0;
- break;
- }
- }
-
- u8_rune(& rune, private_buf);
-
- return rune;
-}
-
/*
* NAME
@@ -804,7 +664,6 @@ bool fnematch(fa *pfa, FILE *f, char **p
char *buf = *pbuf;
int bufsize = *pbufsize;
int c, i, j, k, ns, s;
- int rune;
s = pfa->initstat;
patlen = 0;
@@ -828,19 +687,12 @@ bool fnematch(fa *pfa, FILE *f, char **p
buf[k++] = (c = getc(f)) != EOF ? c : 0;
}
c = (uschar)buf[j];
- if (c < 128)
- rune = c;
- else {
- j--;
- k--;
- ungetc(c, f);
- rune = getrune(f, &buf, &bufsize, quantum, &j, &k);
- }
+ /* assert(c < NCHARS); */
- if ((ns = get_gototab(pfa, s, rune)) != 0)
+ if ((ns = pfa->gototab[s][c]) != 0)
s = ns;
else
- s = cgoto(pfa, s, rune);
+ s = cgoto(pfa, s, c);
if (pfa->out[s]) { /* final state */
patlen = j - i + 1;
@@ -1167,8 +1019,6 @@ static int repeat(const uschar *reptok,
return 0;
}
-extern int u8_rune(int *, const uschar *); /* run.c; should be in header file */
-
int relex(void) /* lexical analyzer for reparse */
{
int c, n;
@@ -1186,12 +1036,6 @@ int relex(void) /* lexical analyzer for
rescan:
starttok = prestr;
- if ((n = u8_rune(&rlxval, prestr)) > 1) {
- prestr += n;
- starttok = prestr;
- return CHAR;
- }
-
switch (c = *prestr++) {
case '|': return OR;
case '*': return STAR;
@@ -1229,15 +1073,10 @@ rescan:
}
else
cflag = 0;
- n = 5 * strlen((const char *) prestr)+1; /* BUG: was 2. what value? */
+ n = 2 * strlen((const char *) prestr)+1;
if (!adjbuf((char **) &buf, &bufsz, n, n, (char **) &bp, "relex1"))
FATAL("out of space for reg expr %.10s...", lastre);
for (; ; ) {
- if ((n = u8_rune(&rlxval, prestr)) > 1) {
- for (i = 0; i < n; i++)
- *bp++ = *prestr++;
- continue;
- }
if ((c = *prestr++) == '\\') {
*bp++ = '\\';
if ((c = *prestr++) == '\0')
@@ -1404,7 +1243,7 @@ int cgoto(fa *f, int s, int c)
int *p, *q;
int i, j, k;
- /* assert(c == HAT || c < NCHARS); BUG: seg fault if disable test */
+ assert(c == HAT || c < NCHARS);
while (f->accept >= maxsetvec) { /* guessing here! */
resizesetvec(__func__);
}
@@ -1420,8 +1259,8 @@ int cgoto(fa *f, int s, int c)
|| (k == DOT && c != 0 && c != HAT)
|| (k == ALL && c != 0)
|| (k == EMPTYRE && c != 0)
- || (k == CCL && member(c, (int *) f->re[p[i]].lval.rp))
- || (k == NCCL && !member(c, (int *) f->re[p[i]].lval.rp) && c != 0 && c != HAT)) {
+ || (k == CCL && member(c, (char *) f->re[p[i]].lval.up))
+ || (k == NCCL && !member(c, (char *) f->re[p[i]].lval.up) && c != 0 && c != HAT)) {
q = f->re[p[i]].lfollow;
for (j = 1; j <= *q; j++) {
if (q[j] >= maxsetvec) {
@@ -1453,7 +1292,7 @@ int cgoto(fa *f, int s, int c)
goto different;
/* setvec is state i */
if (c != HAT)
- set_gototab(f, s, c, i);
+ f->gototab[s][c] = i;
return i;
different:;
}
@@ -1462,13 +1301,13 @@ int cgoto(fa *f, int s, int c)
++(f->curstat);
resize_state(f, f->curstat);
for (i = 0; i < NCHARS; i++)
- set_gototab(f, f->curstat, 0, 0);
+ f->gototab[f->curstat][i] = 0;
xfree(f->posns[f->curstat]);
p = intalloc(setcnt + 1, __func__);
f->posns[f->curstat] = p;
if (c != HAT)
- set_gototab(f, s, c, f->curstat);
+ f->gototab[s][c] = f->curstat;
for (i = 0; i <= setcnt; i++)
p[i] = tmpset[i];
if (setvec[f->accept])
Index: pkgsrc/lang/nawk/files/run.c
diff -u pkgsrc/lang/nawk/files/run.c:1.5 pkgsrc/lang/nawk/files/run.c:1.6
--- pkgsrc/lang/nawk/files/run.c:1.5 Tue Sep 12 19:16:52 2023
+++ pkgsrc/lang/nawk/files/run.c Sun Sep 17 10:32:06 2023
@@ -26,6 +26,7 @@ THIS SOFTWARE.
#include <stdio.h>
#include <ctype.h>
#include <errno.h>
+#include <wchar.h>
#include <wctype.h>
#include <fcntl.h>
#include <setjmp.h>
@@ -39,10 +40,8 @@ THIS SOFTWARE.
#include "awk.h"
#include "awkgram.tab.h"
-
static void stdinit(void);
static void flush_all(void);
-static char *wide_char_to_byte_str(int rune, size_t *outlen);
#if 1
#define tempfree(x) do { if (istemp(x)) tfree(x); } while (/*CONSTCOND*/0)
@@ -580,225 +579,11 @@ Cell *intest(Node **a, int n) /* a[0] is
}
-/* ======== utf-8 code ========== */
-
-/*
- * Awk strings can contain ascii, random 8-bit items (eg Latin-1),
- * or utf-8. u8_isutf tests whether a string starts with a valid
- * utf-8 sequence, and returns 0 if not (e.g., high bit set).
- * u8_nextlen returns length of next valid sequence, which is
- * 1 for ascii, 2..4 for utf-8, or 1 for high bit non-utf.
- * u8_strlen returns length of string in valid utf-8 sequences
- * and/or high-bit bytes. Conversion functions go between byte
- * number and character number.
- *
- * In theory, this behaves the same as before for non-utf8 bytes.
- *
- * Limited checking! This is a potential security hole.
- */
-
-/* is s the beginning of a valid utf-8 string? */
-/* return length 1..4 if yes, 0 if no */
-int u8_isutf(const char *s)
-{
- int n, ret;
- unsigned char c;
-
- c = s[0];
- if (c < 128)
- return 1; /* what if it's 0? */
-
- n = strlen(s);
- if (n >= 2 && ((c>>5) & 0x7) == 0x6 && (s[1] & 0xC0) == 0x80) {
- ret = 2; /* 110xxxxx 10xxxxxx */
- } else if (n >= 3 && ((c>>4) & 0xF) == 0xE && (s[1] & 0xC0) == 0x80
- && (s[2] & 0xC0) == 0x80) {
- ret = 3; /* 1110xxxx 10xxxxxx 10xxxxxx */
- } else if (n >= 4 && ((c>>3) & 0x1F) == 0x1E && (s[1] & 0xC0) == 0x80
- && (s[2] & 0xC0) == 0x80 && (s[3] & 0xC0) == 0x80) {
- ret = 4; /* 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx */
- } else {
- ret = 0;
- }
- return ret;
-}
-
-/* Convert (prefix of) utf8 string to utf-32 rune. */
-/* Sets *rune to the value, returns the length. */
-/* No error checking: watch out. */
-int u8_rune(int *rune, const char *s)
-{
- int n, ret;
- unsigned char c;
-
- c = s[0];
- if (c < 128) {
- *rune = c;
- return 1;
- }
-
- n = strlen(s);
- if (n >= 2 && ((c>>5) & 0x7) == 0x6 && (s[1] & 0xC0) == 0x80) {
- *rune = ((c & 0x1F) << 6) | (s[1] & 0x3F); /* 110xxxxx 10xxxxxx */
- ret = 2;
- } else if (n >= 3 && ((c>>4) & 0xF) == 0xE && (s[1] & 0xC0) == 0x80
- && (s[2] & 0xC0) == 0x80) {
- *rune = ((c & 0xF) << 12) | ((s[1] & 0x3F) << 6) | (s[2] & 0x3F);
- /* 1110xxxx 10xxxxxx 10xxxxxx */
- ret = 3;
- } else if (n >= 4 && ((c>>3) & 0x1F) == 0x1E && (s[1] & 0xC0) == 0x80
- && (s[2] & 0xC0) == 0x80 && (s[3] & 0xC0) == 0x80) {
- *rune = ((c & 0x7) << 18) | ((s[1] & 0x3F) << 12) | ((s[2] & 0x3F) << 6) | (s[3] & 0x3F);
- /* 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx */
- ret = 4;
- } else {
- *rune = c;
- ret = 1;
- }
- return ret; /* returns one byte if sequence doesn't look like utf */
-}
-
-/* return length of next sequence: 1 for ascii or random, 2..4 for valid utf8 */
-int u8_nextlen(const char *s)
-{
- int len;
-
- len = u8_isutf(s);
- if (len == 0)
- len = 1;
- return len;
-}
-
-/* return number of utf characters or single non-utf bytes */
-int u8_strlen(const char *s)
-{
- int i, len, n, totlen;
- unsigned char c;
-
- n = strlen(s);
- totlen = 0;
- for (i = 0; i < n; i += len) {
- c = s[i];
- if (c < 128) {
- len = 1;
- } else {
- len = u8_nextlen(&s[i]);
- }
- totlen++;
- if (i > n)
- FATAL("bad utf count [%s] n=%d i=%d\n", s, n, i);
- }
- return totlen;
-}
-
-/* convert utf-8 char number in a string to its byte offset */
-int u8_char2byte(const char *s, int charnum)
-{
- int n;
- int bytenum = 0;
-
- while (charnum > 0) {
- n = u8_nextlen(s);
- s += n;
- bytenum += n;
- charnum--;
- }
- return bytenum;
-}
-
-/* convert byte offset in s to utf-8 char number that starts there */
-int u8_byte2char(const char *s, int bytenum)
-{
- int i, len, b;
- int charnum = 0; /* BUG: what origin? */
- /* should be 0 to match start==0 which means no match */
-
- b = strlen(s);
- if (bytenum >= b) {
- return -1; /* ??? */
- }
- for (i = 0; i <= bytenum; i += len) {
- len = u8_nextlen(s+i);
- charnum++;
- }
- return charnum;
-}
-
-/* runetochar() adapted from rune.c in the Plan 9 distributione */
-
-enum
-{
- Runeerror = 128, /* from somewhere else */
- Runemax = 0x10FFFF,
-
- Bit1 = 7,
- Bitx = 6,
- Bit2 = 5,
- Bit3 = 4,
- Bit4 = 3,
- Bit5 = 2,
-
- T1 = ((1<<(Bit1+1))-1) ^ 0xFF, /* 0000 0000 */
- Tx = ((1<<(Bitx+1))-1) ^ 0xFF, /* 1000 0000 */
- T2 = ((1<<(Bit2+1))-1) ^ 0xFF, /* 1100 0000 */
- T3 = ((1<<(Bit3+1))-1) ^ 0xFF, /* 1110 0000 */
- T4 = ((1<<(Bit4+1))-1) ^ 0xFF, /* 1111 0000 */
- T5 = ((1<<(Bit5+1))-1) ^ 0xFF, /* 1111 1000 */
-
- Rune1 = (1<<(Bit1+0*Bitx))-1, /* 0000 0000 0000 0000 0111 1111 */
- Rune2 = (1<<(Bit2+1*Bitx))-1, /* 0000 0000 0000 0111 1111 1111 */
- Rune3 = (1<<(Bit3+2*Bitx))-1, /* 0000 0000 1111 1111 1111 1111 */
- Rune4 = (1<<(Bit4+3*Bitx))-1, /* 0011 1111 1111 1111 1111 1111 */
-
- Maskx = (1<<Bitx)-1, /* 0011 1111 */
- Testx = Maskx ^ 0xFF, /* 1100 0000 */
-
-};
-
-int runetochar(char *str, int c)
-{
- /* one character sequence 00000-0007F => 00-7F */
- if (c <= Rune1) {
- str[0] = c;
- return 1;
- }
-
- /* two character sequence 00080-007FF => T2 Tx */
- if (c <= Rune2) {
- str[0] = T2 | (c >> 1*Bitx);
- str[1] = Tx | (c & Maskx);
- return 2;
- }
-
- /* three character sequence 00800-0FFFF => T3 Tx Tx */
- if (c > Runemax)
- c = Runeerror;
- if (c <= Rune3) {
- str[0] = T3 | (c >> 2*Bitx);
- str[1] = Tx | ((c >> 1*Bitx) & Maskx);
- str[2] = Tx | (c & Maskx);
- return 3;
- }
-
- /* four character sequence 010000-1FFFFF => T4 Tx Tx Tx */
- str[0] = T4 | (c >> 3*Bitx);
- str[1] = Tx | ((c >> 2*Bitx) & Maskx);
- str[2] = Tx | ((c >> 1*Bitx) & Maskx);
- str[3] = Tx | (c & Maskx);
- return 4;
-}
-
-
-/* ========== end of utf8 code =========== */
-
-
-
Cell *matchop(Node **a, int n) /* ~ and match() */
{
Cell *x, *y;
char *s, *t;
int i;
- int cstart, cpatlen, len;
fa *pfa;
int (*mf)(fa *, const char *) = match, mode = 0;
@@ -819,21 +604,9 @@ Cell *matchop(Node **a, int n) /* ~ and
}
tempfree(x);
if (n == MATCHFCN) {
- int start = patbeg - s + 1; /* origin 1 */
- if (patlen < 0) {
- start = 0; /* not found */
- } else {
- cstart = u8_byte2char(s, start-1);
- cpatlen = 0;
- for (i = 0; i < patlen; i += len) {
- len = u8_nextlen(patbeg+i);
- cpatlen++;
- }
-
- start = cstart;
- patlen = cpatlen;
- }
-
+ int start = patbeg - s + 1;
+ if (patlen < 0)
+ start = 0;
setfval(rstartloc, (Awkfloat) start);
setfval(rlengthloc, (Awkfloat) patlen);
x = gettemp();
@@ -884,15 +657,10 @@ Cell *relop(Node **a, int n) /* a[0 < a[
int i;
Cell *x, *y;
Awkfloat j;
- bool x_is_nan, y_is_nan;
x = execute(a[0]);
y = execute(a[1]);
- x_is_nan = isnan(x->fval);
- y_is_nan = isnan(y->fval);
if (x->tval&NUM && y->tval&NUM) {
- if ((x_is_nan || y_is_nan) && n != NE)
- return(False);
j = x->fval - y->fval;
i = j<0? -1: (j>0? 1: 0);
} else {
@@ -905,8 +673,7 @@ Cell *relop(Node **a, int n) /* a[0 < a[
else return(False);
case LE: if (i<=0) return(True);
else return(False);
- case NE: if (x_is_nan && y_is_nan) return(True);
- else if (i!=0) return(True);
+ case NE: if (i!=0) return(True);
else return(False);
case EQ: if (i == 0) return(True);
else return(False);
@@ -975,7 +742,6 @@ Cell *indirect(Node **a, int n) /* $( a[
Cell *substr(Node **a, int nnn) /* substr(a[0], a[1], a[2]) */
{
int k, m, n;
- int mb, nb;
char *s;
int temp;
Cell *x, *y, *z = NULL;
@@ -1011,16 +777,12 @@ Cell *substr(Node **a, int nnn) /* subs
n = 0;
else if (n > k - m)
n = k - m;
- /* m is start, n is length from there */
DPRINTF("substr: m=%d, n=%d, s=%s\n", m, n, s);
y = gettemp();
- mb = u8_char2byte(s, m-1); /* byte offset of start char in s */
- nb = u8_char2byte(s, m-1+n); /* byte offset of end+1 char in s */
-
- temp = s[nb]; /* with thanks to John Linderman */
- s[nb] = '\0';
- setsval(y, s + mb);
- s[nb] = temp;
+ temp = s[n+m-1]; /* with thanks to John Linderman */
+ s[n+m-1] = '\0';
+ setsval(y, s + m - 1);
+ s[n+m-1] = temp;
tempfree(x);
return(y);
}
@@ -1041,15 +803,7 @@ Cell *sindex(Node **a, int nnn) /* inde
for (q = p1, p2 = s2; *p2 != '\0' && *q == *p2; q++, p2++)
continue;
if (*p2 == '\0') {
- /* v = (Awkfloat) (p1 - s1 + 1); origin 1 */
-
- /* should be a function: used in match() as well */
- int i, len;
- v = 0;
- for (i = 0; i < p1-s1+1; i += len) {
- len = u8_nextlen(s1+i);
- v++;
- }
+ v = (Awkfloat) (p1 - s1 + 1); /* origin 1 */
break;
}
}
@@ -1059,18 +813,6 @@ Cell *sindex(Node **a, int nnn) /* inde
return(z);
}
-int has_utf8(char *s) /* return 1 if s contains any utf-8 (2 bytes or more) character */
-{
- int n;
-
- for (n = 0; *s != 0; s += n) {
- n = u8_nextlen(s);
- if (n > 1)
- return 1;
- }
- return 0;
-}
-
#define MAXNUMSIZE 50
int format(char **pbuf, int *pbufsize, const char *s, Node *a) /* printf-like conversions */
@@ -1113,6 +855,7 @@ int format(char **pbuf, int *pbufsize, c
s += 2;
continue;
}
+ /* have to be real careful in case this is a huge number, eg, %100000d */
fmtwd = atoi(s+1);
if (fmtwd < 0)
fmtwd = -fmtwd;
@@ -1185,8 +928,7 @@ int format(char **pbuf, int *pbufsize, c
n = fmtwd;
adjbuf(&buf, &bufsize, 1+n+p-buf, recsize, &p, "format5");
switch (flag) {
- case '?':
- snprintf(p, BUFSZ(p), "%s", fmt); /* unknown, so dump it too */
+ case '?': snprintf(p, BUFSZ(p), "%s", fmt); /* unknown, so dump it too */
t = getsval(x);
n = strlen(t);
if (fmtwd > n)
@@ -1200,176 +942,29 @@ int format(char **pbuf, int *pbufsize, c
case 'f': snprintf(p, BUFSZ(p), fmt, getfval(x)); break;
case 'd': snprintf(p, BUFSZ(p), fmt, (intmax_t) getfval(x)); break;
case 'u': snprintf(p, BUFSZ(p), fmt, (uintmax_t) getfval(x)); break;
-
- case 's': {
+ case 's':
t = getsval(x);
n = strlen(t);
- /* if simple format or no utf-8 in the string, sprintf works */
- if (!has_utf8(t) || strcmp(fmt,"%s") == 0) {
- if (fmtwd > n)
- n = fmtwd;
- if (!adjbuf(&buf, &bufsize, 1+n+p-buf, recsize, &p, "format7"))
- FATAL("huge string/format (%d chars) in printf %.30s..." \
- " ran format() out of memory", n, t);
- snprintf(p, BUFSZ(p), fmt, t);
- break;
- }
-
- /* get here if string has utf-8 chars and fmt is not plain %s */
- /* "%-w.ps", where -, w and .p are all optional */
- /* '0' before the w is a flag character */
- /* fmt points at % */
- int ljust = 0, wid = 0, prec = n, pad = 0;
- char *f = fmt+1;
- if (f[0] == '-') {
- ljust = 1;
- f++;
- }
- // flags '0' and '+' are recognized but skipped
- if (f[0] == '0') {
- f++;
- if (f[0] == '+')
- f++;
- }
- if (f[0] == '+') {
- f++;
- if (f[0] == '0')
- f++;
- }
- if (isdigit(f[0])) { /* there is a wid */
- wid = strtol(f, &f, 10);
- }
- if (f[0] == '.') { /* there is a .prec */
- prec = strtol(++f, &f, 10);
- }
- if (prec > u8_strlen(t))
- prec = u8_strlen(t);
- pad = wid>prec ? wid - prec : 0; // has to be >= 0
- int i, k, n;
-
- if (ljust) { // print prec chars from t, then pad blanks
- n = u8_char2byte(t, prec);
- for (k = 0; k < n; k++) {
- //putchar(t[k]);
- *p++ = t[k];
- }
- for (i = 0; i < pad; i++) {
- //printf(" ");
- *p++ = ' ';
- }
- } else { // print pad blanks, then prec chars from t
- for (i = 0; i < pad; i++) {
- //printf(" ");
- *p++ = ' ';
- }
- n = u8_char2byte(t, prec);
- for (k = 0; k < n; k++) {
- //putchar(t[k]);
- *p++ = t[k];
- }
- }
- *p = 0;
+ if (fmtwd > n)
+ n = fmtwd;
+ if (!adjbuf(&buf, &bufsize, 1+n+p-buf, recsize, &p, "format7"))
+ FATAL("huge string/format (%d chars) in printf %.30s... ran format() out of memory", n, t);
+ snprintf(p, BUFSZ(p), fmt, t);
break;
- }
-
- case 'c': {
- /*
- * If a numeric value is given, awk should just turn
- * it into a character and print it:
- * BEGIN { printf("%c\n", 65) }
- * prints "A".
- *
- * But what if the numeric value is > 128 and
- * represents a valid Unicode code point?!? We do
- * our best to convert it back into UTF-8. If we
- * can't, we output the encoding of the Unicode
- * "invalid character", 0xFFFD.
- */
+ case 'c':
if (isnum(x)) {
- int charval = (int) getfval(x);
-
- if (charval != 0) {
- if (charval < 128)
- snprintf(p, BUFSZ(p), fmt, charval);
- else {
- // possible unicode character
- size_t count;
- char *bs = wide_char_to_byte_str(charval, &count);
-
- if (bs == NULL) { // invalid character
- // use unicode invalid character, 0xFFFD
- bs = "\357\277\275";
- count = 3;
- }
- t = bs;
- n = count;
- goto format_percent_c;
- }
- } else {
+ if ((int)getfval(x))
+ snprintf(p, BUFSZ(p), fmt, (int) getfval(x));
+ else {
*p++ = '\0'; /* explicit null byte */
*p = '\0'; /* next output will start here */
}
- break;
- }
- t = getsval(x);
- n = u8_nextlen(t);
- format_percent_c:
- if (n < 2) { /* not utf8 */
+ } else
snprintf(p, BUFSZ(p), fmt, getsval(x)[0]);
- break;
- }
-
- // utf8 character, almost same song and dance as for %s
- int ljust = 0, wid = 0, prec = n, pad = 0;
- char *f = fmt+1;
- if (f[0] == '-') {
- ljust = 1;
- f++;
- }
- // flags '0' and '+' are recognized but skipped
- if (f[0] == '0') {
- f++;
- if (f[0] == '+')
- f++;
- }
- if (f[0] == '+') {
- f++;
- if (f[0] == '0')
- f++;
- }
- if (isdigit(f[0])) { /* there is a wid */
- wid = strtol(f, &f, 10);
- }
- if (f[0] == '.') { /* there is a .prec */
- prec = strtol(++f, &f, 10);
- }
- if (prec > 1) // %c --> only one character
- prec = 1;
- pad = wid>prec ? wid - prec : 0; // has to be >= 0
- int i;
-
- if (ljust) { // print one char from t, then pad blanks
- for (int i = 0; i < n; i++)
- *p++ = t[i];
- for (i = 0; i < pad; i++) {
- //printf(" ");
- *p++ = ' ';
- }
- } else { // print pad blanks, then prec chars from t
- for (i = 0; i < pad; i++) {
- //printf(" ");
- *p++ = ' ';
- }
- for (int i = 0; i < n; i++)
- *p++ = t[i];
- }
- *p = 0;
break;
- }
default:
FATAL("can't happen: bad conversion %c in format()", flag);
}
-
tempfree(x);
p += strlen(p);
s++;
@@ -1670,27 +1265,24 @@ Cell *split(Node **a, int nnn) /* split(
int sep;
char temp, num[50];
int n, tempstat, arg3type;
- int j;
double result;
y = execute(a[0]); /* source string */
origs = s = strdup(getsval(y));
tempfree(y);
arg3type = ptoi(a[3]);
- if (a[2] == NULL) { /* BUG: CSV should override implicit fs but not explicit */
+ if (a[2] == NULL) /* fs string */
fs = getsval(fsloc);
- } else if (arg3type == STRING) { /* split(str,arr,"string") */
+ else if (arg3type == STRING) { /* split(str,arr,"string") */
x = execute(a[2]);
fs = origfs = strdup(getsval(x));
tempfree(x);
- } else if (arg3type == REGEXPR) {
+ } else if (arg3type == REGEXPR)
fs = "(regexpr)"; /* split(str,arr,/regexpr/) */
- } else {
+ else
FATAL("illegal type of split");
- }
sep = *fs;
ap = execute(a[1]); /* array name */
-/* BUG 7/26/22: this appears not to reset array: see C1/asplit */
freesymtab(ap);
DPRINTF("split: s=|%s|, a=%s, sep=|%s|\n", s, NN(ap->nval), fs);
ap->tval &= ~STR;
@@ -1744,41 +1336,7 @@ Cell *split(Node **a, int nnn) /* split(
setsymtab(num, s, 0.0, STR, (Array *) ap->sval);
spdone:
pfa = NULL;
-
- } else if (a[2] == NULL && CSV) { /* CSV only if no explicit separator */
- char *newt = (char *) malloc(strlen(s)); /* for building new string; reuse for each field */
- for (;;) {
- char *fr = newt;
- n++;
- if (*s == '"' ) { /* start of "..." */
- for (s++ ; *s != '\0'; ) {
- if (*s == '"' && s[1] != '\0' && s[1] == '"') {
- s += 2; /* doubled quote */
- *fr++ = '"';
- } else if (*s == '"' && (s[1] == '\0' || s[1] == ',')) {
- s++; /* skip over closing quote */
- break;
- } else {
- *fr++ = *s++;
- }
- }
- *fr++ = 0;
- } else { /* unquoted field */
- while (*s != ',' && *s != '\0')
- *fr++ = *s++;
- *fr++ = 0;
- }
- snprintf(num, sizeof(num), "%d", n);
- if (is_number(newt, &result))
- setsymtab(num, newt, result, STR|NUM, (Array *) ap->sval);
- else
- setsymtab(num, newt, 0.0, STR, (Array *) ap->sval);
- if (*s++ == '\0')
- break;
- }
- free(newt);
-
- } else if (!CSV && sep == ' ') { /* usual case: split on white space */
+ } else if (sep == ' ') {
for (n = 0; ; ) {
#define ISWS(c) ((c) == ' ' || (c) == '\t' || (c) == '\n')
while (ISWS(*s))
@@ -1801,25 +1359,19 @@ Cell *split(Node **a, int nnn) /* split(
if (*s != '\0')
s++;
}
-
} else if (sep == 0) { /* new: split(s, a, "") => 1 char/elem */
- for (n = 0; *s != '\0'; s += u8_nextlen(s)) {
- char buf[10];
+ for (n = 0; *s != '\0'; s++) {
+ char buf[2];
n++;
snprintf(num, sizeof(num), "%d", n);
-
- for (j = 0; j < u8_nextlen(s); j++) {
- buf[j] = s[j];
- }
- buf[j] = '\0';
-
+ buf[0] = *s;
+ buf[1] = '\0';
if (isdigit((uschar)buf[0]))
setsymtab(num, buf, atof(buf), STR|NUM, (Array *) ap->sval);
else
setsymtab(num, buf, 0.0, STR, (Array *) ap->sval);
}
-
- } else if (*s != '\0') { /* some random single character */
+ } else if (*s != '\0') {
for (;;) {
n++;
t = s;
@@ -1978,7 +1530,6 @@ static char *nawk_convert(const char *s,
size_t n = 0;
wchar_t wc;
size_t sz = MB_CUR_MAX;
- int unused;
if (sz == 1) {
buf = tostring(s);
@@ -1998,7 +1549,7 @@ static char *nawk_convert(const char *s,
* doesn't work.)
* Increment said variable to avoid a different warning.
*/
- unused = wctomb(NULL, L'\0');
+ int unused = wctomb(NULL, L'\0');
unused++;
ps = s;
@@ -2052,8 +1603,6 @@ static char *nawk_tolower(const char *s)
return nawk_convert(s, tolower, towlower);
}
-
-
Cell *bltin(Node **a, int n) /* builtin functions. a[0] is type, a[1] is arg list */
{
Cell *x, *y;
@@ -2073,7 +1622,7 @@ Cell *bltin(Node **a, int n) /* builtin
if (isarr(x))
u = ((Array *) x->sval)->nelem; /* GROT. should be function*/
else
- u = u8_strlen(getsval(x));
+ u = strlen(getsval(x));
break;
case FLOG:
errno = 0;
@@ -2595,41 +2144,3 @@ void backsub(char **pb_ptr, const char *
*pb_ptr = pb;
*sptr_ptr = sptr;
}
-
-static char *wide_char_to_byte_str(int rune, size_t *outlen)
-{
- static char buf[5];
- int len;
-
- if (rune < 0 || rune > 0x10FFFF)
- return NULL;
-
- memset(buf, 0, sizeof(buf));
-
- len = 0;
- if (rune <= 0x0000007F) {
- buf[len++] = rune;
- } else if (rune <= 0x000007FF) {
- // 110xxxxx 10xxxxxx
- buf[len++] = 0xC0 | (rune >> 6);
- buf[len++] = 0x80 | (rune & 0x3F);
- } else if (rune <= 0x0000FFFF) {
- // 1110xxxx 10xxxxxx 10xxxxxx
- buf[len++] = 0xE0 | (rune >> 12);
- buf[len++] = 0x80 | ((rune >> 6) & 0x3F);
- buf[len++] = 0x80 | (rune & 0x3F);
-
- } else {
- // 0x00010000 - 0x10FFFF
- // 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx
- buf[len++] = 0xF0 | (rune >> 18);
- buf[len++] = 0x80 | ((rune >> 12) & 0x3F);
- buf[len++] = 0x80 | ((rune >> 6) & 0x3F);
- buf[len++] = 0x80 | (rune & 0x3F);
- }
-
- *outlen = len;
- buf[len++] = '\0';
-
- return buf;
-}
Index: pkgsrc/lang/nawk/files/lib.c
diff -u pkgsrc/lang/nawk/files/lib.c:1.6 pkgsrc/lang/nawk/files/lib.c:1.7
--- pkgsrc/lang/nawk/files/lib.c:1.6 Tue Sep 12 19:16:52 2023
+++ pkgsrc/lang/nawk/files/lib.c Sun Sep 17 10:32:06 2023
@@ -34,8 +34,6 @@ THIS SOFTWARE.
#include <math.h>
#include "awk.h"
-extern int u8_nextlen(const char *s);
-
char EMPTY[] = { '\0' };
FILE *infile = NULL;
bool innew; /* true = infile has not been read by readrec */
@@ -221,19 +219,14 @@ void nextfile(void)
argno++;
}
-extern int readcsvrec(char **pbuf, int *pbufsize, FILE *inf, bool newflag);
-
int readrec(char **pbuf, int *pbufsize, FILE *inf, bool newflag) /* read one record into buf */
{
- int sep, c, isrec; // POTENTIAL BUG? isrec is a macro in awk.h
- char *rr = *pbuf, *buf = *pbuf;
+ int sep, c, isrec;
+ char *rr, *buf = *pbuf;
int bufsize = *pbufsize;
char *rs = getsval(rsloc);
- if (CSV) {
- c = readcsvrec(pbuf, pbufsize, inf, newflag);
- isrec = (c == EOF && rr == buf) ? false : true;
- } else if (*rs && rs[1]) {
+ if (*rs && rs[1]) {
bool found;
fa *pfa = makedfa(rs, 1);
@@ -248,7 +241,6 @@ int readrec(char **pbuf, int *pbufsize,
if (found)
setptr(patbeg, '\0');
isrec = (found == 0 && *buf == '\0') ? false : true;
-
} else {
if ((sep = *rs) == 0) {
sep = '\n';
@@ -286,52 +278,6 @@ int readrec(char **pbuf, int *pbufsize,
return isrec;
}
-
-/*******************
- * loose ends here:
- * \r\n should become \n
- * what about bare \r? Excel uses that for embedded newlines
- * can't have "" in unquoted fields, according to RFC 4180
-*/
-
-
-int readcsvrec(char **pbuf, int *pbufsize, FILE *inf, bool newflag) /* csv can have \n's */
-{ /* so read a complete record that might be multiple lines */
- int sep, c;
- char *rr = *pbuf, *buf = *pbuf;
- int bufsize = *pbufsize;
- bool in_quote = false;
-
- sep = '\n'; /* the only separator; have to skip over \n embedded in "..." */
- rr = buf;
- while ((c = getc(inf)) != EOF) {
- if (c == sep) {
- if (! in_quote)
- break;
- if (rr > buf && rr[-1] == '\r') // remove \r if was \r\n
- rr--;
- }
-
- if (rr-buf+1 > bufsize)
- if (!adjbuf(&buf, &bufsize, 1+rr-buf,
- recsize, &rr, "readcsvrec 1"))
- FATAL("input record `%.30s...' too long", buf);
- *rr++ = c;
- if (c == '"')
- in_quote = ! in_quote;
- }
- if (c == '\n' && rr > buf && rr[-1] == '\r') // remove \r if was \r\n
- rr--;
-
- if (!adjbuf(&buf, &bufsize, 1+rr-buf, recsize, &rr, "readcsvrec 4"))
- FATAL("input record `%.30s...' too long", buf);
- *rr = 0;
- *pbuf = buf;
- *pbufsize = bufsize;
- DPRINTF("readcsvrec saw <%s>, returns %d\n", buf, c);
- return c;
-}
-
char *getargv(int n) /* get ARGV[n] */
{
Cell *x;
@@ -353,9 +299,6 @@ void setclvar(char *s) /* set var=value
Cell *q;
double result;
-/* commit f3d9187d4e0f02294fb1b0e31152070506314e67 broke T.argv test */
-/* I don't understand why it was changed. */
-
for (p=s; *p != '='; p++)
;
e = p;
@@ -400,7 +343,7 @@ void fldbld(void) /* create fields from
savefs();
if (strlen(inputFS) > 1) { /* it's a regular expression */
i = refldbld(r, inputFS);
- } else if (!CSV && (sep = *inputFS) == ' ') { /* default whitespace */
+ } else if ((sep = *inputFS) == ' ') { /* default whitespace */
for (i = 0; ; ) {
while (*r == ' ' || *r == '\t' || *r == '\n')
r++;
@@ -419,58 +362,26 @@ void fldbld(void) /* create fields from
*fr++ = 0;
}
*fr = 0;
- } else if (CSV) { /* CSV processing. no error handling */
- if (*r != 0) {
- for (;;) {
- i++;
- if (i > nfields)
- growfldtab(i);
- if (freeable(fldtab[i]))
- xfree(fldtab[i]->sval);
- fldtab[i]->sval = fr;
- fldtab[i]->tval = FLD | STR | DONTFREE;
- if (*r == '"' ) { /* start of "..." */
- for (r++ ; *r != '\0'; ) {
- if (*r == '"' && r[1] != '\0' && r[1] == '"') {
- r += 2; /* doubled quote */
- *fr++ = '"';
- } else if (*r == '"' && (r[1] == '\0' || r[1] == ',')) {
- r++; /* skip over closing quote */
- break;
- } else {
- *fr++ = *r++;
- }
- }
- *fr++ = 0;
- } else { /* unquoted field */
- while (*r != ',' && *r != '\0')
- *fr++ = *r++;
- *fr++ = 0;
- }
- if (*r++ == 0)
- break;
-
- }
- }
- *fr = 0;
- } else if ((sep = *inputFS) == 0) { /* new: FS="" => 1 char/field */
- for (i = 0; *r != '\0'; ) {
- char buf[10];
+ } else if ((sep = *inputFS) == 0) { /* new: FS="" => 1 char/field */
+ for (i = 0; *r != '\0'; r += n) {
+ char buf[MB_LEN_MAX + 1];
+
i++;
if (i > nfields)
growfldtab(i);
if (freeable(fldtab[i]))
xfree(fldtab[i]->sval);
- n = u8_nextlen(r);
- for (j = 0; j < n; j++)
- buf[j] = *r++;
- buf[j] = '\0';
+ n = mblen(r, MB_LEN_MAX);
+ if (n < 0)
+ n = 1;
+ memcpy(buf, r, n);
+ buf[n] = '\0';
fldtab[i]->sval = tostring(buf);
fldtab[i]->tval = FLD | STR;
}
*fr = 0;
} else if (*r != 0) { /* if 0, it's a null field */
- /* subtle case: if length(FS) == 1 && length(RS > 0)
+ /* subtlecase : if length(FS) == 1 && length(RS > 0)
* \n is NOT a field separator (cf awk book 61,84).
* this variable is tested in the inner while loop.
*/
@@ -885,11 +796,11 @@ bool is_valid_number(const char *s, bool
while (isspace(*s))
s++;
- /* no hex floating point, sorry */
+ // no hex floating point, sorry
if (s[0] == '0' && tolower(s[1]) == 'x')
return false;
- /* allow +nan, -nan, +inf, -inf, any other letter, no */
+ // allow +nan, -nan, +inf, -inf, any other letter, no
if (s[0] == '+' || s[0] == '-') {
is_nan = (strncasecmp(s+1, "nan", 3) == 0);
is_inf = (strncasecmp(s+1, "inf", 3) == 0);
@@ -923,7 +834,7 @@ convert:
if (no_trailing != NULL)
*no_trailing = (*ep == '\0');
- /* return true if found the end, or trailing stuff is allowed */
+ // return true if found the end, or trailing stuff is allowed
retval = *ep == '\0' || trailing_stuff_ok;
return retval;
Index: pkgsrc/lang/nawk/files/proto.h
diff -u pkgsrc/lang/nawk/files/proto.h:1.6 pkgsrc/lang/nawk/files/proto.h:1.7
--- pkgsrc/lang/nawk/files/proto.h:1.6 Tue Sep 12 19:16:52 2023
+++ pkgsrc/lang/nawk/files/proto.h Sun Sep 17 10:32:06 2023
@@ -43,13 +43,14 @@ extern fa *mkdfa(const char *, bool);
extern int makeinit(fa *, bool);
extern void penter(Node *);
extern void freetr(Node *);
+extern int hexstr(const uschar **);
extern int quoted(const uschar **);
-extern int *cclenter(const char *);
+extern char *cclenter(const char *);
extern noreturn void overflo(const char *);
extern void cfoll(fa *, Node *);
extern int first(Node *);
extern void follow(Node *);
-extern int member(int, int *);
+extern int member(int, const char *);
extern int match(fa *, const char *);
extern int pmatch(fa *, const char *);
extern int nematch(fa *, const char *);
Index: pkgsrc/lang/nawk/files/tran.c
diff -u pkgsrc/lang/nawk/files/tran.c:1.6 pkgsrc/lang/nawk/files/tran.c:1.7
--- pkgsrc/lang/nawk/files/tran.c:1.6 Tue Sep 12 19:16:52 2023
+++ pkgsrc/lang/nawk/files/tran.c Sun Sep 17 10:32:06 2023
@@ -308,7 +308,7 @@ Awkfloat setfval(Cell *vp, Awkfloat f) /
} else if (&vp->fval == NF) {
donerec = false; /* mark $0 invalid */
setlastfld(f);
- DPRINTF("setfval: setting NF to %g\n", f);
+ DPRINTF("setting NF to %g\n", f);
} else if (isrec(vp)) {
donefld = false; /* mark $1... invalid */
donerec = true;
@@ -348,10 +348,6 @@ char *setsval(Cell *vp, const char *s) /
(void*)vp, NN(vp->nval), s, vp->tval, donerec, donefld);
if ((vp->tval & (NUM | STR)) == 0)
funnyvar(vp, "assign to");
- if (CSV && (vp == rsloc))
- WARNING("danger: don't set RS when --csv is in effect");
- if (CSV && (vp == fsloc))
- WARNING("danger: don't set FS when --csv is in effect");
if (isfld(vp)) {
donerec = false; /* mark $0 invalid */
fldno = atoi(vp->nval);
@@ -379,7 +375,7 @@ char *setsval(Cell *vp, const char *s) /
donerec = false; /* mark $0 invalid */
f = getfval(vp);
setlastfld(f);
- DPRINTF("setsval: setting NF to %g\n", f);
+ DPRINTF("setting NF to %g\n", f);
}
return(vp->sval);
Index: pkgsrc/lang/nawk/files/nawk.1
diff -u pkgsrc/lang/nawk/files/nawk.1:1.3 pkgsrc/lang/nawk/files/nawk.1:1.4
--- pkgsrc/lang/nawk/files/nawk.1:1.3 Tue Sep 12 19:16:52 2023
+++ pkgsrc/lang/nawk/files/nawk.1 Sun Sep 17 10:32:06 2023
@@ -1,11 +1,11 @@
-.\" $NetBSD: nawk.1,v 1.3 2023/09/12 19:16:52 vins Exp $
+.\" $NetBSD: nawk.1,v 1.4 2023/09/17 10:32:06 vins Exp $
.\"
-.\" This file is copied from awk.1 but with the following modifications
+.\" This file is copied from nawk.1 but with the following modifications
.\" for pkgsrc:
.\"
-.\" * awk is changed to nawk.
-.\" * Awk is changed to Nawk.
-.\" * AWK is changed to NAWK.
+.\" * nawk is changed to nnawk.
+.\" * Nawk is changed to Nnawk.
+.\" * NAWK is changed to NNAWK.
.\"
.de EX
.nf
@@ -29,8 +29,6 @@ nawk \- pattern-directed scanning and pr
[
.BI \-F
.I fs
-|
-.B \-\^\-csv
]
[
.BI \-v
@@ -87,12 +85,6 @@ The
.I fs
option defines the input field separator to be the regular expression
.IR fs .
-The
-.B \-\^\-csv
-option causes
-.I nawk
-to process records using (more or less) standard comma-separated values
-(CSV) format.
.PP
An input line is normally made up of fields separated by white space,
or by the regular expression
@@ -219,9 +211,9 @@ and
.B sqrt
are built in.
Other built-in functions:
-.TF "\fBlength(\fR[\fIv\^\fR]\fB)\fR"
+.TF length
.TP
-\fBlength(\fR[\fIv\^\fR]\fB)\fR
+.B length
the length of its argument
taken as a string,
number of elements in an array for an array argument,
@@ -229,15 +221,15 @@ or length of
.B $0
if no argument.
.TP
-.B rand()
+.B rand
random number on [0,1).
.TP
-\fBsrand(\fR[\fIs\^\fR]\fB)\fR
+.B srand
sets seed for
.B rand
and returns the previous seed.
.TP
-.BI int( x\^ )
+.B int
truncates to an integer value.
.TP
\fBsubstr(\fIs\fB, \fIm\fR [\fB, \fIn\^\fR]\fB)\fR
@@ -406,7 +398,7 @@ in a pattern.
A pattern may consist of two patterns separated by a comma;
in this case, the action is performed for all lines
from an occurrence of the first pattern
-through an occurrence of the second, inclusive.
+though an occurrence of the second.
.PP
A relational expression is one of the following:
.IP
@@ -416,7 +408,7 @@ A relational expression is one of the fo
.br
.IB expression " in " array-name
.br
-.BI ( expr ,\| expr ,\| ... ") in " array-name
+.BI ( expr , expr,... ") in " array-name
.PP
where a
.I relop
@@ -516,7 +508,7 @@ separates multiple subscripts (default 0
Functions may be defined (at the position of a pattern-action statement) thus:
.IP
.B
-function foo(a, b, c) { ... }
+function foo(a, b, c) { ...; return x }
.PP
Parameters are passed by value if scalar and by reference if array name;
functions may be called recursively.
@@ -582,8 +574,8 @@ BEGIN { # Simulate echo(1)
.IR sed (1)
.br
A. V. Aho, B. W. Kernighan, P. J. Weinberger,
-.IR "The NAWK Programming Language, Second Edition" ,
-Addison-Wesley, 2024. ISBN 978-0-13-826972-2, 0-13-826972-6.
+.IR "The NAWK Programming Language" ,
+Addison-Wesley, 1988. ISBN 0-201-07981-X.
.SH BUGS
There are no explicit conversions between numbers and strings.
To force an expression to be treated as a number add 0 to it;
@@ -593,8 +585,7 @@ to force it to be treated as a string co
The scope rules for variables in functions are a botch;
the syntax is worse.
.PP
-Input is expected to be UTF-8 encoded. Other multibyte
-character sets are not handled.
+Only eight-bit characters sets are handled correctly.
.SH UNUSUAL FLOATING-POINT VALUES
.I Nawk
was designed before IEEE 754 arithmetic defined Not-A-Number (NaN)
Home |
Main Index |
Thread Index |
Old Index