[src/trunk]: src/bin/sh Fix a bug with here document processing reported on t...

[kre <kre%NetBSD.org@localhost>]

details:   https://anonhg.NetBSD.org/src/rev/d3d55a5a20b8
branches:  trunk
changeset: 985811:d3d55a5a20b8
user:      kre <kre%NetBSD.org@localhost>
date:      Thu Sep 09 01:14:04 2021 +0000

description:
Fix a bug with here document processing reported on the austin group list
by oguzismailuysal%gmail.com@localhost (2021-09-08) (applies to all ash descendant
shells).

There were places in the parser where newline tokens were handled
without reading any pending here documents (leaving those until a
later newline token).

Eg: in
        : << do | for x in xxx
        do
        do echo $x
        done

The here doc text (<<do) should start immediately after the next newline
(which is after xxx).   But it wasn't happening until after the following
newline (after the line containing only "do").

Bizarrely, this
        : << do | for x in xxx
        do echo $x
        do
        done
"worked" because of that.

For other cases that also failed, see the hard_cases test case in
src/tests/bin/sh/t_here.sh   Note that there's nothing magic about
the particular end delimiter word chosen here, any will do, using
the ones selected for the tests (and shown here) simply makes it
all more mysterious!   The one here is the exact case reported.

Fix this by reading the here docs when processing newline tokens
when they are encountered in other than the "normal" way.   Whether
this catches every possibility is unknown currently - may never be
known for certain, but there are no more I can currently think of.

No pullups needed, this isn't a significant enough bug (ie: no one
actually writes code like this) to warrant that.

diffstat:

 bin/sh/parser.c |  8 +++++---
 1 files changed, 5 insertions(+), 3 deletions(-)

diffs (36 lines):

diff -r 095e7f8e4e8a -r d3d55a5a20b8 bin/sh/parser.c
--- a/bin/sh/parser.c   Thu Sep 09 00:04:51 2021 +0000
+++ b/bin/sh/parser.c   Thu Sep 09 01:14:04 2021 +0000
@@ -1,4 +1,4 @@
-/*     $NetBSD: parser.c,v 1.171 2020/08/19 22:41:47 kre Exp $ */
+/*     $NetBSD: parser.c,v 1.172 2021/09/09 01:14:04 kre Exp $ */
 
 /*-
  * Copyright (c) 1991, 1993
@@ -37,7 +37,7 @@
 #if 0
 static char sccsid[] = "@(#)parser.c   8.7 (Berkeley) 5/16/95";
 #else
-__RCSID("$NetBSD: parser.c,v 1.171 2020/08/19 22:41:47 kre Exp $");
+__RCSID("$NetBSD: parser.c,v 1.172 2021/09/09 01:14:04 kre Exp $");
 #endif
 #endif /* not lint */
 
@@ -422,6 +422,8 @@
                        n1->nfor.args = ap;
                        if (lasttoken != TNL && lasttoken != TSEMI)
                                synexpect(TSEMI, 0);
+                       if (lasttoken == TNL)
+                               readheredocs();
                } else {
                        static char argvars[5] = {
                            CTLVAR, VSNORMAL|VSQUOTE, '@', '=', '\0'
@@ -2506,7 +2508,7 @@
 linebreak(void)
 {
        while (readtoken() == TNL)
-               ;
+               readheredocs();
 }
 
 /*