[src/trunk]: src/usr.bin/make/unit-tests make: demonstrate how to undefine va...

[rillig <rillig%NetBSD.org@localhost>]

details:   https://anonhg.NetBSD.org/src/rev/fceb02e4464e
branches:  trunk
changeset: 953003:fceb02e4464e
user:      rillig <rillig%NetBSD.org@localhost>
date:      Tue Feb 23 14:17:21 2021 +0000

description:
make: demonstrate how to undefine variables during evaluation

For a very long time now, I had thought that it would be impossible to
undefine global variables during the evaluation of variable expressions.
This is something that the memory management in Var_Parse relies upon,
see the comment 'the value of the variable must not change'.

After several unsuccessful attempts at referring to an already freed
previous value of a variable, today I discovered how to unset a global
variable while evaluating an expression, which has the same effect.  To
demonstrate that this use-after-free can reliably crash make, it would
need a memory allocator with a debug mode that never re-allocates the
same memory block after it has been used once.  This is something that
jemalloc cannot do at the moment.  Valgrind would be another idea, but
that has not been ported to NetBSD.

Undefining a global variable while evaluating an expression is made
possible by an implementation detail of the modifier ':@'.  That
modifier undefines the loop variable, without restoring its previous
value, see ApplyModifier_Loop.

By the very old conventions of ODE Make, these loop variables are named
'.V.' and thus do not conflict with variables from other naming
conventions.  In NetBSD and pkgsrc, these loop variables are typically
called 'var', sometimes '_var' with a leading underscore, which also
doesn't conflict with the typical form 'VAR' of variables in the global
namespace.  Therefore, in practice these loop variables don't interfere
with other variables.

One case that can practically arise is when an outer variable has a
modifier ':@word@${VAR.${word}}@' and one of the referenced variables
uses the same variable name in the modifier, see varmod-loop.mk 1.10
line 91 for a detailed explanation.

By using the ${:@VAR@@} modifier in a place that is evaluated with
cmdline scope, it is not only possible to undefine global variables, it
is possible to undefine cmdline variables as well.  When evaluated in a
specific make target, the expression ${:@\@@@} can even be used to
undefine the variable '.TARGET', which will probably crash make with an
assertion failure.

diffstat:

 usr.bin/make/unit-tests/var-class-cmdline.exp |    2 +-
 usr.bin/make/unit-tests/var-class-cmdline.mk  |    6 +-
 usr.bin/make/unit-tests/varmod-loop.exp       |    6 +-
 usr.bin/make/unit-tests/varmod-loop.mk        |  111 ++++++++++++++++++++++++-
 4 files changed, 113 insertions(+), 12 deletions(-)

diffs (189 lines):

diff -r 641743ae30ca -r fceb02e4464e usr.bin/make/unit-tests/var-class-cmdline.exp
--- a/usr.bin/make/unit-tests/var-class-cmdline.exp     Tue Feb 23 11:31:52 2021 +0000
+++ b/usr.bin/make/unit-tests/var-class-cmdline.exp     Tue Feb 23 14:17:21 2021 +0000
@@ -1,4 +1,4 @@
 make: "var-class-cmdline.mk" line 23: global
-make: "var-class-cmdline.mk" line 30: makeflags
+make: "var-class-cmdline.mk" line 32: makeflags
 makeflags
 exit status 0
diff -r 641743ae30ca -r fceb02e4464e usr.bin/make/unit-tests/var-class-cmdline.mk
--- a/usr.bin/make/unit-tests/var-class-cmdline.mk      Tue Feb 23 11:31:52 2021 +0000
+++ b/usr.bin/make/unit-tests/var-class-cmdline.mk      Tue Feb 23 14:17:21 2021 +0000
@@ -1,4 +1,4 @@
-# $NetBSD: var-class-cmdline.mk,v 1.3 2021/02/22 22:04:28 rillig Exp $
+# $NetBSD: var-class-cmdline.mk,v 1.4 2021/02/23 14:17:21 rillig Exp $
 #
 # Tests for variables specified on the command line.
 #
@@ -23,9 +23,11 @@
 .info ${VAR}
 
 # The global variable is "overridden" by simply deleting it and then
-# installing the cmdline variable instead.  Since there is no way to
+# installing the cmdline variable instead.  Since there is no obvious way to
 # undefine a cmdline variable, there is no need to remember the old value
 # of the global variable could become visible again.
+#
+# See varmod-loop.mk for a non-obvious way to undefine a cmdline variable.
 .MAKEFLAGS: VAR=makeflags
 .info ${VAR}
 
diff -r 641743ae30ca -r fceb02e4464e usr.bin/make/unit-tests/varmod-loop.exp
--- a/usr.bin/make/unit-tests/varmod-loop.exp   Tue Feb 23 11:31:52 2021 +0000
+++ b/usr.bin/make/unit-tests/varmod-loop.exp   Tue Feb 23 14:17:21 2021 +0000
@@ -1,10 +1,10 @@
-ParseReadLine (117): 'USE_8_DOLLARS=   ${:U1:@var@${8_DOLLARS}@} ${8_DOLLARS} $$$$$$$$'
+ParseReadLine (132): 'USE_8_DOLLARS=   ${:U1:@var@${8_DOLLARS}@} ${8_DOLLARS} $$$$$$$$'
 CondParser_Eval: ${USE_8_DOLLARS} != "\$\$\$\$ \$\$\$\$ \$\$\$\$"
 lhs = "$$$$ $$$$ $$$$", rhs = "$$$$ $$$$ $$$$", op = !=
-ParseReadLine (122): 'SUBST_CONTAINING_LOOP:= ${USE_8_DOLLARS}'
+ParseReadLine (137): 'SUBST_CONTAINING_LOOP:= ${USE_8_DOLLARS}'
 CondParser_Eval: ${SUBST_CONTAINING_LOOP} != "\$\$ \$\$\$\$ \$\$\$\$"
 lhs = "$$ $$$$ $$$$", rhs = "$$ $$$$ $$$$", op = !=
-ParseReadLine (147): '.MAKEFLAGS: -d0'
+ParseReadLine (162): '.MAKEFLAGS: -d0'
 ParseDoDependency(.MAKEFLAGS: -d0)
 :+one+ +two+ +three+:
 :x1y x2y x3y:
diff -r 641743ae30ca -r fceb02e4464e usr.bin/make/unit-tests/varmod-loop.mk
--- a/usr.bin/make/unit-tests/varmod-loop.mk    Tue Feb 23 11:31:52 2021 +0000
+++ b/usr.bin/make/unit-tests/varmod-loop.mk    Tue Feb 23 14:17:21 2021 +0000
@@ -1,4 +1,4 @@
-# $NetBSD: varmod-loop.mk,v 1.9 2021/02/04 21:42:47 rillig Exp $
+# $NetBSD: varmod-loop.mk,v 1.10 2021/02/23 14:17:21 rillig Exp $
 #
 # Tests for the :@var@...${var}...@ variable modifier.
 
@@ -34,13 +34,26 @@
        @echo :${:U1 2 3:@\\@x${${:Ux:S,x,\\,}}y@}:
 
        # The variable name can technically be empty, and in this situation
-       # the variable value cannot be accessed since the empty variable is
-       # protected to always return an empty string.
+       # the variable value cannot be accessed since the empty "variable"
+       # is protected to always return an empty string.
        @echo empty: :${:U1 2 3:@@x${}y@}:
 
-# The :@ modifier resolves the variables a little more often than expected.
-# In particular, it resolves _all_ variables from the scope, and not only
-# the loop variable (in this case v).
+
+# The :@ modifier resolves the variables from the replacement text once more
+# than expected.  In particular, it resolves _all_ variables from the scope,
+# and not only the loop variable (in this case v).
+SRCS=          source
+CFLAGS.source= before
+ALL_CFLAGS:=   ${SRCS:@src@${CFLAGS.${src}}@}  # note the ':='
+CFLAGS.source+=        after
+.if ${ALL_CFLAGS} != "before"
+.  error
+.endif
+
+
+# In the following example, the modifier ':@' expands the '$$' to '$'.  This
+# means that when the resulting expression is evaluated, these resulting '$'
+# will be interpreted as starting a subexpression.
 #
 # The d means direct reference, the i means indirect reference.
 RESOLVE=       ${RES1} $${RES1}
@@ -48,9 +61,11 @@
 RES2=          2d${RES3} 2i$${RES3}
 RES3=          3
 
+# TODO: convert to '.if'.
 mod-loop-resolve:
        @echo $@:${RESOLVE:@v@w${v}w@:Q}:
 
+
 # Until 2020-07-20, the variable name of the :@ modifier could end with one
 # or two dollar signs, which were silently ignored.
 # There's no point in allowing a dollar sign in that position.
@@ -145,3 +160,87 @@
 .  error
 .endif
 .MAKEFLAGS: -d0
+
+# After looping over the words of the expression, the loop variable gets
+# undefined.  The modifier ':@' uses an ordinary global variable for this,
+# which is different from the '.for' loop, which replaces ${var} with
+# ${:Uvalue} in the body of the loop.  This choice of implementation detail
+# can be used for a nasty side effect.  The expression ${:U:@VAR@@} evaluates
+# to an empty string, plus it undefines the variable 'VAR'.  This is the only
+# possibility to undefine a global variable during evaluation.
+GLOBAL=                before-global
+RESULT:=       ${:U${GLOBAL} ${:U:@GLOBAL@@} ${GLOBAL:Uundefined}}
+.if ${RESULT} != "before-global  undefined"
+.  error
+.endif
+
+# The above side effect of undefining a variable from a certain scope can be
+# further combined with the otherwise undocumented implementation detail that
+# the argument of an '.if' directive is evaluated in cmdline scope.  Putting
+# these together makes it possible to undefine variables from the cmdline
+# scope, something that is not possible in a straight-forward way.
+.MAKEFLAGS: CMDLINE=cmdline
+.if ${:U${CMDLINE}${:U:@CMDLINE@@}} != "cmdline"
+.  error
+.endif
+# Now the cmdline variable got undefined.
+.if ${CMDLINE} != "cmdline"
+.  error
+.endif
+# At this point, it still looks as if the cmdline variable were defined,
+# since the value of CMDLINE is still "cmdline".  That impression is only
+# superficial though, the cmdline variable is actually deleted.  To
+# demonstrate this, it is now possible to override its value using a global
+# variable, something that was not possible before:
+CMDLINE=       global
+.if ${CMDLINE} != "global"
+.  error
+.endif
+# Now undefine that global variable again, to get back to the original value.
+.undef CMDLINE
+.if ${CMDLINE} != "cmdline"
+.  error
+.endif
+# What actually happened is that when CMDLINE was set by the '.MAKEFLAGS'
+# target in the cmdline scope, that same variable was exported to the
+# environment, see Var_SetWithFlags.
+.unexport CMDLINE
+.if ${CMDLINE} != "cmdline"
+.  error
+.endif
+# The above '.unexport' has no effect since UnexportVar requires a global
+# variable of the same name to be defined, otherwise nothing is unexported.
+CMDLINE=       global
+.unexport CMDLINE
+.undef CMDLINE
+.if ${CMDLINE} != "cmdline"
+.  error
+.endif
+# This still didn't work since there must not only be a global variable, the
+# variable must be marked as exported as well, which it wasn't before.
+CMDLINE=       global
+.export CMDLINE
+.unexport CMDLINE
+.undef CMDLINE
+.if ${CMDLINE:Uundefined} != "undefined"
+.  error
+.endif
+# Finally the variable 'CMDLINE' from the cmdline scope is gone, and all its
+# traces from the environment are gone as well.  To do that, a global variable
+# had to be defined and exported, something that is far from obvious.  To
+# recap, here is the essence of the above story:
+.MAKEFLAGS: CMDLINE=cmdline    # have a cmdline + environment variable
+.if ${:U:@CMDLINE@@}}          # undefine cmdline, keep environment
+.endif
+CMDLINE=       global          # needed for deleting the environment
+.export CMDLINE                        # needed for deleting the environment
+.unexport CMDLINE              # delete the environment
+.undef CMDLINE                 # delete the global helper variable
+.if ${CMDLINE:Uundefined} != "undefined"
+.  error                       # 'CMDLINE' is gone now from all scopes
+.endif
+
+
+# TODO: Actually trigger the undefined behavior (use after free) that was
+#  already suspected in Var_Parse, in the comment 'the value of the variable
+#  must not change'.