Lua shared object asymmetry loading Xlib.

[Phil Rulon <pjr%zephyrite.net@localhost>]

Over the last few days, an unusual issue has come to light.  If a lua
module loads Xlib and makes calls into it lua is returning a boolean value
instead of a table.  This is happening with lua 5.3.4 from pkgsrc.  If the
core lua5.3.3 is used the module will load properly but segfault at program
exit.  This has not been observed with other shared objects.
  I have a small code example that demonstrates the problem.  My own
machine is an amd64 with 7.1 and 2018Q3 pkgsrc. The problem has been
confirmed by Marc Balmer <marc%msys.ch@localhost> running amd64-current.  Marc has
also tested the issue on a Fedora machine and it does not occur.
Marc confirmed that his luamotif module is also now broken on amd64-current
but not Fedora.
This does not occur when other shared libs are loaded.  Additionally, if
os.exit is called at the end of the script, the program exits cleanly under
/usr/bin/lua.
The problem has also been tested on a /usr/local build of lua5.3.5 and
behaves the same.

Here is an example.

msd$ cat Xlib.c
#include <X11/Xlib.h>
#include <lua.h>
#include <lauxlib.h>
int luaopen_Xlib(lua_State *);

static int
open_display(lua_State *L) {
  XCloseDisplay(XOpenDisplay(NULL));
  return 0;
}
int
luaopen_Xlib(lua_State *L) {
  struct luaL_Reg luaXlib[] = {
    { "XOpenDisplay", open_display },
    { NULL, NULL }
  };
  luaL_newlib(L, luaXlib);
  return 1;
}

msd$ cat test.lua
local Xlib = require 'Xlib'
print(type(Xlib))
print(type(Xlib.XopenDisplay))
dpy = Xlib.XOpenDisplay()
-- os.exit()            if this is in the program exits clean.

msd$ cat Makefile
CFLAGS         = -g -I/usr/X11R7/include -fPIC -std=gnu99 -Werror -c
LUA_MODULES    = Xlib
LUA_SRCS.Xlib  = Xlib.c
LUA_DPLIBS    += X11 /usr/X11R7/lib
LDADD         += -R/usr/X11R7/lib

.include <bsd.lua.mk>

msd$ make
#   compile  xlib/Xlib.o
gcc -g -I/usr/X11R7/include -fPIC -std=gnu99 -Werror -c -fPIC
-std=gnu99   -Werror     -c    Xlib.c
#     build  xlib/Xlib.so
rm -f Xlib.so
gcc -Wl,--warn-shared-textrel  -Wl,-x -shared Xlib.o  -Wl,-soname,Xlib.so
-o Xlib.so  -R/usr/X11R7/lib -L/usr/X11R7/lib -lX11 -llua

msd$ /usr/bin/lua -v test.lua
Lua 5.3.3  Copyright (C) 1994-2016 Lua.org, PUC-Rio
table
function
[1]   Segmentation fault (core dumped) /usr/bin/lua -v test.lua

msd$ gdb -q /usr/bin/lua lua.core
Reading symbols from /usr/bin/lua...(no debugging symbols found)...done.
[New process 1]
Core was generated by `lua'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007f7ff5001458 in ?? ()
(gdb) where
#0  0x00007f7ff5001458 in ?? ()
#1  0x00007f7ff64f3d69 in __cxa_finalize () from /usr/lib/libc.so.12
#2  0x00007f7ff64f3a4a in exit () from /usr/lib/libc.so.12
#3  0x000000000040176c in ___start ()
#4  0x00007f7ff7ffa000 in ?? ()
#5  0x0000000000000003 in ?? ()
#6  0x00007f7ffffffdf0 in ?? ()
#7  0x00007f7ffffffdfd in ?? ()
#8  0x00007f7ffffffe00 in ?? ()
#9  0x0000000000000000 in ?? ()
(gdb) quit

msd$ /usr/pkg/bin/lua5.3 -v test.lua
Lua 5.3.4  Copyright (C) 1994-2017 Lua.org, PUC-Rio
boolean
/usr/pkg/bin/lua5.3: test.lua:4: attempt to index a boolean value (local
'Xlib')
stack traceback:
        test.lua:4: in main chunk
        [C]: in ?