lib/54017: jemalloc deadlock?

[martin%NetBSD.org@localhost]

>Number:         54017
>Category:       lib
>Synopsis:       jemalloc deadlock?
>Confidential:   no
>Severity:       serious
>Priority:       high
>Responsible:    lib-bug-people
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Tue Feb 26 10:00:00 +0000 2019
>Originator:     Martin Husemann
>Release:        NetBSD 8.99.34
>Organization:
The NetBSD Foundation, Inc.
>Environment:
System: NetBSD painkiller.duskware.de 8.99.34 NetBSD 8.99.34 (GENERIC64) #43: Mon Feb 25 11:11:53 CET 2019 martin%seven-days-to-the-wolves.aprisoft.de@localhost:/work/src/sys/arch/evbarm/compile/GENERIC64 evbarm
Architecture: aarch64
Machine: evbarm
>Description:

Randomly, but right now pretty reproducably, python27 "locks up" when
building Firefox (in my case FF65 on aarch64).

It is not exactly clear what leads to this.

Before this happens, apparently python has forked some other processes,
and not collected one dying child status yet:

  772 pts/3  I+    0:01.76 |     `-- make install 
  778 pts/3  I+    0:00.01 |       `-- /bin/sh -c set -e;\t\t\t\t\t if test -n "" &&  /usr/sbin/pkg_info
  959 pts/3  I+    0:02.03 |         `-- /usr/bin/make _MAKE OPSYS OS_VERSION LOWER_OPSYS _PKGSRCDIR PKG
 3757 pts/3  I+    0:01.21 |           `-- python2.7 /usr/pkgobj/www/firefox/work/firefox-65.0/configure
 3229 pts/3  Il+   1:28.44 |             `-- /usr/pkgobj/www/firefox/work/build/_virtualenvs/init/bin/py
21913 pts/3  Il+   0:00.69 |               |-- /usr/pkgobj/www/firefox/work/build/_virtualenvs/init/bin/
22011 pts/3  I+    0:00.04 |               |-- /usr/pkgobj/www/firefox/work/build/_virtualenvs/init/bin/
22382 pts/3  O+   47:46.57 |               |-- /usr/pkgobj/www/firefox/work/build/_virtualenvs/init/bin/
19425 pts/3  Z+    0:00.00 |               | `-- (python)
22659 pts/3  I+    0:00.04 |               `-- /usr/pkgobj/www/firefox/work/build/_virtualenvs/init/bin/

Attaching gdb to the python process shows two threads trying to malloc()
two identical quite huge memory ranges. The process seems not to be out
of VM nor hit resource limits, but the jemalloc base_mtx is kept.
Unfortunately it was not clear to me how to map the mutex owner to a lwp
or something gdb would know.

Since both threads block on the mutex and the backtrace shows no recursion,
maybe some other thread somehow got the mutex but now is gone without
releasing it?

[Switching to LWP 1 of process 22382]
pthread__mutex_pause () at /work/src/lib/libpthread/pthread_mutex.c:228
228	/work/src/lib/libpthread/pthread_mutex.c: No such file or directory.
(gdb) set pagination off
(gdb) bt
#0  pthread__mutex_pause () at /work/src/lib/libpthread/pthread_mutex.c:228
#1  0x0000fffffbc39070 in pthread__mutex_spin (ptm=ptm@entry=0xfffffbc254f8, owner=owner@entry=0xfffff7113000) at /work/src/lib/libpthread/pthread_mutex.c:253
#2  0x0000fffffbc393a4 in pthread__mutex_lock_slow (ptm=ptm@entry=0xfffffbc254f8, ts=ts@entry=0x0) at /work/src/lib/libpthread/pthread_mutex.c:343
#3  0x0000fffffbc39998 in pthread_mutex_lock (ptm=ptm@entry=0xfffffbc254f8) at /work/src/lib/libpthread/pthread_mutex.c:202
#4  0x0000fffffbb4fd04 in base_chunk_node_alloc () at /work/src/lib/libc/stdlib/jemalloc.c:1167
#5  0x0000fffffbb4fd74 in huge_malloc (size=<optimized out>) at /work/src/lib/libc/stdlib/jemalloc.c:2795
#6  0x0000fffffbb50a44 in malloc (size=1048613) at /work/src/lib/libc/stdlib/jemalloc.c:3776
#7  0x0000fffffbd5bcc0 in PyObject_Malloc (nbytes=<optimized out>, nbytes@entry=1048613) at Objects/obmalloc.c:986
#8  0x0000fffffbd64b60 in PyString_FromStringAndSize (str=str@entry=0x0, size=1048576) at Objects/stringobject.c:88
#9  0x0000fffffbdf0f00 in posix_read (self=<optimized out>, args=<optimized out>) at ./Modules/posixmodule.c:6804
#10 0x0000fffffbdbaec8 in ext_do_call (nk=65535, na=-88950752, flags=<optimized out>, pp_stack=0xffffffff8b58, func=0xfffffb8fa3b0) at Python/ceval.c:4683
#11 PyEval_EvalFrameEx (f=f@entry=0xfffffb82b9d8, throwflag=throwflag@entry=0) at Python/ceval.c:3048
#12 0x0000fffffbdbd970 in PyEval_EvalCodeEx (co=0xfffffb826730, globals=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=65535, kws=<optimized out>, kwcount=kwcount@entry=0, defs=0x1, defcount=0, closure=<optimized out>) at Python/ceval.c:3604
#13 0x0000fffffbdbaaf8 in fast_function (nk=0, na=<optimized out>, n=3, pp_stack=0xffffffff8d40, func=0xfffffb52ade8) at Python/ceval.c:4467
#14 call_function (oparg=<optimized out>, pp_stack=0xffffffff8d40) at Python/ceval.c:4392
#15 PyEval_EvalFrameEx (f=f@entry=0xfffffab2b820, throwflag=throwflag@entry=0) at Python/ceval.c:3009
#16 0x0000fffffbdbd970 in PyEval_EvalCodeEx (co=0xfffffb83b7b0, globals=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=0, kws=<optimized out>, kwcount=kwcount@entry=0, defs=0xfffff8460ca0, defcount=0, closure=<optimized out>) at Python/ceval.c:3604
#17 0x0000fffffbdbaaf8 in fast_function (nk=0, na=<optimized out>, n=18, pp_stack=0xffffffff8f30, func=0xfffffb52d5f0) at Python/ceval.c:4467
#18 call_function (oparg=<optimized out>, pp_stack=0xffffffff8f30) at Python/ceval.c:4392
#19 PyEval_EvalFrameEx (f=f@entry=0xfffffab2b020, throwflag=throwflag@entry=0) at Python/ceval.c:3009
#20 0x0000fffffbdbd970 in PyEval_EvalCodeEx (co=0xfffffb826a30, globals=<optimized out>, locals=locals@entry=0x0, args=args@entry=0xfffff8ba5338, argcount=65535, kws=kws@entry=0xfffff8460ca0, kwcount=kwcount@entry=5, defs=0xfffff849a280, defs@entry=0xfffffb645428, defcount=defcount@entry=13, closure=<optimized out>) at Python/ceval.c:3604
#21 0x0000fffffbd418b4 in function_call (func=0xfffffb52d1b8, arg=0xfffff8ba5320, kw=0xfffff849a280) at Objects/funcobject.c:523
#22 0x0000fffffbd1a9cc in PyObject_Call (func=func@entry=0xfffffb52d1b8, arg=arg@entry=0xfffff8ba5320, kw=kw@entry=0xfffff849a280) at Objects/abstract.c:2547
#23 0x0000fffffbd2a0d4 in instancemethod_call (func=0xfffffb52d1b8, arg=0xfffff8ba5320, kw=0xfffff849a280) at Objects/classobject.c:2600
#24 0x0000fffffbd1a9cc in PyObject_Call (func=func@entry=0xfffff84dfeb0, arg=arg@entry=0xfffff8470810, kw=kw@entry=0xfffff849a280) at Objects/abstract.c:2547
#25 0x0000fffffbd778c8 in slot_tp_init (self=<optimized out>, args=0xfffff8470810, kwds=0xfffff849a280) at Objects/typeobject.c:5860
#26 0x0000fffffbd745c4 in type_call (type=<optimized out>, args=0xfffff8470810, kwds=0xfffff849a280) at Objects/typeobject.c:765
#27 0x0000fffffbd1a9cc in PyObject_Call (func=func@entry=0xfffffb76b020, arg=arg@entry=0xfffff8470810, kw=kw@entry=0xfffff849a280) at Objects/abstract.c:2547
#28 0x0000fffffbdb852c in do_call (nk=<optimized out>, na=<optimized out>, pp_stack=0xffffffff9520, func=0xfffffb76b020) at Python/ceval.c:4589
#29 call_function (oparg=<optimized out>, pp_stack=0xffffffff9520) at Python/ceval.c:4394
#30 PyEval_EvalFrameEx (f=f@entry=0xfffff87e2820, throwflag=throwflag@entry=0) at Python/ceval.c:3009
#31 0x0000fffffbdbcc7c in fast_function (nk=<optimized out>, na=<optimized out>, n=4, pp_stack=0xffffffff9670, func=0xfffff84d1398) at Python/ceval.c:4457
#32 call_function (oparg=<optimized out>, pp_stack=0xffffffff9670) at Python/ceval.c:4392
#33 PyEval_EvalFrameEx (f=f@entry=0xfffff87e5420, throwflag=throwflag@entry=0) at Python/ceval.c:3009
#34 0x0000fffffbdbd970 in PyEval_EvalCodeEx (co=0xfffff85b5b30, globals=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=65535, kws=<optimized out>, kwcount=kwcount@entry=0, defs=0x1, defcount=1, closure=<optimized out>) at Python/ceval.c:3604
#35 0x0000fffffbdbaaf8 in fast_function (nk=0, na=<optimized out>, n=5, pp_stack=0xffffffff9860, func=0xfffff84d1668) at Python/ceval.c:4467
#36 call_function (oparg=<optimized out>, pp_stack=0xffffffff9860) at Python/ceval.c:4392
#37 PyEval_EvalFrameEx (f=f@entry=0xfffff87ca020, throwflag=throwflag@entry=0) at Python/ceval.c:3009
#38 0x0000fffffbdbd970 in PyEval_EvalCodeEx (co=0xfffff85b5b30, globals=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=65535, kws=<optimized out>, kwcount=kwcount@entry=0, defs=0x1, defcount=1, closure=<optimized out>) at Python/ceval.c:3604
#39 0x0000fffffbdbaaf8 in fast_function (nk=0, na=<optimized out>, n=5, pp_stack=0xffffffff9a50, func=0xfffff84d1668) at Python/ceval.c:4467
#40 call_function (oparg=<optimized out>, pp_stack=0xffffffff9a50) at Python/ceval.c:4392
#41 PyEval_EvalFrameEx (f=f@entry=0xfffff87e2c20, throwflag=throwflag@entry=0) at Python/ceval.c:3009
#42 0x0000fffffbdbd970 in PyEval_EvalCodeEx (co=0xfffff85b5b30, globals=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=65535, kws=<optimized out>, kwcount=kwcount@entry=0, defs=0x1, defcount=1, closure=<optimized out>) at Python/ceval.c:3604
#43 0x0000fffffbdbaaf8 in fast_function (nk=0, na=<optimized out>, n=5, pp_stack=0xffffffff9c40, func=0xfffff84d1668) at Python/ceval.c:4467
#44 call_function (oparg=<optimized out>, pp_stack=0xffffffff9c40) at Python/ceval.c:4392
#45 PyEval_EvalFrameEx (f=f@entry=0xfffff87b1420, throwflag=throwflag@entry=0) at Python/ceval.c:3009
#46 0x0000fffffbdbd970 in PyEval_EvalCodeEx (co=0xfffff85b5b30, globals=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=65535, kws=<optimized out>, kwcount=kwcount@entry=0, defs=0xfffffb905090, defcount=1, closure=<optimized out>) at Python/ceval.c:3604
#47 0x0000fffffbdbaaf8 in fast_function (nk=0, na=<optimized out>, n=4, pp_stack=0xffffffff9e30, func=0xfffff84d1668) at Python/ceval.c:4467
#48 call_function (oparg=<optimized out>, pp_stack=0xffffffff9e30) at Python/ceval.c:4392
#49 PyEval_EvalFrameEx (f=f@entry=0xfffff8798820, throwflag=throwflag@entry=0) at Python/ceval.c:3009
#50 0x0000fffffbdbcc7c in fast_function (nk=<optimized out>, na=<optimized out>, n=8, pp_stack=0xffffffff9f80, func=0xfffff84cdf50) at Python/ceval.c:4457
#51 call_function (oparg=<optimized out>, pp_stack=0xffffffff9f80) at Python/ceval.c:4392
#52 PyEval_EvalFrameEx (f=f@entry=0xfffff8798020, throwflag=throwflag@entry=0) at Python/ceval.c:3009
#53 0x0000fffffbdbcc7c in fast_function (nk=<optimized out>, na=<optimized out>, n=10, pp_stack=0xffffffffa0d0, func=0xfffff84d4aa0) at Python/ceval.c:4457
#54 call_function (oparg=<optimized out>, pp_stack=0xffffffffa0d0) at Python/ceval.c:4392
#55 PyEval_EvalFrameEx (f=f@entry=0xfffff8791c20, throwflag=throwflag@entry=0) at Python/ceval.c:3009
#56 0x0000fffffbdbd970 in PyEval_EvalCodeEx (co=0xfffff85adbb0, globals=<optimized out>, locals=locals@entry=0x0, args=args@entry=0xfffff846d4e0, argcount=65535, kws=kws@entry=0x0, kwcount=kwcount@entry=0, defs=0x6, defs@entry=0xfffff856c458, defcount=defcount@entry=7, closure=<optimized out>) at Python/ceval.c:3604
#57 0x0000fffffbd417d0 in function_call (func=0xfffff881f2a8, arg=0xfffff846d4c8, kw=0x0) at Objects/funcobject.c:523
#58 0x0000fffffbd1a9cc in PyObject_Call (func=func@entry=0xfffff881f2a8, arg=arg@entry=0xfffff846d4c8, kw=kw@entry=0x0) at Objects/abstract.c:2547
#59 0x0000fffffbdb6ce0 in ext_do_call (nk=65535, na=-23664, flags=<optimized out>, pp_stack=0xffffffffa388, func=0xfffff881f2a8) at Python/ceval.c:4686
#60 PyEval_EvalFrameEx (f=f@entry=0xfffff8ac8bc0, throwflag=throwflag@entry=0) at Python/ceval.c:3048
#61 0x0000fffffbdbd970 in PyEval_EvalCodeEx (co=0xfffff85ad1b0, globals=<optimized out>, locals=locals@entry=0x0, args=args@entry=0xfffff846d2d8, argcount=65535, kws=kws@entry=0xfffffba10068, kwcount=kwcount@entry=0, defs=0xfffff90fbdc4, defs@entry=0x0, defcount=defcount@entry=0, closure=<optimized out>) at Python/ceval.c:3604
#62 0x0000fffffbd418b4 in function_call (func=0xfffff8472b18, arg=0xfffff846d2c0, kw=0xfffff847c398) at Objects/funcobject.c:523
#63 0x0000fffffbd1a9cc in PyObject_Call (func=func@entry=0xfffff8472b18, arg=arg@entry=0xfffff846d2c0, kw=kw@entry=0x0) at Objects/abstract.c:2547
#64 0x0000fffffbdb6ce0 in ext_do_call (nk=65535, na=-22976, flags=<optimized out>, pp_stack=0xffffffffa638, func=0xfffff8472b18) at Python/ceval.c:4686
#65 PyEval_EvalFrameEx (f=f@entry=0xfffff8478620, throwflag=throwflag@entry=0) at Python/ceval.c:3048
#66 0x0000fffffbdbd970 in PyEval_EvalCodeEx (co=0xfffff9010db0, globals=<optimized out>, locals=locals@entry=0x0, args=args@entry=0xfffff8cbba40, argcount=65535, kws=kws@entry=0xfffffba10068, kwcount=kwcount@entry=0, defs=0xfffff96d12f2, defs@entry=0x0, defcount=defcount@entry=0, closure=<optimized out>) at Python/ceval.c:3604
#67 0x0000fffffbd418b4 in function_call (func=0xfffff901bed8, arg=0xfffff8cbba28, kw=0xfffff8479168) at Objects/funcobject.c:523
#68 0x0000fffffbd1a9cc in PyObject_Call (func=func@entry=0xfffff901bed8, arg=arg@entry=0xfffff8cbba28, kw=kw@entry=0x1) at Objects/abstract.c:2547
#69 0x0000fffffbdb6ce0 in ext_do_call (nk=65535, na=-117282466, flags=<optimized out>, pp_stack=0xffffffffa8e8, func=0xfffff901bed8) at Python/ceval.c:4686
#70 PyEval_EvalFrameEx (f=f@entry=0xfffff8477ad0, throwflag=throwflag@entry=0) at Python/ceval.c:3048
#71 0x0000fffffbdbcc7c in fast_function (nk=<optimized out>, na=<optimized out>, n=1, pp_stack=0xffffffffaa30, func=0xfffff96ee848) at Python/ceval.c:4457
#72 call_function (oparg=<optimized out>, pp_stack=0xffffffffaa30) at Python/ceval.c:4392
#73 PyEval_EvalFrameEx (f=f@entry=0xfffff897de18, throwflag=throwflag@entry=0) at Python/ceval.c:3009
#74 0x0000fffffbdbcc7c in fast_function (nk=<optimized out>, na=<optimized out>, n=1, pp_stack=0xffffffffab80, func=0xfffff96eeed8) at Python/ceval.c:4457
#75 call_function (oparg=<optimized out>, pp_stack=0xffffffffab80) at Python/ceval.c:4392
#76 PyEval_EvalFrameEx (f=f@entry=0xfffff8631220, throwflag=throwflag@entry=0) at Python/ceval.c:3009
#77 0x0000fffffbdbd970 in PyEval_EvalCodeEx (co=0xfffff8e0f630, globals=<optimized out>, locals=locals@entry=0x0, args=args@entry=0xfffff8cbe800, argcount=65535, kws=kws@entry=0x0, kwcount=kwcount@entry=0, defs=0xfffffbe76000 <_PyImport_DynLoadFiletab>, defs@entry=0x0, defcount=defcount@entry=0, closure=<optimized out>) at Python/ceval.c:3604
#78 0x0000fffffbd417d0 in function_call (func=0xfffff8e0e8c0, arg=0xfffff8cbe7e8, kw=0x0) at Objects/funcobject.c:523
#79 0x0000fffffbd1a9cc in PyObject_Call (func=func@entry=0xfffff8e0e8c0, arg=arg@entry=0xfffff8cbe7e8, kw=kw@entry=0x0) at Objects/abstract.c:2547
#80 0x0000fffffbd2a0d4 in instancemethod_call (func=0xfffff8e0e8c0, arg=0xfffff8cbe7e8, kw=0x0) at Objects/classobject.c:2600
#81 0x0000fffffbd1a9cc in PyObject_Call (func=func@entry=0xfffff8e37500, arg=arg@entry=0xfffff8638090, kw=kw@entry=0x0) at Objects/abstract.c:2547
#82 0x0000fffffbd778c8 in slot_tp_init (self=<optimized out>, args=0xfffff8638090, kwds=0x0) at Objects/typeobject.c:5860
#83 0x0000fffffbd745c4 in type_call (type=<optimized out>, args=0xfffff8638090, kwds=0x0) at Objects/typeobject.c:765
#84 0x0000fffffbd1a9cc in PyObject_Call (func=func@entry=0xfffff8d86c20, arg=arg@entry=0xfffff8638090, kw=kw@entry=0x0) at Objects/abstract.c:2547
#85 0x0000fffffbdb852c in do_call (nk=<optimized out>, na=<optimized out>, pp_stack=0xffffffffb170, func=0xfffff8d86c20) at Python/ceval.c:4589
#86 call_function (oparg=<optimized out>, pp_stack=0xffffffffb170) at Python/ceval.c:4394
#87 PyEval_EvalFrameEx (f=f@entry=0xfffff8477210, throwflag=throwflag@entry=0) at Python/ceval.c:3009
#88 0x0000fffffbdbcc7c in fast_function (nk=<optimized out>, na=<optimized out>, n=1, pp_stack=0xffffffffb2c0, func=0xfffff96ee8c0) at Python/ceval.c:4457
#89 call_function (oparg=<optimized out>, pp_stack=0xffffffffb2c0) at Python/ceval.c:4392
#90 PyEval_EvalFrameEx (f=f@entry=0xfffff8478240, throwflag=throwflag@entry=0) at Python/ceval.c:3009
#91 0x0000fffffbdbcc7c in fast_function (nk=<optimized out>, na=<optimized out>, n=1, pp_stack=0xffffffffb410, func=0xfffff901d410) at Python/ceval.c:4457
#92 call_function (oparg=<optimized out>, pp_stack=0xffffffffb410) at Python/ceval.c:4392
#93 PyEval_EvalFrameEx (f=f@entry=0xfffff95bf020, throwflag=throwflag@entry=0) at Python/ceval.c:3009
#94 0x0000fffffbdbd970 in PyEval_EvalCodeEx (co=0xfffff90152b0, globals=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=0, kws=<optimized out>, kwcount=kwcount@entry=0, defs=0x0, defcount=0, closure=<optimized out>) at Python/ceval.c:3604
#95 0x0000fffffbdbaaf8 in fast_function (nk=0, na=<optimized out>, n=8, pp_stack=0xffffffffb600, func=0xfffff901d488) at Python/ceval.c:4467
#96 call_function (oparg=<optimized out>, pp_stack=0xffffffffb600) at Python/ceval.c:4392
#97 PyEval_EvalFrameEx (f=f@entry=0xfffff9552c20, throwflag=throwflag@entry=0) at Python/ceval.c:3009
#98 0x0000fffffbdbd970 in PyEval_EvalCodeEx (co=0xfffff85ad4b0, globals=<optimized out>, locals=locals@entry=0x0, args=args@entry=0xfffff84d6518, argcount=65535, kws=kws@entry=0x0, kwcount=kwcount@entry=0, defs=0x1, defs@entry=0x0, defcount=defcount@entry=0, closure=<optimized out>) at Python/ceval.c:3604
#99 0x0000fffffbd417d0 in function_call (func=0xfffff8472c08, arg=0xfffff84d6500, kw=0x0) at Objects/funcobject.c:523
#100 0x0000fffffbd1a9cc in PyObject_Call (func=func@entry=0xfffff8472c08, arg=arg@entry=0xfffff84d6500, kw=kw@entry=0x0) at Objects/abstract.c:2547
#101 0x0000fffffbd2a0d4 in instancemethod_call (func=0xfffff8472c08, arg=0xfffff84d6500, kw=0x0) at Objects/classobject.c:2600
#102 0x0000fffffbd1a9cc in PyObject_Call (func=func@entry=0xfffff8bc2e60, arg=arg@entry=0xfffff856ce50, kw=kw@entry=0x0) at Objects/abstract.c:2547
#103 0x0000fffffbd778c8 in slot_tp_init (self=<optimized out>, args=0xfffff856ce50, kwds=0x0) at Objects/typeobject.c:5860
#104 0x0000fffffbd745c4 in type_call (type=<optimized out>, args=0xfffff856ce50, kwds=0x0) at Objects/typeobject.c:765
#105 0x0000fffffbd1a9cc in PyObject_Call (func=func@entry=0xfffff9593420, arg=arg@entry=0xfffff856ce50, kw=kw@entry=0x0) at Objects/abstract.c:2547
#106 0x0000fffffbdb852c in do_call (nk=<optimized out>, na=<optimized out>, pp_stack=0xffffffffbbf0, func=0xfffff9593420) at Python/ceval.c:4589
#107 call_function (oparg=<optimized out>, pp_stack=0xffffffffbbf0) at Python/ceval.c:4394
#108 PyEval_EvalFrameEx (f=f@entry=0xfffff873e020, throwflag=<optimized out>) at Python/ceval.c:3009
#109 0x0000fffffbd36dc8 in gen_send_ex (arg=0x0, exc=<optimized out>, gen=<optimized out>, gen=<optimized out>) at Objects/genobject.c:85
#110 0x0000fffffbdb94e8 in PyEval_EvalFrameEx (f=f@entry=0xfffff8791820, throwflag=<optimized out>) at Python/ceval.c:2828
#111 0x0000fffffbd36dc8 in gen_send_ex (arg=0x0, exc=<optimized out>, gen=<optimized out>, gen=<optimized out>) at Objects/genobject.c:85
#112 0x0000fffffbdb94e8 in PyEval_EvalFrameEx (f=f@entry=0xfffff8d93c20, throwflag=<optimized out>) at Python/ceval.c:2828
#113 0x0000fffffbd36dc8 in gen_send_ex (arg=0x0, exc=<optimized out>, gen=<optimized out>, gen=<optimized out>) at Objects/genobject.c:85
#114 0x0000fffffbdb94e8 in PyEval_EvalFrameEx (f=f@entry=0xfffff8d93820, throwflag=<optimized out>) at Python/ceval.c:2828
#115 0x0000fffffbd36dc8 in gen_send_ex (arg=0x0, exc=<optimized out>, gen=<optimized out>, gen=<optimized out>) at Objects/genobject.c:85
#116 0x0000fffffbdb94e8 in PyEval_EvalFrameEx (f=f@entry=0xfffff8e95de0, throwflag=<optimized out>) at Python/ceval.c:2828
#117 0x0000fffffbd36dc8 in gen_send_ex (arg=0x0, exc=<optimized out>, gen=<optimized out>, gen=<optimized out>) at Objects/genobject.c:85
#118 0x0000fffffbdb94e8 in PyEval_EvalFrameEx (f=f@entry=0xfffff8d92c20, throwflag=<optimized out>) at Python/ceval.c:2828
#119 0x0000fffffbd36dc8 in gen_send_ex (arg=0x0, exc=<optimized out>, gen=<optimized out>, gen=<optimized out>) at Objects/genobject.c:85
#120 0x0000fffffbd4694c in listextend (self=self@entry=0xfffff8e1c710, b=<optimized out>) at Objects/listobject.c:872
#121 0x0000fffffbd46b74 in list_init (self=0xfffff8e1c710, args=<optimized out>, kw=<optimized out>) at Objects/listobject.c:2459
#122 0x0000fffffbd745c4 in type_call (type=<optimized out>, args=0xfffffb8526d0, kwds=0x0) at Objects/typeobject.c:765
#123 0x0000fffffbd1a9cc in PyObject_Call (func=func@entry=0xfffffbe89158 <PyList_Type>, arg=arg@entry=0xfffffb8526d0, kw=kw@entry=0x0) at Objects/abstract.c:2547
#124 0x0000fffffbdb852c in do_call (nk=<optimized out>, na=<optimized out>, pp_stack=0xffffffffc630, func=0xfffffbe89158 <PyList_Type>) at Python/ceval.c:4589
#125 call_function (oparg=<optimized out>, pp_stack=0xffffffffc630) at Python/ceval.c:4394
#126 PyEval_EvalFrameEx (f=f@entry=0xfffff8d13420, throwflag=throwflag@entry=0) at Python/ceval.c:3009
#127 0x0000fffffbdbd970 in PyEval_EvalCodeEx (co=0xfffff9335eb0, globals=<optimized out>, locals=locals@entry=0x0, args=args@entry=0xfffffba10068, argcount=65535, kws=kws@entry=0xfffff91dbf80, kwcount=kwcount@entry=7, defs=0xfffffab70e8d, defs@entry=0xfffff91b51d0, defcount=defcount@entry=8, closure=<optimized out>) at Python/ceval.c:3604
#128 0x0000fffffbd418b4 in function_call (func=0xfffff8e8e6e0, arg=0xfffffba10050, kw=0xfffff8e945c8) at Objects/funcobject.c:523
#129 0x0000fffffbd1a9cc in PyObject_Call (func=func@entry=0xfffff8e8e6e0, arg=arg@entry=0xfffffba10050, kw=kw@entry=0xfffffac5aed8) at Objects/abstract.c:2547
#130 0x0000fffffbdb6ce0 in ext_do_call (nk=65535, na=-74428272, flags=<optimized out>, pp_stack=0xffffffffc8e8, func=0xfffff8e8e6e0) at Python/ceval.c:4686
#131 PyEval_EvalFrameEx (f=f@entry=0xfffff9516c20, throwflag=throwflag@entry=0) at Python/ceval.c:3048
#132 0x0000fffffbdbd970 in PyEval_EvalCodeEx (co=0xfffffb2feb30, globals=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=0, kws=<optimized out>, kwcount=kwcount@entry=0, defs=0x1, defcount=0, closure=<optimized out>) at Python/ceval.c:3604
#133 0x0000fffffbdbaaf8 in fast_function (nk=0, na=<optimized out>, n=1, pp_stack=0xffffffffcad0, func=0xfffffac5aed8) at Python/ceval.c:4467
#134 call_function (oparg=<optimized out>, pp_stack=0xffffffffcad0) at Python/ceval.c:4392
#135 PyEval_EvalFrameEx (f=f@entry=0xfffffaa2d820, throwflag=throwflag@entry=0) at Python/ceval.c:3009
#136 0x0000fffffbdbd970 in PyEval_EvalCodeEx (co=0xfffffb2ed3b0, globals=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=65535, kws=<optimized out>, kwcount=kwcount@entry=0, defs=0xfffffb8d5168, defcount=0, closure=<optimized out>) at Python/ceval.c:3604
#137 0x0000fffffbdbaaf8 in fast_function (nk=0, na=<optimized out>, n=1, pp_stack=0xffffffffccc0, func=0xfffffac5a938) at Python/ceval.c:4467
#138 call_function (oparg=<optimized out>, pp_stack=0xffffffffccc0) at Python/ceval.c:4392
#139 PyEval_EvalFrameEx (f=f@entry=0xfffffaeac960, throwflag=throwflag@entry=0) at Python/ceval.c:3009
#140 0x0000fffffbdbd970 in PyEval_EvalCodeEx (co=co@entry=0xfffffb2fecb0, globals=globals@entry=0xfffffb8d5168, locals=locals@entry=0xfffffb8d5168, args=args@entry=0x0, argcount=argcount@entry=0, kws=kws@entry=0x0, kwcount=kwcount@entry=0, defs=0xfffffbdda4ac <PyRun_SimpleFileExFlags+220>, defs@entry=0x0, defcount=defcount@entry=0, closure=<optimized out>, closure@entry=0x0) at Python/ceval.c:3604
#141 0x0000fffffbdbdbfc in PyEval_EvalCode (co=co@entry=0xfffffb2fecb0, globals=globals@entry=0xfffffb8d5168, locals=locals@entry=0xfffffb8d5168) at Python/ceval.c:669
#142 0x0000fffffbdd80c0 in run_mod (mod=mod@entry=0xfffffab6b588, filename=filename@entry=0xffffffffd908 "/usr/pkgobj/www/firefox/work/firefox-65.0/configure.py", globals=globals@entry=0xfffffb8d5168, locals=locals@entry=0xfffffb8d5168, flags=flags@entry=0xffffffffce40, arena=arena@entry=0xfffffab09060) at Python/pythonrun.c:1385
#143 0x0000fffffbdd92f8 in PyRun_FileExFlags (fp=fp@entry=0xfffffbc25640, filename=filename@entry=0xffffffffd908 "/usr/pkgobj/www/firefox/work/firefox-65.0/configure.py", start=start@entry=257, globals=globals@entry=0xfffffb8d5168, locals=locals@entry=0xfffffb8d5168, closeit=closeit@entry=1, flags=flags@entry=0xffffffffce40) at Python/pythonrun.c:1371
#144 0x0000fffffbdda4ac in PyRun_SimpleFileExFlags (fp=fp@entry=0xfffffbc25640, filename=0xffffffffd908 "/usr/pkgobj/www/firefox/work/firefox-65.0/configure.py", closeit=closeit@entry=1, flags=0xffffffffce40, flags@entry=0xffffffffceb0) at Python/pythonrun.c:957
#145 0x0000fffffbdda9b8 in PyRun_AnyFileExFlags (fp=fp@entry=0xfffffbc25640, filename=<optimized out>, closeit=closeit@entry=1, flags=flags@entry=0xffffffffceb0) at Python/pythonrun.c:761
#146 0x0000fffffbdebd50 in Py_Main (argc=<optimized out>, argv=<optimized out>) at Modules/main.c:645
#147 0x0000000200100968 in ___start ()
#148 0x0000fffffbed0af0 in _rtld_start () from /usr/libexec/ld.elf_so
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
(gdb) up
#1  0x0000fffffbc39070 in pthread__mutex_spin (ptm=ptm@entry=0xfffffbc254f8, owner=owner@entry=0xfffff7113000) at /work/src/lib/libpthread/pthread_mutex.c:253
253	in /work/src/lib/libpthread/pthread_mutex.c
(gdb) p *ptm
$1 = {ptm_magic = 858980355, ptm_errorcheck = 0 '\000', {ptm_ceiling = 0 '\000', ptm_unused = 0 '\000'}, ptm_owner = 0xfffff7113000, ptm_waiters = 0x0, ptm_recursed = 0, ptm_spare2 = 0x0}
(gdb) up
#2  0x0000fffffbc393a4 in pthread__mutex_lock_slow (ptm=ptm@entry=0xfffffbc254f8, ts=ts@entry=0x0) at /work/src/lib/libpthread/pthread_mutex.c:343
343	in /work/src/lib/libpthread/pthread_mutex.c
(gdb) p base_mtx
$2 = {ptm_magic = 858980355, ptm_errorcheck = 0 '\000', {ptm_ceiling = 0 '\000', ptm_unused = 0 '\000'}, ptm_owner = 0xfffff7113000, ptm_waiters = 0x0, ptm_recursed = 0, ptm_spare2 = 0x0}
(gdb) info thread
  Id   Target Id         Frame 
  1    process 22382     pthread__mutex_pause () at /work/src/lib/libpthread/pthread_mutex.c:228
* 2    LWP 1 of process 22382 pthread__mutex_pause () at /work/src/lib/libpthread/pthread_mutex.c:228
(gdb) thread 1
[Switching to thread 1 (process 22382)]
#0  pthread__mutex_pause () at /work/src/lib/libpthread/pthread_mutex.c:228
228	in /work/src/lib/libpthread/pthread_mutex.c
(gdb) bt
#0  pthread__mutex_pause () at /work/src/lib/libpthread/pthread_mutex.c:228
#1  0x0000fffffbc39070 in pthread__mutex_spin (ptm=ptm@entry=0xfffffbc254f8, owner=owner@entry=0xfffff7113000) at /work/src/lib/libpthread/pthread_mutex.c:253
#2  0x0000fffffbc393a4 in pthread__mutex_lock_slow (ptm=ptm@entry=0xfffffbc254f8, ts=ts@entry=0x0) at /work/src/lib/libpthread/pthread_mutex.c:343
#3  0x0000fffffbc39998 in pthread_mutex_lock (ptm=ptm@entry=0xfffffbc254f8) at /work/src/lib/libpthread/pthread_mutex.c:202
#4  0x0000fffffbb4fd04 in base_chunk_node_alloc () at /work/src/lib/libc/stdlib/jemalloc.c:1167
#5  0x0000fffffbb4fd74 in huge_malloc (size=<optimized out>) at /work/src/lib/libc/stdlib/jemalloc.c:2795
#6  0x0000fffffbb50a44 in malloc (size=1048613) at /work/src/lib/libc/stdlib/jemalloc.c:3776
#7  0x0000fffffbd5bcc0 in PyObject_Malloc (nbytes=<optimized out>, nbytes@entry=1048613) at Objects/obmalloc.c:986
#8  0x0000fffffbd64b60 in PyString_FromStringAndSize (str=str@entry=0x0, size=1048576) at Objects/stringobject.c:88
#9  0x0000fffffbdf0f00 in posix_read (self=<optimized out>, args=<optimized out>) at ./Modules/posixmodule.c:6804
#10 0x0000fffffbdbaec8 in ext_do_call (nk=65535, na=-88950752, flags=<optimized out>, pp_stack=0xffffffff8b58, func=0xfffffb8fa3b0) at Python/ceval.c:4683
#11 PyEval_EvalFrameEx (f=f@entry=0xfffffb82b9d8, throwflag=throwflag@entry=0) at Python/ceval.c:3048
#12 0x0000fffffbdbd970 in PyEval_EvalCodeEx (co=0xfffffb826730, globals=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=65535, kws=<optimized out>, kwcount=kwcount@entry=0, defs=0x1, defcount=0, closure=<optimized out>) at Python/ceval.c:3604
#13 0x0000fffffbdbaaf8 in fast_function (nk=0, na=<optimized out>, n=3, pp_stack=0xffffffff8d40, func=0xfffffb52ade8) at Python/ceval.c:4467
#14 call_function (oparg=<optimized out>, pp_stack=0xffffffff8d40) at Python/ceval.c:4392
#15 PyEval_EvalFrameEx (f=f@entry=0xfffffab2b820, throwflag=throwflag@entry=0) at Python/ceval.c:3009
#16 0x0000fffffbdbd970 in PyEval_EvalCodeEx (co=0xfffffb83b7b0, globals=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=0, kws=<optimized out>, kwcount=kwcount@entry=0, defs=0xfffff8460ca0, defcount=0, closure=<optimized out>) at Python/ceval.c:3604
#17 0x0000fffffbdbaaf8 in fast_function (nk=0, na=<optimized out>, n=18, pp_stack=0xffffffff8f30, func=0xfffffb52d5f0) at Python/ceval.c:4467
#18 call_function (oparg=<optimized out>, pp_stack=0xffffffff8f30) at Python/ceval.c:4392
#19 PyEval_EvalFrameEx (f=f@entry=0xfffffab2b020, throwflag=throwflag@entry=0) at Python/ceval.c:3009
#20 0x0000fffffbdbd970 in PyEval_EvalCodeEx (co=0xfffffb826a30, globals=<optimized out>, locals=locals@entry=0x0, args=args@entry=0xfffff8ba5338, argcount=65535, kws=kws@entry=0xfffff8460ca0, kwcount=kwcount@entry=5, defs=0xfffff849a280, defs@entry=0xfffffb645428, defcount=defcount@entry=13, closure=<optimized out>) at Python/ceval.c:3604
#21 0x0000fffffbd418b4 in function_call (func=0xfffffb52d1b8, arg=0xfffff8ba5320, kw=0xfffff849a280) at Objects/funcobject.c:523
#22 0x0000fffffbd1a9cc in PyObject_Call (func=func@entry=0xfffffb52d1b8, arg=arg@entry=0xfffff8ba5320, kw=kw@entry=0xfffff849a280) at Objects/abstract.c:2547
#23 0x0000fffffbd2a0d4 in instancemethod_call (func=0xfffffb52d1b8, arg=0xfffff8ba5320, kw=0xfffff849a280) at Objects/classobject.c:2600
#24 0x0000fffffbd1a9cc in PyObject_Call (func=func@entry=0xfffff84dfeb0, arg=arg@entry=0xfffff8470810, kw=kw@entry=0xfffff849a280) at Objects/abstract.c:2547
#25 0x0000fffffbd778c8 in slot_tp_init (self=<optimized out>, args=0xfffff8470810, kwds=0xfffff849a280) at Objects/typeobject.c:5860
#26 0x0000fffffbd745c4 in type_call (type=<optimized out>, args=0xfffff8470810, kwds=0xfffff849a280) at Objects/typeobject.c:765
#27 0x0000fffffbd1a9cc in PyObject_Call (func=func@entry=0xfffffb76b020, arg=arg@entry=0xfffff8470810, kw=kw@entry=0xfffff849a280) at Objects/abstract.c:2547
#28 0x0000fffffbdb852c in do_call (nk=<optimized out>, na=<optimized out>, pp_stack=0xffffffff9520, func=0xfffffb76b020) at Python/ceval.c:4589
#29 call_function (oparg=<optimized out>, pp_stack=0xffffffff9520) at Python/ceval.c:4394
#30 PyEval_EvalFrameEx (f=f@entry=0xfffff87e2820, throwflag=throwflag@entry=0) at Python/ceval.c:3009
#31 0x0000fffffbdbcc7c in fast_function (nk=<optimized out>, na=<optimized out>, n=4, pp_stack=0xffffffff9670, func=0xfffff84d1398) at Python/ceval.c:4457
#32 call_function (oparg=<optimized out>, pp_stack=0xffffffff9670) at Python/ceval.c:4392
#33 PyEval_EvalFrameEx (f=f@entry=0xfffff87e5420, throwflag=throwflag@entry=0) at Python/ceval.c:3009
#34 0x0000fffffbdbd970 in PyEval_EvalCodeEx (co=0xfffff85b5b30, globals=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=65535, kws=<optimized out>, kwcount=kwcount@entry=0, defs=0x1, defcount=1, closure=<optimized out>) at Python/ceval.c:3604
#35 0x0000fffffbdbaaf8 in fast_function (nk=0, na=<optimized out>, n=5, pp_stack=0xffffffff9860, func=0xfffff84d1668) at Python/ceval.c:4467
#36 call_function (oparg=<optimized out>, pp_stack=0xffffffff9860) at Python/ceval.c:4392
#37 PyEval_EvalFrameEx (f=f@entry=0xfffff87ca020, throwflag=throwflag@entry=0) at Python/ceval.c:3009
#38 0x0000fffffbdbd970 in PyEval_EvalCodeEx (co=0xfffff85b5b30, globals=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=65535, kws=<optimized out>, kwcount=kwcount@entry=0, defs=0x1, defcount=1, closure=<optimized out>) at Python/ceval.c:3604
#39 0x0000fffffbdbaaf8 in fast_function (nk=0, na=<optimized out>, n=5, pp_stack=0xffffffff9a50, func=0xfffff84d1668) at Python/ceval.c:4467
#40 call_function (oparg=<optimized out>, pp_stack=0xffffffff9a50) at Python/ceval.c:4392
#41 PyEval_EvalFrameEx (f=f@entry=0xfffff87e2c20, throwflag=throwflag@entry=0) at Python/ceval.c:3009
#42 0x0000fffffbdbd970 in PyEval_EvalCodeEx (co=0xfffff85b5b30, globals=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=65535, kws=<optimized out>, kwcount=kwcount@entry=0, defs=0x1, defcount=1, closure=<optimized out>) at Python/ceval.c:3604
#43 0x0000fffffbdbaaf8 in fast_function (nk=0, na=<optimized out>, n=5, pp_stack=0xffffffff9c40, func=0xfffff84d1668) at Python/ceval.c:4467
#44 call_function (oparg=<optimized out>, pp_stack=0xffffffff9c40) at Python/ceval.c:4392
#45 PyEval_EvalFrameEx (f=f@entry=0xfffff87b1420, throwflag=throwflag@entry=0) at Python/ceval.c:3009
#46 0x0000fffffbdbd970 in PyEval_EvalCodeEx (co=0xfffff85b5b30, globals=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=65535, kws=<optimized out>, kwcount=kwcount@entry=0, defs=0xfffffb905090, defcount=1, closure=<optimized out>) at Python/ceval.c:3604
#47 0x0000fffffbdbaaf8 in fast_function (nk=0, na=<optimized out>, n=4, pp_stack=0xffffffff9e30, func=0xfffff84d1668) at Python/ceval.c:4467
#48 call_function (oparg=<optimized out>, pp_stack=0xffffffff9e30) at Python/ceval.c:4392
#49 PyEval_EvalFrameEx (f=f@entry=0xfffff8798820, throwflag=throwflag@entry=0) at Python/ceval.c:3009
#50 0x0000fffffbdbcc7c in fast_function (nk=<optimized out>, na=<optimized out>, n=8, pp_stack=0xffffffff9f80, func=0xfffff84cdf50) at Python/ceval.c:4457
#51 call_function (oparg=<optimized out>, pp_stack=0xffffffff9f80) at Python/ceval.c:4392
#52 PyEval_EvalFrameEx (f=f@entry=0xfffff8798020, throwflag=throwflag@entry=0) at Python/ceval.c:3009
#53 0x0000fffffbdbcc7c in fast_function (nk=<optimized out>, na=<optimized out>, n=10, pp_stack=0xffffffffa0d0, func=0xfffff84d4aa0) at Python/ceval.c:4457
#54 call_function (oparg=<optimized out>, pp_stack=0xffffffffa0d0) at Python/ceval.c:4392
#55 PyEval_EvalFrameEx (f=f@entry=0xfffff8791c20, throwflag=throwflag@entry=0) at Python/ceval.c:3009
#56 0x0000fffffbdbd970 in PyEval_EvalCodeEx (co=0xfffff85adbb0, globals=<optimized out>, locals=locals@entry=0x0, args=args@entry=0xfffff846d4e0, argcount=65535, kws=kws@entry=0x0, kwcount=kwcount@entry=0, defs=0x6, defs@entry=0xfffff856c458, defcount=defcount@entry=7, closure=<optimized out>) at Python/ceval.c:3604
#57 0x0000fffffbd417d0 in function_call (func=0xfffff881f2a8, arg=0xfffff846d4c8, kw=0x0) at Objects/funcobject.c:523
#58 0x0000fffffbd1a9cc in PyObject_Call (func=func@entry=0xfffff881f2a8, arg=arg@entry=0xfffff846d4c8, kw=kw@entry=0x0) at Objects/abstract.c:2547
#59 0x0000fffffbdb6ce0 in ext_do_call (nk=65535, na=-23664, flags=<optimized out>, pp_stack=0xffffffffa388, func=0xfffff881f2a8) at Python/ceval.c:4686
#60 PyEval_EvalFrameEx (f=f@entry=0xfffff8ac8bc0, throwflag=throwflag@entry=0) at Python/ceval.c:3048
#61 0x0000fffffbdbd970 in PyEval_EvalCodeEx (co=0xfffff85ad1b0, globals=<optimized out>, locals=locals@entry=0x0, args=args@entry=0xfffff846d2d8, argcount=65535, kws=kws@entry=0xfffffba10068, kwcount=kwcount@entry=0, defs=0xfffff90fbdc4, defs@entry=0x0, defcount=defcount@entry=0, closure=<optimized out>) at Python/ceval.c:3604
#62 0x0000fffffbd418b4 in function_call (func=0xfffff8472b18, arg=0xfffff846d2c0, kw=0xfffff847c398) at Objects/funcobject.c:523
#63 0x0000fffffbd1a9cc in PyObject_Call (func=func@entry=0xfffff8472b18, arg=arg@entry=0xfffff846d2c0, kw=kw@entry=0x0) at Objects/abstract.c:2547
#64 0x0000fffffbdb6ce0 in ext_do_call (nk=65535, na=-22976, flags=<optimized out>, pp_stack=0xffffffffa638, func=0xfffff8472b18) at Python/ceval.c:4686
#65 PyEval_EvalFrameEx (f=f@entry=0xfffff8478620, throwflag=throwflag@entry=0) at Python/ceval.c:3048
#66 0x0000fffffbdbd970 in PyEval_EvalCodeEx (co=0xfffff9010db0, globals=<optimized out>, locals=locals@entry=0x0, args=args@entry=0xfffff8cbba40, argcount=65535, kws=kws@entry=0xfffffba10068, kwcount=kwcount@entry=0, defs=0xfffff96d12f2, defs@entry=0x0, defcount=defcount@entry=0, closure=<optimized out>) at Python/ceval.c:3604
#67 0x0000fffffbd418b4 in function_call (func=0xfffff901bed8, arg=0xfffff8cbba28, kw=0xfffff8479168) at Objects/funcobject.c:523
#68 0x0000fffffbd1a9cc in PyObject_Call (func=func@entry=0xfffff901bed8, arg=arg@entry=0xfffff8cbba28, kw=kw@entry=0x1) at Objects/abstract.c:2547
#69 0x0000fffffbdb6ce0 in ext_do_call (nk=65535, na=-117282466, flags=<optimized out>, pp_stack=0xffffffffa8e8, func=0xfffff901bed8) at Python/ceval.c:4686
#70 PyEval_EvalFrameEx (f=f@entry=0xfffff8477ad0, throwflag=throwflag@entry=0) at Python/ceval.c:3048
#71 0x0000fffffbdbcc7c in fast_function (nk=<optimized out>, na=<optimized out>, n=1, pp_stack=0xffffffffaa30, func=0xfffff96ee848) at Python/ceval.c:4457
#72 call_function (oparg=<optimized out>, pp_stack=0xffffffffaa30) at Python/ceval.c:4392
#73 PyEval_EvalFrameEx (f=f@entry=0xfffff897de18, throwflag=throwflag@entry=0) at Python/ceval.c:3009
#74 0x0000fffffbdbcc7c in fast_function (nk=<optimized out>, na=<optimized out>, n=1, pp_stack=0xffffffffab80, func=0xfffff96eeed8) at Python/ceval.c:4457
#75 call_function (oparg=<optimized out>, pp_stack=0xffffffffab80) at Python/ceval.c:4392
#76 PyEval_EvalFrameEx (f=f@entry=0xfffff8631220, throwflag=throwflag@entry=0) at Python/ceval.c:3009
#77 0x0000fffffbdbd970 in PyEval_EvalCodeEx (co=0xfffff8e0f630, globals=<optimized out>, locals=locals@entry=0x0, args=args@entry=0xfffff8cbe800, argcount=65535, kws=kws@entry=0x0, kwcount=kwcount@entry=0, defs=0xfffffbe76000 <_PyImport_DynLoadFiletab>, defs@entry=0x0, defcount=defcount@entry=0, closure=<optimized out>) at Python/ceval.c:3604
#78 0x0000fffffbd417d0 in function_call (func=0xfffff8e0e8c0, arg=0xfffff8cbe7e8, kw=0x0) at Objects/funcobject.c:523
#79 0x0000fffffbd1a9cc in PyObject_Call (func=func@entry=0xfffff8e0e8c0, arg=arg@entry=0xfffff8cbe7e8, kw=kw@entry=0x0) at Objects/abstract.c:2547
#80 0x0000fffffbd2a0d4 in instancemethod_call (func=0xfffff8e0e8c0, arg=0xfffff8cbe7e8, kw=0x0) at Objects/classobject.c:2600
#81 0x0000fffffbd1a9cc in PyObject_Call (func=func@entry=0xfffff8e37500, arg=arg@entry=0xfffff8638090, kw=kw@entry=0x0) at Objects/abstract.c:2547
#82 0x0000fffffbd778c8 in slot_tp_init (self=<optimized out>, args=0xfffff8638090, kwds=0x0) at Objects/typeobject.c:5860
#83 0x0000fffffbd745c4 in type_call (type=<optimized out>, args=0xfffff8638090, kwds=0x0) at Objects/typeobject.c:765
#84 0x0000fffffbd1a9cc in PyObject_Call (func=func@entry=0xfffff8d86c20, arg=arg@entry=0xfffff8638090, kw=kw@entry=0x0) at Objects/abstract.c:2547
#85 0x0000fffffbdb852c in do_call (nk=<optimized out>, na=<optimized out>, pp_stack=0xffffffffb170, func=0xfffff8d86c20) at Python/ceval.c:4589
#86 call_function (oparg=<optimized out>, pp_stack=0xffffffffb170) at Python/ceval.c:4394
#87 PyEval_EvalFrameEx (f=f@entry=0xfffff8477210, throwflag=throwflag@entry=0) at Python/ceval.c:3009
#88 0x0000fffffbdbcc7c in fast_function (nk=<optimized out>, na=<optimized out>, n=1, pp_stack=0xffffffffb2c0, func=0xfffff96ee8c0) at Python/ceval.c:4457
#89 call_function (oparg=<optimized out>, pp_stack=0xffffffffb2c0) at Python/ceval.c:4392
#90 PyEval_EvalFrameEx (f=f@entry=0xfffff8478240, throwflag=throwflag@entry=0) at Python/ceval.c:3009
#91 0x0000fffffbdbcc7c in fast_function (nk=<optimized out>, na=<optimized out>, n=1, pp_stack=0xffffffffb410, func=0xfffff901d410) at Python/ceval.c:4457
#92 call_function (oparg=<optimized out>, pp_stack=0xffffffffb410) at Python/ceval.c:4392
#93 PyEval_EvalFrameEx (f=f@entry=0xfffff95bf020, throwflag=throwflag@entry=0) at Python/ceval.c:3009
#94 0x0000fffffbdbd970 in PyEval_EvalCodeEx (co=0xfffff90152b0, globals=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=0, kws=<optimized out>, kwcount=kwcount@entry=0, defs=0x0, defcount=0, closure=<optimized out>) at Python/ceval.c:3604
#95 0x0000fffffbdbaaf8 in fast_function (nk=0, na=<optimized out>, n=8, pp_stack=0xffffffffb600, func=0xfffff901d488) at Python/ceval.c:4467
#96 call_function (oparg=<optimized out>, pp_stack=0xffffffffb600) at Python/ceval.c:4392
#97 PyEval_EvalFrameEx (f=f@entry=0xfffff9552c20, throwflag=throwflag@entry=0) at Python/ceval.c:3009
#98 0x0000fffffbdbd970 in PyEval_EvalCodeEx (co=0xfffff85ad4b0, globals=<optimized out>, locals=locals@entry=0x0, args=args@entry=0xfffff84d6518, argcount=65535, kws=kws@entry=0x0, kwcount=kwcount@entry=0, defs=0x1, defs@entry=0x0, defcount=defcount@entry=0, closure=<optimized out>) at Python/ceval.c:3604
#99 0x0000fffffbd417d0 in function_call (func=0xfffff8472c08, arg=0xfffff84d6500, kw=0x0) at Objects/funcobject.c:523
#100 0x0000fffffbd1a9cc in PyObject_Call (func=func@entry=0xfffff8472c08, arg=arg@entry=0xfffff84d6500, kw=kw@entry=0x0) at Objects/abstract.c:2547
#101 0x0000fffffbd2a0d4 in instancemethod_call (func=0xfffff8472c08, arg=0xfffff84d6500, kw=0x0) at Objects/classobject.c:2600
#102 0x0000fffffbd1a9cc in PyObject_Call (func=func@entry=0xfffff8bc2e60, arg=arg@entry=0xfffff856ce50, kw=kw@entry=0x0) at Objects/abstract.c:2547
#103 0x0000fffffbd778c8 in slot_tp_init (self=<optimized out>, args=0xfffff856ce50, kwds=0x0) at Objects/typeobject.c:5860
#104 0x0000fffffbd745c4 in type_call (type=<optimized out>, args=0xfffff856ce50, kwds=0x0) at Objects/typeobject.c:765
#105 0x0000fffffbd1a9cc in PyObject_Call (func=func@entry=0xfffff9593420, arg=arg@entry=0xfffff856ce50, kw=kw@entry=0x0) at Objects/abstract.c:2547
#106 0x0000fffffbdb852c in do_call (nk=<optimized out>, na=<optimized out>, pp_stack=0xffffffffbbf0, func=0xfffff9593420) at Python/ceval.c:4589
#107 call_function (oparg=<optimized out>, pp_stack=0xffffffffbbf0) at Python/ceval.c:4394
#108 PyEval_EvalFrameEx (f=f@entry=0xfffff873e020, throwflag=<optimized out>) at Python/ceval.c:3009
#109 0x0000fffffbd36dc8 in gen_send_ex (arg=0x0, exc=<optimized out>, gen=<optimized out>, gen=<optimized out>) at Objects/genobject.c:85
#110 0x0000fffffbdb94e8 in PyEval_EvalFrameEx (f=f@entry=0xfffff8791820, throwflag=<optimized out>) at Python/ceval.c:2828
#111 0x0000fffffbd36dc8 in gen_send_ex (arg=0x0, exc=<optimized out>, gen=<optimized out>, gen=<optimized out>) at Objects/genobject.c:85
#112 0x0000fffffbdb94e8 in PyEval_EvalFrameEx (f=f@entry=0xfffff8d93c20, throwflag=<optimized out>) at Python/ceval.c:2828
#113 0x0000fffffbd36dc8 in gen_send_ex (arg=0x0, exc=<optimized out>, gen=<optimized out>, gen=<optimized out>) at Objects/genobject.c:85
#114 0x0000fffffbdb94e8 in PyEval_EvalFrameEx (f=f@entry=0xfffff8d93820, throwflag=<optimized out>) at Python/ceval.c:2828
#115 0x0000fffffbd36dc8 in gen_send_ex (arg=0x0, exc=<optimized out>, gen=<optimized out>, gen=<optimized out>) at Objects/genobject.c:85
#116 0x0000fffffbdb94e8 in PyEval_EvalFrameEx (f=f@entry=0xfffff8e95de0, throwflag=<optimized out>) at Python/ceval.c:2828
#117 0x0000fffffbd36dc8 in gen_send_ex (arg=0x0, exc=<optimized out>, gen=<optimized out>, gen=<optimized out>) at Objects/genobject.c:85
#118 0x0000fffffbdb94e8 in PyEval_EvalFrameEx (f=f@entry=0xfffff8d92c20, throwflag=<optimized out>) at Python/ceval.c:2828
#119 0x0000fffffbd36dc8 in gen_send_ex (arg=0x0, exc=<optimized out>, gen=<optimized out>, gen=<optimized out>) at Objects/genobject.c:85
#120 0x0000fffffbd4694c in listextend (self=self@entry=0xfffff8e1c710, b=<optimized out>) at Objects/listobject.c:872
#121 0x0000fffffbd46b74 in list_init (self=0xfffff8e1c710, args=<optimized out>, kw=<optimized out>) at Objects/listobject.c:2459
#122 0x0000fffffbd745c4 in type_call (type=<optimized out>, args=0xfffffb8526d0, kwds=0x0) at Objects/typeobject.c:765
#123 0x0000fffffbd1a9cc in PyObject_Call (func=func@entry=0xfffffbe89158 <PyList_Type>, arg=arg@entry=0xfffffb8526d0, kw=kw@entry=0x0) at Objects/abstract.c:2547
#124 0x0000fffffbdb852c in do_call (nk=<optimized out>, na=<optimized out>, pp_stack=0xffffffffc630, func=0xfffffbe89158 <PyList_Type>) at Python/ceval.c:4589
#125 call_function (oparg=<optimized out>, pp_stack=0xffffffffc630) at Python/ceval.c:4394
#126 PyEval_EvalFrameEx (f=f@entry=0xfffff8d13420, throwflag=throwflag@entry=0) at Python/ceval.c:3009
#127 0x0000fffffbdbd970 in PyEval_EvalCodeEx (co=0xfffff9335eb0, globals=<optimized out>, locals=locals@entry=0x0, args=args@entry=0xfffffba10068, argcount=65535, kws=kws@entry=0xfffff91dbf80, kwcount=kwcount@entry=7, defs=0xfffffab70e8d, defs@entry=0xfffff91b51d0, defcount=defcount@entry=8, closure=<optimized out>) at Python/ceval.c:3604
#128 0x0000fffffbd418b4 in function_call (func=0xfffff8e8e6e0, arg=0xfffffba10050, kw=0xfffff8e945c8) at Objects/funcobject.c:523
#129 0x0000fffffbd1a9cc in PyObject_Call (func=func@entry=0xfffff8e8e6e0, arg=arg@entry=0xfffffba10050, kw=kw@entry=0xfffffac5aed8) at Objects/abstract.c:2547
#130 0x0000fffffbdb6ce0 in ext_do_call (nk=65535, na=-74428272, flags=<optimized out>, pp_stack=0xffffffffc8e8, func=0xfffff8e8e6e0) at Python/ceval.c:4686
#131 PyEval_EvalFrameEx (f=f@entry=0xfffff9516c20, throwflag=throwflag@entry=0) at Python/ceval.c:3048
#132 0x0000fffffbdbd970 in PyEval_EvalCodeEx (co=0xfffffb2feb30, globals=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=0, kws=<optimized out>, kwcount=kwcount@entry=0, defs=0x1, defcount=0, closure=<optimized out>) at Python/ceval.c:3604
#133 0x0000fffffbdbaaf8 in fast_function (nk=0, na=<optimized out>, n=1, pp_stack=0xffffffffcad0, func=0xfffffac5aed8) at Python/ceval.c:4467
#134 call_function (oparg=<optimized out>, pp_stack=0xffffffffcad0) at Python/ceval.c:4392
#135 PyEval_EvalFrameEx (f=f@entry=0xfffffaa2d820, throwflag=throwflag@entry=0) at Python/ceval.c:3009
#136 0x0000fffffbdbd970 in PyEval_EvalCodeEx (co=0xfffffb2ed3b0, globals=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=65535, kws=<optimized out>, kwcount=kwcount@entry=0, defs=0xfffffb8d5168, defcount=0, closure=<optimized out>) at Python/ceval.c:3604
#137 0x0000fffffbdbaaf8 in fast_function (nk=0, na=<optimized out>, n=1, pp_stack=0xffffffffccc0, func=0xfffffac5a938) at Python/ceval.c:4467
#138 call_function (oparg=<optimized out>, pp_stack=0xffffffffccc0) at Python/ceval.c:4392
#139 PyEval_EvalFrameEx (f=f@entry=0xfffffaeac960, throwflag=throwflag@entry=0) at Python/ceval.c:3009
#140 0x0000fffffbdbd970 in PyEval_EvalCodeEx (co=co@entry=0xfffffb2fecb0, globals=globals@entry=0xfffffb8d5168, locals=locals@entry=0xfffffb8d5168, args=args@entry=0x0, argcount=argcount@entry=0, kws=kws@entry=0x0, kwcount=kwcount@entry=0, defs=0xfffffbdda4ac <PyRun_SimpleFileExFlags+220>, defs@entry=0x0, defcount=defcount@entry=0, closure=<optimized out>, closure@entry=0x0) at Python/ceval.c:3604
#141 0x0000fffffbdbdbfc in PyEval_EvalCode (co=co@entry=0xfffffb2fecb0, globals=globals@entry=0xfffffb8d5168, locals=locals@entry=0xfffffb8d5168) at Python/ceval.c:669
#142 0x0000fffffbdd80c0 in run_mod (mod=mod@entry=0xfffffab6b588, filename=filename@entry=0xffffffffd908 "/usr/pkgobj/www/firefox/work/firefox-65.0/configure.py", globals=globals@entry=0xfffffb8d5168, locals=locals@entry=0xfffffb8d5168, flags=flags@entry=0xffffffffce40, arena=arena@entry=0xfffffab09060) at Python/pythonrun.c:1385
#143 0x0000fffffbdd92f8 in PyRun_FileExFlags (fp=fp@entry=0xfffffbc25640, filename=filename@entry=0xffffffffd908 "/usr/pkgobj/www/firefox/work/firefox-65.0/configure.py", start=start@entry=257, globals=globals@entry=0xfffffb8d5168, locals=locals@entry=0xfffffb8d5168, closeit=closeit@entry=1, flags=flags@entry=0xffffffffce40) at Python/pythonrun.c:1371
#144 0x0000fffffbdda4ac in PyRun_SimpleFileExFlags (fp=fp@entry=0xfffffbc25640, filename=0xffffffffd908 "/usr/pkgobj/www/firefox/work/firefox-65.0/configure.py", closeit=closeit@entry=1, flags=0xffffffffce40, flags@entry=0xffffffffceb0) at Python/pythonrun.c:957
#145 0x0000fffffbdda9b8 in PyRun_AnyFileExFlags (fp=fp@entry=0xfffffbc25640, filename=<optimized out>, closeit=closeit@entry=1, flags=flags@entry=0xffffffffceb0) at Python/pythonrun.c:761
#146 0x0000fffffbdebd50 in Py_Main (argc=<optimized out>, argv=<optimized out>) at Modules/main.c:645
#147 0x0000000200100968 in ___start ()
#148 0x0000fffffbed0af0 in _rtld_start () from /usr/libexec/ld.elf_so


>How-To-Repeat:

Try to build firefox from pkgsrc.

>Fix:
n/a