Re: atexit(), dlclose() and more atexit()

[Joerg Sonnenberger <joerg%bec.de@localhost>]

On Sun, Jun 28, 2020 at 10:56:01PM +0200, Rhialto wrote:
> On Sun 28 Jun 2020 at 22:39:28 +0200, Joerg Sonnenberger wrote:
> > On Sun, Jun 28, 2020 at 10:35:27PM +0200, Rhialto wrote:
> > > I have at hand a program (the current svn trunk of VICE, to be exact)
> > > which does the following:
> > > 
> > > 1. In the original thread, it dlopen()s libavformat.
> > > 2. libavformat establishes an atexit() handler.
> > > 3. The main thread starts a new thread, and registers an atexit()
> > >    handler to clean up that thread.
> > > 4. main thread exit()s.
> > > 5. atexit() handler obtains its lock, and calls the handler established in 3.
> > > 6. said handler tells the new thread to clean up and finish. One of the last
> > >    things the thread does, is to dlclose() libavformat.
> > > 7. libavformat's atexit handling gets called, which tries to obtain the
> > >    same lock that was already obtained, in a different thread, in step
> > >    5.
> > > 8. deadlock.
> > > 
> > > Who is in the wrong here?
> > 
> > libavformat. Never use atexit() with a handler in a library that can be
> > closed.
> 
> The funny thing is that libavformat uses an atexit handler due to issues
> with dynamic (un)loading (or so they claim). This is from their file
> ffmpeg-4.2.2/libavformat/avisynth.c:
> 
> /* A conflict between C++ global objects, atexit, and dynamic loading requires
>  * us to register our own atexit handler to prevent double freeing. */

It is fundamentally wrong to use a handler in a library that can be
unloaded. Some systems hack around that problem by looping over all
atexit handlers on dlclose, but that's exactly that, a costly hack. The
most common way this triggers is a segfault, actually.

The code should be using either a C dtor with the appropiate attribute
or __cxa_atexit directly, but the former is preferable.

Joerg