Re: multithreaded python: can't start new thread when building calibre

[Thomas Klausner <wiz%NetBSD.org@localhost>]

On Thu, Feb 26, 2015 at 02:43:53PM +0100, Thomas Klausner wrote:
> When building calibre-2.x on NetBSD, I always had the problem that
> building failed because:
> 
> ####### Building extension magick #######
> Compiling magick
> Traceback (most recent call last):
>   File "setup.py", line 99, in <module>
>     sys.exit(main())
>   File "setup.py", line 85, in main
>     command.run_all(opts)
>   File "/scratch/wip/calibre/work/calibre-2.15.0/setup/__init__.py", line 181, in run_all
>     self.run_cmd(self, opts)
>   File "/scratch/wip/calibre/work/calibre-2.15.0/setup/__init__.py", line 178, in run_cmd
>     cmd.run(opts)
>   File "/scratch/wip/calibre/work/calibre-2.15.0/setup/extensions.py", line 412, in run
>     self.build(ext, dest)
>   File "/scratch/wip/calibre/work/calibre-2.15.0/setup/extensions.py", line 456, in build
>     if not parallel_build(jobs, self.info):
>   File "/scratch/wip/calibre/work/calibre-2.15.0/setup/parallel_build.py", line 32, in parallel_build
>     p = Pool(cpu_count)
>   File "/usr/pkg/lib/python2.7/multiprocessing/dummy/__init__.py", line 151, in Pool
>     return ThreadPool(processes, initializer, initargs)
>   File "/usr/pkg/lib/python2.7/multiprocessing/pool.py", line 718, in __init__
>     Pool.__init__(self, processes, initializer, initargs)
>   File "/usr/pkg/lib/python2.7/multiprocessing/pool.py", line 159, in __init__
>     self._repopulate_pool()
>   File "/usr/pkg/lib/python2.7/multiprocessing/pool.py", line 223, in _repopulate_pool
>     w.start()
>   File "/usr/pkg/lib/python2.7/multiprocessing/dummy/__init__.py", line 75, in start
>     threading.Thread.start(self)
>   File "/usr/pkg/lib/python2.7/threading.py", line 745, in start
>     _start_new_thread(self.__bootstrap, ())
> thread.error: can't start new thread
>  *** Error code 1
> 
> The same chunk works fine in the interactive shell, not sure why it so
> reliably breaks here.
> 
> http://debian.2.n7.nabble.com/Bug-760865-calibre-FTBFS-on-mips-thread-error-can-t-start-new-thread-td3353306.html
> 
> pointed me to the workaround of hardcoding the number of CPUs to one:
> 
> -    from multiprocessing import cpu_count
> -    try:
> -        cpu_count = cpu_count()
> -    except NotImplementedError:
> -        cpu_count = 1
> +    cpu_count = 1
> 
> now calibre builds fine for me.
> 
> I wonder why it's necessary though. Does anyone have an idea? Is there
> a problem in python + threading on NetBSD, or does it run into some
> resource issues? But why so consistently in the (pbulk and manual)
> build, but not when I execute the cpu_count() function in an
> interactive shell?

joerg asked if it helps if I increase the thread limit in the shell,
and it does.

When I change the thread ulimit from the default 160 to 500, calibre
builds fine without this patch.

Now I wonder who eats all the threads :)

pkgsrc does not currently support increasing the thread limit, should
we add support for this?
 Thomas