Python & multiprocessing: Fork'ed !
2025-11-22 , Breakout Room

This talk will explore the various ways of spawning processes in Python (subprocess, multiprocessing, concurrent.futures, fork, vfork, posix_spawn, etc.) and how those implementations inside Python have been leading to regular dead-locks and hard crashes for many years in production environments.


This talk will introduce the multiple different ways of spawning sub-processes in Python with simple & easy to understand examples. We will then have a brief look at the different implementations and their various use-cases.

Then we will show how spawning processes in Python has always been fraught with problems for over 15 years. We will show real-world examples of seemingly unexplainable dead-locks and hard crashes when spawning processes in Python, especially in the context of multi-threaded applications.

Lastly, we explain the reasons for those hard-to-debug dead-locks & hard crashes, show the problems that are associated to this day with using os.fork() & os.register_at_fork(). Then we will offer up how alternatives like os.posix_spawn() + vfork can help to make your Python programs more robust.


What is the anticipated audience for your presentation?:

Intermediate

Teijo Holzer has been working in the Visual Effects industry for WetaFX in Wellington, New Zealand for over 15 years on feature films like Avatar & The Hobbit. He has over 15 years of professional Python experience, and has been instrumental in discovering extremely difficult bugs in Python & associated libraries (e.g. PySide), providing reproducible test cases and suggestions for fixes.