slopfront
preferences

Achieve true parallelism in Python 3.12

209 points · submitted Apr 2024 by u/ThatsAHumanPerson · Resource · archived

Article link: https://rishiraj.me/articles/2024-04/python_subinterpreter_parallelism

I have written an article, which should be helpful to folks at all experience levels, covering various multi-tasking paradigms in computers, and how they apply in CPython, with its unique limitations like the Global Interpreter Lock. Using this knowledge, we look at traditional ways to achieve "true parallelism" (i.e. multiple tasks running at the same time) in Python.

Finally, we build a solution utilizing newer concepts in Python 3.12 to run any arbitrary pure Python code in parallel across multiple threads. All the code used to achieve this, along with the benchmarking code are available in the repository linked in the blog-post.

This is my first time writing a technical post in Python. Any feedback would be really appreciated! 😊

31 comments · 95% upvoted · other discussions

comments

ryanstephendavis 43 points · Apr 2024

Well-written article... good refresher on a lot of software engineering concepts and it's cool to see some code with the new subinterpreters!

Mount_Gamer 11 points · Apr 2024

Any idea if the sub interpreters will use less memory than the multiprocessing method?

ThatsAHumanPerson [OP] 3 points · Apr 2024

There are some memory leaks associated with sub-interpreters in the current Python implementation. Refer https://github.com/python/cpython/issues/110411. These are expected to be fixed soon, so, I didn't cover these points.

Mount_Gamer 1 points · Apr 2024

A good find, look forward to trying.

andrejlr 11 points · Apr 2024

Nice summary. Was looking forward to subibinterpreters. This article shows though that they have and probably for a long time will have too many limitations.

Practically all production code in my experience is using some native extensions . And after reading this I just don't want to start new experiments for parallel code which is already working .

Often, starting new processes is not such a big penalty, as each of them might run hours of computation and consume magnitudes more memory than the pure Python process itself . So one can easily afford it .

The real overhead within the default process.Pool lib is the need to serialize ( pickle ) objects between the main process and worker processes .

In our experienc, this is where Rey is unbeatable for single machine parallelism . It will give you even same process.Pool.map interface as a one-line replacement, if you want . But under the hood, it is using pyarrow memory format and also a pyarrow store allowing multiple Proccesses to do a direct memory access without serialization

120decibel 11 points · Apr 2024

You might want to cover multiprocessing a bit more in dept as it site-steps the GIL through subprocesses as well. Otherwise great write up, I enjoyed reading it a lot! Thanks!

ThatsAHumanPerson [OP] 2 points · Apr 2024

Added. Thanks!

[deleted] 4 points · Apr 2024

Nice to see. In Py 3.13 this should be implementable in pure python, hopefully

protienbudspromax 6 points · Apr 2024

Nice one OP. I knew about that GIL pep but wasnt sure what they implemented in place of that. Was a good read!

Glass_Dingo_3984 -3 points · Apr 2024

Thanks very good idea 💡 👍 😀 😊 👏 👌

ThatsAHumanPerson [OP] 3 points · Apr 2024

Really appreciate all the positive response here!

I've made a few changes incorporating the common feedback points:-

  1. Briefly covered multiprocessing module
  2. Clarified that the performance improvement seen with sub-interpreters isn't conclusive as I've used a very simplistic benchmark.
JayZFeelsBad4Me 3 points · Apr 2024

Side note - your website looks great but any plans to add dark mode?

[deleted] 2 points · Apr 2024

Great write up though I'd really recommend limiting the text width of your text content. Currently it's very tought to read 200+ character lines on desktop. You can do that with css style max-width: 70ch (for 70character limit)

ThatsAHumanPerson [OP] 1 points · Apr 2024

Added max-width for better legibility on bigger screens.

bajadrix 4 points · Apr 2024

You can obtain true parallelism in pure Python with multiprocess approach by using multiprocessing module.

Smallpaul 5 points · Apr 2024

Yeah, that's what the blog says:

"The simplest way to achieve parallelism in Python is using multiprocessing module, which spawns multiple separate Python processes, with some sort of inter-process communication to the parent process. Since spawning a process has some overhead (and isn’t very interesting), so, for the purpose of this article, we’ll limit our discussion to what we can achieve using a single Python process."

sdmike21 1 points · Apr 2024

The IPC overhead of multiprocessing is non-trivial and makes it fairly unsuitable for certain types of work. Being able to use shared memory within a single process would make those types of tasks significantly simpler and more performant than a comparable implementation in multiprocessing even if you could use multiprocessing.shared_memory.SharedMemory in the exact same same way that shared memory between "true" threads works.

Glass_Dingo_3984 -2 points · Apr 2024

I don't know why it's working but it's working

WhiteGoldRing 4 points · Apr 2024

Why not use the multiprocessing nodule?

Goobyalus 13 points · Apr 2024

Conclusion

  1. Sub-interpreters seem like a promising mechanism for parallel Python code with significant advantages (>20% performance improvement over multiprocessing), etc.

...

WhiteGoldRing 5 points · Apr 2024

Tested using only one, extremely specific for loop. That statement needs more evidence to support it.

ThatsAHumanPerson [OP] 6 points · Apr 2024

Updated to briefly cover multiprocessing module, and also the caveat about the benchmark logic being simplistic.

[deleted] 0 points · Apr 2024

[deleted]

WhiteGoldRing 4 points · Apr 2024

It's not crap. It's just not meant to be used like for your specific use case.

Glass_Dingo_3984 -10 points · Apr 2024

I don't know but maybe that's because he are a lady programmer, if u don't lasy

barath_chinmala 1 points · Apr 2024

Truly informative and a well written article man. Keep up the good work

Jesus-was-a-mushroom 1 points · Apr 2024

Link broken?

ThatsAHumanPerson [OP] 1 points · Apr 2024

Link broken

Seems fine to me. Can you recheck?

worriedjacket -47 points · Apr 2024

Insane to use C++ instead of rust

E-woke 16 points · Apr 2024

Bruh

Glass_Dingo_3984 -2 points · Apr 2024

Glass_Dingo_3984 -6 points · Apr 2024

Ow yes if you us turtle