1 million Python processes will use something like 5TB of memory; threads are evil in Python because the GIL prevents them from being useful in most cases.
You will have race conditions with multiprocessing as well. Further, multiprocessing is flaky and slow in Python, much worse than threading in other languages. Threads are performant partially because they have access to the same address space, but to do this correctly you need locks (there are exceptions, but they carry their own caveats). I don't know about the links you posted, but threads are widely used, and rightly so.
You're going to have to elaborate on multiprocessing being "flaky and slow" in Python; that's not my experience and I'm not sure what attributes of Python would render it so.
I don't know about the links you posted, but threads are widely used, and
rightly so.
"Threads are evil" derives from a research paper from U of C Berkeley 10 years ago....
They're widely used because they're often the only solution a programmer knows and, as Mark Summerfield noted, most languages don't offer any alternatives. We've seen functional languages with immutable state rise up on the one hand and message passing (such as Erlang) on the other to offer alternatives to the many problems inherent in multithreaded programming.
They're flaky because a process will sometimes mysteriously be killed or stall indefinitely. They are slow because all communication between processes must be serialized on the sender and deserialized on the receiver, because processes don't support a shared memory space. Further, as I mentioned elsewhere, each process requires its own interpreter, which takes 5MB, so you won't be scaling these processes.
I don't dispute that there are people out there who think threads are evil, I dispute that this belief is widely held, or that this belief isn't widely held because no one knows about processes. Experienced developers still prefer threads to processes. Processes are not a secret; they're lesser used because they are more memory intensive and intercommunication is horribly slow.
What's wrong with spawning a million processes? I thought threads were evil.
weberc2 9 points · Jan 2017
1 million Python processes will use something like 5TB of memory; threads are evil in Python because the GIL prevents them from being useful in most cases.
alcalde 1 points · Jan 2017
Threads are considered evil in general, not just in Python....
http://stackoverflow.com/questions/1191553/why-might-threads-be-considered-evil
https://thesynchronousblog.wordpress.com/2008/08/16/threads-are-evil/
It's the locking and race conditions and other problems of threading that make them evil; it has nothing to do with a GIL.
As Guido pointed out, threads were never intended for parallel computation.
https://youtu.be/EBRMq2Ioxsc?t=33m50s
weberc2 1 points · Jan 2017
You will have race conditions with multiprocessing as well. Further, multiprocessing is flaky and slow in Python, much worse than threading in other languages. Threads are performant partially because they have access to the same address space, but to do this correctly you need locks (there are exceptions, but they carry their own caveats). I don't know about the links you posted, but threads are widely used, and rightly so.
alcalde 1 points · Jan 2017
You're going to have to elaborate on multiprocessing being "flaky and slow" in Python; that's not my experience and I'm not sure what attributes of Python would render it so.
"Threads are evil" derives from a research paper from U of C Berkeley 10 years ago....
https://www2.eecs.berkeley.edu/Pubs/TechRpts/2006/EECS-2006-1.pdf
They're widely used because they're often the only solution a programmer knows and, as Mark Summerfield noted, most languages don't offer any alternatives. We've seen functional languages with immutable state rise up on the one hand and message passing (such as Erlang) on the other to offer alternatives to the many problems inherent in multithreaded programming.
weberc2 1 points · Jan 2017
They're flaky because a process will sometimes mysteriously be killed or stall indefinitely. They are slow because all communication between processes must be serialized on the sender and deserialized on the receiver, because processes don't support a shared memory space. Further, as I mentioned elsewhere, each process requires its own interpreter, which takes 5MB, so you won't be scaling these processes.
I don't dispute that there are people out there who think threads are evil, I dispute that this belief is widely held, or that this belief isn't widely held because no one knows about processes. Experienced developers still prefer threads to processes. Processes are not a secret; they're lesser used because they are more memory intensive and intercommunication is horribly slow.