I'm sure we all know this, but just in case-- even with the GIL, threads are fine for IO bound situations and when calling code outside the GIL. The GIL just prevents threads from being useful for scaling out CPU bound work in Python.
I don't actually think this is particularly interesting. In my opinion, there are a lot of problems that can be easily thought of and implemented as processes that communicate via atomic message queues. If you've structured things properly, you ought to be able to use processes or threads interchangeably.
There is of course the obvious problem of resource use when using processes instead of threads. Processes are a lot heavier.
There are bigger problems though.
When using multiple processes, you have to serialize objects to pass them through your queue. There's overhead there, sure, but the bigger issue in my experience is passing messages that contain objects from third party libraries that don't play nicely with pickle and needing to write a bunch of boilerplate to get around it. The other issue is with third party libraries that are thread safe but create mutable in-memory shared state, like connection pools and caches-- strange things can happen.
So, threads are nice. They avoid implementation details of multiprocessing (inefficiency, serialization, mutable state in third party libs). They can be very intuitive tools for building concurrent applications. I just wish Python would easily handle huge thread counts like golang, erlang, etc., so that I can run more than ~1k concurrent threads.
I really liked the idea of erlang (and now Elixir) but have been turned off by the syntax. It looks like Go will get me similar kinds of features while having much nicer syntax. I should get around to writing some, and Grumpy looks like a nice component of the Go ecosystem. I hope it progresses well.
You just described how I feel about Go and Rust - I already like Go because of it's simplicity and power (it's like a nicer C) that you can learn over dinner.
On the other hand there's Rust that is very promising, can totally replace C and C++ and is growing by the day - yet I feel like the syntax is horrible, reminescent of the mess that C++ is. I've decided yesterday to try to look past my bias and force myself to learn (and who knows, love) Rust too, since I think it will be a good complement to Go's weaknesses - allows me to write low-level code like C with the nice parts of generics and OOP without having to touch C++.
I don't mind Rust's syntax, but I have a hard time understanding how things like closures work and how functions are passed. It seems like everything needs a RefCell, and I don't know when or why. Even coming from C++, the memory model (while safe) still places a lot of demands on the programmer which aren't required in Go--I can do a LOT of optimizing in Go before something becomes easier to write in Rust, and at that point the Rust code to outperform the optimized Go version is still nontrivial...
Pretty sure you're responding to bait :)
I'm sure we all know this, but just in case-- even with the GIL, threads are fine for IO bound situations and when calling code outside the GIL. The GIL just prevents threads from being useful for scaling out CPU bound work in Python.
I don't actually think this is particularly interesting. In my opinion, there are a lot of problems that can be easily thought of and implemented as processes that communicate via atomic message queues. If you've structured things properly, you ought to be able to use processes or threads interchangeably.
There is of course the obvious problem of resource use when using processes instead of threads. Processes are a lot heavier.
There are bigger problems though.
When using multiple processes, you have to serialize objects to pass them through your queue. There's overhead there, sure, but the bigger issue in my experience is passing messages that contain objects from third party libraries that don't play nicely with pickle and needing to write a bunch of boilerplate to get around it. The other issue is with third party libraries that are thread safe but create mutable in-memory shared state, like connection pools and caches-- strange things can happen.
So, threads are nice. They avoid implementation details of multiprocessing (inefficiency, serialization, mutable state in third party libs). They can be very intuitive tools for building concurrent applications. I just wish Python would easily handle huge thread counts like golang, erlang, etc., so that I can run more than ~1k concurrent threads.
[deleted] 4 points · Jan 2017
you mean, like Go's channels? Without the serialization overhead because it's all under the same address space. =)
AnimalFarmPig 5 points · Jan 2017
I really liked the idea of erlang (and now Elixir) but have been turned off by the syntax. It looks like Go will get me similar kinds of features while having much nicer syntax. I should get around to writing some, and Grumpy looks like a nice component of the Go ecosystem. I hope it progresses well.
[deleted] 4 points · Jan 2017
You just described how I feel about Go and Rust - I already like Go because of it's simplicity and power (it's like a nicer C) that you can learn over dinner.
On the other hand there's Rust that is very promising, can totally replace C and C++ and is growing by the day - yet I feel like the syntax is horrible, reminescent of the mess that C++ is. I've decided yesterday to try to look past my bias and force myself to learn (and who knows, love) Rust too, since I think it will be a good complement to Go's weaknesses - allows me to write low-level code like C with the nice parts of generics and OOP without having to touch C++.
weberc2 6 points · Jan 2017
I don't mind Rust's syntax, but I have a hard time understanding how things like closures work and how functions are passed. It seems like everything needs a RefCell, and I don't know when or why. Even coming from C++, the memory model (while safe) still places a lot of demands on the programmer which aren't required in Go--I can do a LOT of optimizing in Go before something becomes easier to write in Rust, and at that point the Rust code to outperform the optimized Go version is still nontrivial...