Not every language needs to do everything. It'll be great if they improve concurrency but Python is in my opinion best used as a glue language or a language for rapid development. Other people probably have other great uses for it but no one sane uses Python for its performance. If concurrency is your thing, probably shouldn't be writing in Python, there's Go or Java that work great in that realm.
I'm curious to see which operations you feel are more painful in Python than Go. As far as I can tell, the main differences are that you choose which event loop to use, rather than having one baked into the runtime, and most of the standard libraries are not designed to work asynchronously. On the other hand, you're less likely to run into memory leaks or panics.
That's it - support for asynchronous operations was patched into Python later as an afterthought, Go was born with it from day one. I don't have to sprinkle async/await all around to get asynchronous behavior, I still get parallelism for free (the whole reason for this thread), you're basically writing asynchronous code as you would do normally without having to think hard about what you're doing lest you freeze the event loop by doing something that blocks everything (like reading from a file using regular file operations). The multithreaded event loop is certainly nice (can you choose a multithreaded event loop for Python?) and having goroutines multiplexed automatically for you and preemptively scheduled so nobody starves for CPU is also great (where in Python you have to yield manually). What happens if you forget to use asyncio.sleep and uses time.sleep instead?
As for memory leaks it is also pretty hard to leak memory on Go because of the garbage collector (which again, is much better than Python's - go1.8's GC is realtime and keeps pauses to under a millisecond) and that can be detected by using tools that ship with Go during testing - same with data races.
As for panics, you can wrap your main logic and use Recover() to detect panics without crashing (even if just to report the panic somewhere and restart the program) - and you can only panic if you explicitly call Panic() or if you're doing something wrong anyways (acessing an out-of-bounds item in an array, dereferencing a null pointer, etc, things that shouldn't be normally happening). In Python if you try to access an element that does not exist in an array you'll get an exception too, that if you don't catch, your program will crash. So on Go you can "catch" panics with Recover().
Don't take me wrong, Python is great, but it's not the best tool for writing massively parallel and concurrent applications nowadays, when you can do it in Go and have it perform very well at native speeds, don't have to deal with callback hell (like in nodejs) and you can just write your code linearly as you would normally do and the runtime takes care of the rest - EVERY Go library is asynchronous by nature.
If someone needs Python's asyncio to do more, that's a sign that they should consider writing that particular piece in Go instead.
To be honest, I haven't actually used Python's asyncio, so I don't have much experience with it.
But as far as the memory leak issue I mentioned, the problem is that goroutines are not garbage collected in Go. So if you have a goroutine producing values on a channel and you forget to completely exhaust the channel on the other end, you'll effectively leak memory.
Also, you don't need this performance. You might think you do, but I can assure you you don't!
Prove concurrency is the bottleneck. With actual numbers, facts and research and real-world benchmarks on your entire stack.
Fix the actual performance problems (they are probably IO, Database, serializing and such). Add an async worker or two.
Rince, repeat.
Is Python itself the actual performance problem? You've reached the bottom. Now go spend $2000/year on larger or more servers. Because an engineer rewriting your stack in go/haskell/erlang costs ten to hundred times that.
Same goes for Ruby.
And all this assumes you have managed to get enough users and grow large enough that performance is an actual, real problem, hurting your bottomline or growth.
All these interpreted languages (well, maybe not PHP) solve this by making such communication out-of-band. Asynchronous workers are common and "Good enough" for 99.99% of all startups.
I think this is a very practical view on the subject. Unfortunately programmers often have an engineering mindset where they want to build the best in every aspect. What your saying echoes my experiences, use a language like Python that you can get a lot of shit done in a short period of time, come back and optimize it when its feature complete and you can tell people want to use the thing you made. Despite knowing Haskell and Rust my most successful projects were written in PHP and Python, some of which are still being heavily used 9 years after their creation. They were thrown together in short time and because of the rapid development cycle those languages fit well into, the sites quickly evolved to meet the users need.
I'm sure there are cases where you need rock solid performance and concurrency but I don't think that's the norm.
The proof is also in Twitter (Ruby on Rails), the OP explaining Youtube runs Python, and many more such gigantic success-stories. Starting with a platform that "does not scale".
And that ignores all the projects that failed: many more. It's far worse to fail when you've just spent years engineering a concurrent messaging protocol for a failed project, than when you spent four days of Rails scaffolding for a failed project.
i'm looking forward to guilds too, but its not out yet :) And likely, you'll have to change coding style considerably to do multi-processing with guilds. My guess is that guilds will be most practically applicable to implementing things like Sidekiq or other engines that currently run an independent Ruby process.
This just makes me think: when is python going to get its shit together and get concurrency working properly?
stormcrowsx 13 points · Jan 2017
Not every language needs to do everything. It'll be great if they improve concurrency but Python is in my opinion best used as a glue language or a language for rapid development. Other people probably have other great uses for it but no one sane uses Python for its performance. If concurrency is your thing, probably shouldn't be writing in Python, there's Go or Java that work great in that realm.
anedisi 3 points · Jan 2017
when i need some service with concurrency and asynchronicity i write that in javascript and run it in nodejs.
it worked ok for now.
__eastwood 4 points · Jan 2017
This breaks down when you need CPU bound concurrent tasks
[deleted] 1 points · Jan 2017
Valid point. I've been thinking of look at Pony for concurrency.
Uncaffeinated 1 points · Jan 2017
I'm not even sure what needs to be improved about concurrency (as opposed to parallelism). Python know has async and await from what I've heard.
[deleted] 3 points · Jan 2017
Absurdly painful to use compared to how Go does it.
Uncaffeinated 1 points · Jan 2017
I'm curious to see which operations you feel are more painful in Python than Go. As far as I can tell, the main differences are that you choose which event loop to use, rather than having one baked into the runtime, and most of the standard libraries are not designed to work asynchronously. On the other hand, you're less likely to run into memory leaks or panics.
[deleted] 1 points · Jan 2017
That's it - support for asynchronous operations was patched into Python later as an afterthought, Go was born with it from day one. I don't have to sprinkle async/await all around to get asynchronous behavior, I still get parallelism for free (the whole reason for this thread), you're basically writing asynchronous code as you would do normally without having to think hard about what you're doing lest you freeze the event loop by doing something that blocks everything (like reading from a file using regular file operations). The multithreaded event loop is certainly nice (can you choose a multithreaded event loop for Python?) and having goroutines multiplexed automatically for you and preemptively scheduled so nobody starves for CPU is also great (where in Python you have to yield manually). What happens if you forget to use asyncio.sleep and uses time.sleep instead?
As for memory leaks it is also pretty hard to leak memory on Go because of the garbage collector (which again, is much better than Python's - go1.8's GC is realtime and keeps pauses to under a millisecond) and that can be detected by using tools that ship with Go during testing - same with data races.
As for panics, you can wrap your main logic and use Recover() to detect panics without crashing (even if just to report the panic somewhere and restart the program) - and you can only panic if you explicitly call Panic() or if you're doing something wrong anyways (acessing an out-of-bounds item in an array, dereferencing a null pointer, etc, things that shouldn't be normally happening). In Python if you try to access an element that does not exist in an array you'll get an exception too, that if you don't catch, your program will crash. So on Go you can "catch" panics with Recover().
Don't take me wrong, Python is great, but it's not the best tool for writing massively parallel and concurrent applications nowadays, when you can do it in Go and have it perform very well at native speeds, don't have to deal with callback hell (like in nodejs) and you can just write your code linearly as you would normally do and the runtime takes care of the rest - EVERY Go library is asynchronous by nature.
If someone needs Python's asyncio to do more, that's a sign that they should consider writing that particular piece in Go instead.
Uncaffeinated 0 points · Jan 2017
To be honest, I haven't actually used Python's asyncio, so I don't have much experience with it.
But as far as the memory leak issue I mentioned, the problem is that goroutines are not garbage collected in Go. So if you have a goroutine producing values on a channel and you forget to completely exhaust the channel on the other end, you'll effectively leak memory.
berkes -2 points · Jan 2017
Also, you don't need this performance. You might think you do, but I can assure you you don't!
Same goes for Ruby.
And all this assumes you have managed to get enough users and grow large enough that performance is an actual, real problem, hurting your bottomline or growth.
[deleted] 2 points · Jan 2017
Performance isn't everything, there is latency too, specially in distributed/SOA architectures where latencies add up.
berkes 0 points · Jan 2017
All these interpreted languages (well, maybe not PHP) solve this by making such communication out-of-band. Asynchronous workers are common and "Good enough" for 99.99% of all startups.
stormcrowsx 2 points · Jan 2017
I think this is a very practical view on the subject. Unfortunately programmers often have an engineering mindset where they want to build the best in every aspect. What your saying echoes my experiences, use a language like Python that you can get a lot of shit done in a short period of time, come back and optimize it when its feature complete and you can tell people want to use the thing you made. Despite knowing Haskell and Rust my most successful projects were written in PHP and Python, some of which are still being heavily used 9 years after their creation. They were thrown together in short time and because of the rapid development cycle those languages fit well into, the sites quickly evolved to meet the users need.
I'm sure there are cases where you need rock solid performance and concurrency but I don't think that's the norm.
berkes 1 points · Jan 2017
The proof is also in Twitter (Ruby on Rails), the OP explaining Youtube runs Python, and many more such gigantic success-stories. Starting with a platform that "does not scale".
And that ignores all the projects that failed: many more. It's far worse to fail when you've just spent years engineering a concurrent messaging protocol for a failed project, than when you spent four days of Rails scaffolding for a failed project.
iconoclaus 11 points · Jan 2017
you could ask this of almost any dynamically typed language. btw, i think you meant parallelism, not concurrency.
renrutal 2 points · Jan 2017
Elixir
iconoclaus 2 points · Jan 2017
that's actually a really good counter-example afaik
hector_villalobos 1 points · Jan 2017
Check again?: http://olivierlacan.com/posts/concurrency-in-ruby-3-with-guilds/
iconoclaus 1 points · Jan 2017
i'm looking forward to guilds too, but its not out yet :) And likely, you'll have to change coding style considerably to do multi-processing with guilds. My guess is that guilds will be most practically applicable to implementing things like Sidekiq or other engines that currently run an independent Ruby process.
[deleted] 1 points · Jan 2017
I'm sorry, I'm afraid I get them mixed up at times. What is the difference between the two?
iconoclaus 1 points · Jan 2017
no prob. see my other comment.
forreddits -1 points · Jan 2017
Parallelism is one form of concurrency.
iconoclaus 7 points · Jan 2017
my understanding is that parallelism and concurrency are two forms of asynchronicity. Also see Rob Pike's talk on the same.
mirhagk 2 points · Jan 2017
Concurrency is working on multiple things. Parallelism is doing it at the same time.