To run Python code as a Go compiled executable. This allows them to avoid the Python Global Interpreter Lock (which severely limits Python's scalability within a single process) and run the Python code using Go modules as if they were Python modules.
Really, unless you're Google or another Python shop interesting in moving to Go, you probably don't have a use for this.
Getting around the GIL is a huge step. Python's chief weakness is lack of parallel scalability, IMO. Most solutions around this (joblib, dask, etc.) don't feel like complete solutions. Not when, for example, in C++ or Julia you can slap a macro above a loop to make it instantly parallel.
For the record, this is not a complete solution either. Grumpy is a tool that compiles a single Python file (no real module support) while eschewing most of Python's dynamic features and supporting very little of its standard library.
This is misleading. The features you mention are only missing because they haven't yet been implemented, but the intention is to produce a drop-in replacement for CPython. It's only an "incomplete solution" in the sense that it's alpha software, not that its goals are incompatible.
Yes, it's incomplete in the "it's alpha software", sense; not in the "there's something fundamentally wrong with it" sense. No one is suggesting it's anything other than alpha software.
Although /u/theseoafs didnt mention it, grumpy explicitly doesn't aim for support of the python C API. That's a big limitation and will rule it out for a big chunk of python users.
The features you mention are only missing because they haven't yet been implemented, but the intention is to produce a drop-in replacement for CPython. It's only an "incomplete solution" in the sense that it's alpha software, not that its goals are incompatible.
LOL. And GNU Hurd is only an "incomplete solution" in the sense that it should not be used in production and there are many missing features, not that its goals are incompatible with being with a usable unix-like system.
Furthermore, you think that this is the first time we hear about a project that is going to remove the GIL and be a drop-in replacement for CPython? /u/theseoafs is not misleading anyone. Once they have a decent beta version supporting all the features CPython has and some benchmarks showing that there is actually a performance improvement then we'll check it out.
That's fine, and this project may as well; my point is there is a distinction between incomplete in time and incomplete in scope. This project is only the former as far as I can tell.
... my point is there is a distinction between incomplete in time and incomplete in scope. This project is only the former as far as I can tell.
No. In the project web page at Github they state that it won't be a drop-in replacement for CPython 2.7. Also from the discussion on HN it is strongly implied (apart from the language features that won't be supported) that it won't support C extensions ever.
You said producing a drop-in replacement for CPython was in the scope of the project. It is not a dispute. I'm just correcting that statement.
The project's scope is more or less CPython without C extensions. The author is unambiguous.
The project's scope is to implement as much CPython 2.7 features as they can that at the same time don't sacrifice performance and without using a GIL. That's actually very far from implementing more or less CPython without C extensions that is different from what you originally stated before. The author is unambiguous. You are not.
What are you confused about?
Nothing. Hopefully you are not confused anymore about what the project is actually trying to achieve.
Considering this has been tried multiple times before with no success (including by, wait for it, Google), what are your reasons for thinking this announcement isn't premature other than "they haven't failed yet"?
I don't see anything critically wrong with the approach, other than the sheer work required to build a quirks-compatible CPython replacement. In particular, the runtime seems similar to CPython except for the GIL. Building on Go's runtime and concurrency primaries should help here as well.
You're awfully sarcastic for someone who doesn't understand "alpha"...
Oh, I don't understand "alpha". Here's the thing: we've seen this over and over again. Implementing a runtime that has better performance for a subset of the language and writing a blog post about it is the easy part. Heck, just a few weeks ago a guy presented a JS runtime that was way better than V8. But once people started to ask him if it implemented the whole language and that if it was actually the case then this was a scientific breakthrough shit did hit the fan.
So if you want to be optimistic about this project then go ahead. But don't come here telling us that "it is just alpha" and "the goals are not incompatible" history have showed us that they actually are and that's why it is a damn hard problem to solve. So when you say this:
... The features you mention are only missing because they haven't yet been implemented...
I just have to laugh hard. "Are only missing". Once those features are implemented is when performance graphs like this one stop being as pretty as they were in the past.
The only design decision that prevents it from compiling the standard library is the lack of C support. The solution is to reimplement those modules in Go or Python. Anything else that doesn't compile is due to bugs in this alpha implementation (as far as I can tell, anyway).
Right, this comment was about compiling the standard library, not building a GIL-less Python.
EDIT: From what I understand, supporting C extensions and threadsafe garbage collection are the two biggest challenges to removing the GIL. This project uses Go's threadsafe garbage collector, so the latter shouldn't be a problem.
People seem to be divided, because while the language is not bad, it's nothing special (seems like the only significant thing it brings to the table is the go routines, but other languages are adding similar functionality), there's a bit of hype going on because it's backed by Google.
It also initially was trying to be marketed as a C replacement, but it is nowhere close to fit the niche that C fits. Rust seems like a language that has potential to replace C, but it is too early to say.
Go is more on level of Python, Java etc, although a bit lower level than them. Some people compared it to D, but I never used it, so can't comment.
The goal of this language is to be a very simple and opinionated language, some people like that some don't.
If you are asking the question you can try and see if you like it, it has low learning curve, so it's not hard to learn it, and then you will have your opinion about it.
I've heard it's got a bit of boilerplate and written to be as easy to write as possible to help with the incredible amount of new talent Google gets out of schools.
Id say it's more about making it easy to read and maintain. The google code base is 2 billion+ LOC, so designing a language that eases the burden of maintaining a code base of that size seems like a sensible goal.
I think so. If you have prior programming experience and a free afternoon, you should be able to start writing real (i.e., non-toy) programs. Check out http://tour.golang.org.
Go has a lot of quirks--most notably, no generics--which turn off a lot of folks. But if you want to build performant, scalable software quickly, Go is worth a look.
I think so. If you have prior programming experience and a free afternoon, you should be able to start writing real (i.e., non-toy) programs. Check out http://tour.golang.org.
Not entirely true. You should be familiar with pointer based languages like C. If you come from a python/ruby/Java experience you will spend a whole lot longer to get used to pointers.
Source: work had a hackathon. I decided to code in go. Spent a lot of time correcting my code related to pointers before it would work.
I don't know about python and ruby, but you can actually run into a lot of really weird bugs in Java if you don't have at least a cursory understanding of how pointers and references work.
The big problem that companies like Google and Facebook and the like are constantly trying to solve is that folks who know dynamic languages like Python or PHP or Ruby are a dime a dozen, but these languages really don't scale well to even ordinary enterprise scales let alone the kind of volume these companies see.
That's why we see so many transpilers and why Twitter is written in PHP even though the general industry consensus is that it's probably one of the worst non joke languages in existence.
Python/Ruby/Java are all pointer based languages. C, C++, and Go simply have value types in addition to pointers. But yes it is important to understand the difference between pass by value and pass by copy. Fair point.
Regardless of what we think of Go, the language has found its killer domain.
It is the main language used to develop GNU/Linux based containers (Docker and Kubernetes) and Microsoft is a big supporter of it in VSCode, so anyone that cares about having skills in this domain should learn Go.
There are some neat simple few line solutions that get you parallelism. The title is a bit link bait and not entirely accurate, but I've found this pattern to be extremely effective: http://chriskiehl.com/article/parallelism-in-one-line/
whenever you need a lot of interprocess communication. E.g. working on a big, shared array together. I do a lot of data science, and sometimes I can't allow my dataset to be copied back and forth between the different processes... e.g. when it's 100 GB in size. It leads to me having to rewrite algorithms in sometimes cumbersome ways.
that's what I mean by "rewriting algorithms in cumbersome ways" ;-) If threading would just work as intended, parallelizing would be a trivial job because I could share the array, so it essentially be a one-liner. Splitting/Dispaching means non-trivial modifications and possible bugs due to out-of-memory failures etc.
And that's when such a modification is possible. Sometimes, you cannot just split the data because your algorithm has random write patterns. Then you're out of luck and have to rethink your whole algorithm.
If you truly have huge datasets, then you'd need to distribute processing to a grid of machines, a computing cluster, right? Your code being able to split up the work for many semi-independent instances of Python will be an advantage.
I have access to very large machines. So no, I don't usually need to distribute across machines. But even for simple one-time jobs that don't require a ton of resources I hate that I even have to think about how to split up my data just because I want to use 16 cores instead of just 1. Yes it's doable, but sometimes I feel like the time I gain by parallelizing my workload gets lost by me having to deal with splitting up my dataset / merging the results.
This is wrong: Python has plenty of parallel features - you just have to spend ten seconds looking.
The most convenient is the concurrent.futures module. You can use the exact same syntax for either threading or multiprocessing. Yesterday I sped up an AWS S3 downloader about 8x with threading, and about six lines of code.
A few years ago I wrote a transform process that handled about 4 billion records a day - using multiprocessing on two 32-core machines to handle downloading files, transforming them, and uploading them again. All in parallel using pypy, multiprocessing & threading. This process worked great, and surprised everyone with how fast it was. A rewrite of a part of it showed that Go was about 2.5 times faster, which was fine, but not fast enough to warrant a rewrite until we needed to scale up quite a bit more.
The only scenario in which Python's parallelism is limited is when you've got a CPU-bound process that either can't afford the extra memory or start-up times of multiprocessing or needs a lot of communication between processes. Then you want threading but the GIL will limit you. Other than this case, Python has fine parallelism features.
Yes, for your use case it works fine. My use case is closer to your last point, where python still generally sucks (though I haven't used Dask enough to say if it's a decent solution). It's currently being addressed by Julia, which bypasses the "core loop in C++, wrapped in python" problem
As someone who doesn't use Go or Python: what is the use for this?
vplatt 302 points · Jan 2017
To run Python code as a Go compiled executable. This allows them to avoid the Python Global Interpreter Lock (which severely limits Python's scalability within a single process) and run the Python code using Go modules as if they were Python modules.
Really, unless you're Google or another Python shop interesting in moving to Go, you probably don't have a use for this.
VodkaHaze 35 points · Jan 2017
Getting around the GIL is a huge step. Python's chief weakness is lack of parallel scalability, IMO. Most solutions around this (joblib, dask, etc.) don't feel like complete solutions. Not when, for example, in C++ or Julia you can slap a macro above a loop to make it instantly parallel.
[deleted] 38 points · Jan 2017
For the record, this is not a complete solution either. Grumpy is a tool that compiles a single Python file (no real module support) while eschewing most of Python's dynamic features and supporting very little of its standard library.
weberc2 24 points · Jan 2017
This is misleading. The features you mention are only missing because they haven't yet been implemented, but the intention is to produce a drop-in replacement for CPython. It's only an "incomplete solution" in the sense that it's alpha software, not that its goals are incompatible.
[deleted] 5 points · Jan 2017
OK, but it's not even particularly close to meeting that goal right now, so it's an incomplete solution.
weberc2 22 points · Jan 2017
Yes, it's incomplete in the "it's alpha software", sense; not in the "there's something fundamentally wrong with it" sense. No one is suggesting it's anything other than alpha software.
the_birds_and_bees 9 points · Jan 2017
Although /u/theseoafs didnt mention it, grumpy explicitly doesn't aim for support of the python C API. That's a big limitation and will rule it out for a big chunk of python users.
weberc2 1 points · Jan 2017
I understand and agree. The parallelism support might draw in even more users. Who knows? This project is super nascent.
Spider_pig448 0 points · Jan 2017
No, it's an incomplete implementation of a complete solution.
throwawayco111 -8 points · Jan 2017
LOL. And GNU Hurd is only an "incomplete solution" in the sense that it should not be used in production and there are many missing features, not that its goals are incompatible with being with a usable unix-like system.
Furthermore, you think that this is the first time we hear about a project that is going to remove the GIL and be a drop-in replacement for CPython? /u/theseoafs is not misleading anyone. Once they have a decent beta version supporting all the features CPython has and some benchmarks showing that there is actually a performance improvement then we'll check it out.
weberc2 9 points · Jan 2017
You're awfully sarcastic for someone who doesn't understand "alpha"... Better luck next time.
floodyberry 10 points · Jan 2017
Everyone else who tried this and failed made it to the "alpha" stage as well.
weberc2 1 points · Jan 2017
That's fine, and this project may as well; my point is there is a distinction between incomplete in time and incomplete in scope. This project is only the former as far as I can tell.
throwawayco111 2 points · Jan 2017
No. In the project web page at Github they state that it won't be a drop-in replacement for CPython 2.7. Also from the discussion on HN it is strongly implied (apart from the language features that won't be supported) that it won't support C extensions ever.
weberc2 1 points · Jan 2017
No one disputes this. The project's scope is more or less CPython without C extensions. The author is unambiguous. What are you confused about?
throwawayco111 1 points · Jan 2017
You said producing a drop-in replacement for CPython was in the scope of the project. It is not a dispute. I'm just correcting that statement.
The project's scope is to implement as much CPython 2.7 features as they can that at the same time don't sacrifice performance and without using a GIL. That's actually very far from implementing more or less CPython without C extensions that is different from what you originally stated before. The author is unambiguous. You are not.
Nothing. Hopefully you are not confused anymore about what the project is actually trying to achieve.
load 0 more replies →
floodyberry 1 points · Jan 2017
Considering this has been tried multiple times before with no success (including by, wait for it, Google), what are your reasons for thinking this announcement isn't premature other than "they haven't failed yet"?
weberc2 1 points · Jan 2017
I don't see anything critically wrong with the approach, other than the sheer work required to build a quirks-compatible CPython replacement. In particular, the runtime seems similar to CPython except for the GIL. Building on Go's runtime and concurrency primaries should help here as well.
load 0 more replies →
throwawayco111 0 points · Jan 2017
Oh, I don't understand "alpha". Here's the thing: we've seen this over and over again. Implementing a runtime that has better performance for a subset of the language and writing a blog post about it is the easy part. Heck, just a few weeks ago a guy presented a JS runtime that was way better than V8. But once people started to ask him if it implemented the whole language and that if it was actually the case then this was a scientific breakthrough shit did hit the fan.
So if you want to be optimistic about this project then go ahead. But don't come here telling us that "it is just alpha" and "the goals are not incompatible" history have showed us that they actually are and that's why it is a damn hard problem to solve. So when you say this:
I just have to laugh hard. "Are only missing". Once those features are implemented is when performance graphs like this one stop being as pretty as they were in the past.
EDIT: just checked the discussion at HN with some of the developers involved. Don't expect ever this be a drop-in replacement for CPython.
[deleted] 25 points · Jan 2017
It doesn't need GIL, because it offers subset of functionality that python has. It is not even capable of compiling all python standard library.
This code is for Google to move away from python. They can include python libraries in their go code and then one by one rewrite it in go.
weberc2 4 points · Jan 2017
The only design decision that prevents it from compiling the standard library is the lack of C support. The solution is to reimplement those modules in Go or Python. Anything else that doesn't compile is due to bugs in this alpha implementation (as far as I can tell, anyway).
[deleted] 2 points · Jan 2017
PyPy also doesn't natively support C extension and also needs GIL. C support is one of reason of GIL's existence but not the only one.
weberc2 1 points · Jan 2017
Right, this comment was about compiling the standard library, not building a GIL-less Python.
EDIT: From what I understand, supporting C extensions and threadsafe garbage collection are the two biggest challenges to removing the GIL. This project uses Go's threadsafe garbage collector, so the latter shouldn't be a problem.
EDIT2: This page has some good information: https://wiki.python.org/moin/GlobalInterpreterLock
sultry_somnambulist 10 points · Jan 2017
Speaking of this, what's the general trajectory of go? Is it worth dabbling into at this point? People seem to be very divided over the language.
[deleted] 15 points · Jan 2017
People seem to be divided, because while the language is not bad, it's nothing special (seems like the only significant thing it brings to the table is the go routines, but other languages are adding similar functionality), there's a bit of hype going on because it's backed by Google.
It also initially was trying to be marketed as a C replacement, but it is nowhere close to fit the niche that C fits. Rust seems like a language that has potential to replace C, but it is too early to say.
Go is more on level of Python, Java etc, although a bit lower level than them. Some people compared it to D, but I never used it, so can't comment. The goal of this language is to be a very simple and opinionated language, some people like that some don't.
If you are asking the question you can try and see if you like it, it has low learning curve, so it's not hard to learn it, and then you will have your opinion about it.
TankorSmash 2 points · Jan 2017
I've heard it's got a bit of boilerplate and written to be as easy to write as possible to help with the incredible amount of new talent Google gets out of schools.
Only a rumour though.
the_birds_and_bees 5 points · Jan 2017
Id say it's more about making it easy to read and maintain. The google code base is 2 billion+ LOC, so designing a language that eases the burden of maintaining a code base of that size seems like a sensible goal.
derraidor 1 points · Jan 2017
There is a talk by one of the creators (Russ Cox?) stating this
[deleted] 2 points · Jan 2017
If you like it, then do it. No reason not to.
The community is sizable and there's a ton of tooling so it's not really like you're taking a risk.
weberc2 4 points · Jan 2017
I think so. If you have prior programming experience and a free afternoon, you should be able to start writing real (i.e., non-toy) programs. Check out http://tour.golang.org.
Go has a lot of quirks--most notably, no generics--which turn off a lot of folks. But if you want to build performant, scalable software quickly, Go is worth a look.
th3_pund1t 8 points · Jan 2017
Not entirely true. You should be familiar with pointer based languages like C. If you come from a python/ruby/Java experience you will spend a whole lot longer to get used to pointers.
Source: work had a hackathon. I decided to code in go. Spent a lot of time correcting my code related to pointers before it would work.
recycled_ideas 6 points · Jan 2017
I don't know about python and ruby, but you can actually run into a lot of really weird bugs in Java if you don't have at least a cursory understanding of how pointers and references work.
The big problem that companies like Google and Facebook and the like are constantly trying to solve is that folks who know dynamic languages like Python or PHP or Ruby are a dime a dozen, but these languages really don't scale well to even ordinary enterprise scales let alone the kind of volume these companies see.
That's why we see so many transpilers and why Twitter is written in PHP even though the general industry consensus is that it's probably one of the worst non joke languages in existence.
weberc2 0 points · Jan 2017
Python/Ruby/Java are all pointer based languages. C, C++, and Go simply have value types in addition to pointers. But yes it is important to understand the difference between pass by value and pass by copy. Fair point.
pjmlp -3 points · Jan 2017
Regardless of what we think of Go, the language has found its killer domain.
It is the main language used to develop GNU/Linux based containers (Docker and Kubernetes) and Microsoft is a big supporter of it in VSCode, so anyone that cares about having skills in this domain should learn Go.
kalifornia_love -7 points · Jan 2017
I'd say go ahead and start dabbling ;)
Sorry. I had to and I'm also interested in seeing people's responses to this.
[deleted] 1 points · Jan 2017
that is a very strange thing to say
Twirrim 1 points · Jan 2017
There are some neat simple few line solutions that get you parallelism. The title is a bit link bait and not entirely accurate, but I've found this pattern to be extremely effective: http://chriskiehl.com/article/parallelism-in-one-line/
tetroxid 2 points · Jan 2017
import multiprocessingBeatLeJuce 3 points · Jan 2017
there are instances where you NEED threads instead of processes.
tetroxid -1 points · Jan 2017
Could you make an example?
BeatLeJuce 2 points · Jan 2017
whenever you need a lot of interprocess communication. E.g. working on a big, shared array together. I do a lot of data science, and sometimes I can't allow my dataset to be copied back and forth between the different processes... e.g. when it's 100 GB in size. It leads to me having to rewrite algorithms in sometimes cumbersome ways.
tetroxid -1 points · Jan 2017
Split and dispatch the data to be processed, then gather the results?
BeatLeJuce 3 points · Jan 2017
that's what I mean by "rewriting algorithms in cumbersome ways" ;-) If threading would just work as intended, parallelizing would be a trivial job because I could share the array, so it essentially be a one-liner. Splitting/Dispaching means non-trivial modifications and possible bugs due to out-of-memory failures etc.
And that's when such a modification is possible. Sometimes, you cannot just split the data because your algorithm has random write patterns. Then you're out of luck and have to rethink your whole algorithm.
tetroxid 1 points · Jan 2017
If you truly have huge datasets, then you'd need to distribute processing to a grid of machines, a computing cluster, right? Your code being able to split up the work for many semi-independent instances of Python will be an advantage.
BeatLeJuce 3 points · Jan 2017
I have access to very large machines. So no, I don't usually need to distribute across machines. But even for simple one-time jobs that don't require a ton of resources I hate that I even have to think about how to split up my data just because I want to use 16 cores instead of just 1. Yes it's doable, but sometimes I feel like the time I gain by parallelizing my workload gets lost by me having to deal with splitting up my dataset / merging the results.
load 0 more replies →
kenfar -3 points · Jan 2017
This is wrong: Python has plenty of parallel features - you just have to spend ten seconds looking.
The most convenient is the concurrent.futures module. You can use the exact same syntax for either threading or multiprocessing. Yesterday I sped up an AWS S3 downloader about 8x with threading, and about six lines of code.
A few years ago I wrote a transform process that handled about 4 billion records a day - using multiprocessing on two 32-core machines to handle downloading files, transforming them, and uploading them again. All in parallel using pypy, multiprocessing & threading. This process worked great, and surprised everyone with how fast it was. A rewrite of a part of it showed that Go was about 2.5 times faster, which was fine, but not fast enough to warrant a rewrite until we needed to scale up quite a bit more.
The only scenario in which Python's parallelism is limited is when you've got a CPU-bound process that either can't afford the extra memory or start-up times of multiprocessing or needs a lot of communication between processes. Then you want threading but the GIL will limit you. Other than this case, Python has fine parallelism features.
VodkaHaze 7 points · Jan 2017
Yes, for your use case it works fine. My use case is closer to your last point, where python still generally sucks (though I haven't used Dask enough to say if it's a decent solution). It's currently being addressed by Julia, which bypasses the "core loop in C++, wrapped in python" problem
kenfar 0 points · Jan 2017
Are you positive that you can't use multiprocessing?
I've seen so many cases where people weren't aware that it even exists, or if aware didn't think they could use it.