To run Python code as a Go compiled executable. This allows them to avoid the Python Global Interpreter Lock (which severely limits Python's scalability within a single process) and run the Python code using Go modules as if they were Python modules.
Really, unless you're Google or another Python shop interesting in moving to Go, you probably don't have a use for this.
Getting around the GIL is a huge step. Python's chief weakness is lack of parallel scalability, IMO. Most solutions around this (joblib, dask, etc.) don't feel like complete solutions. Not when, for example, in C++ or Julia you can slap a macro above a loop to make it instantly parallel.
For the record, this is not a complete solution either. Grumpy is a tool that compiles a single Python file (no real module support) while eschewing most of Python's dynamic features and supporting very little of its standard library.
This is misleading. The features you mention are only missing because they haven't yet been implemented, but the intention is to produce a drop-in replacement for CPython. It's only an "incomplete solution" in the sense that it's alpha software, not that its goals are incompatible.
Yes, it's incomplete in the "it's alpha software", sense; not in the "there's something fundamentally wrong with it" sense. No one is suggesting it's anything other than alpha software.
Although /u/theseoafs didnt mention it, grumpy explicitly doesn't aim for support of the python C API. That's a big limitation and will rule it out for a big chunk of python users.
The features you mention are only missing because they haven't yet been implemented, but the intention is to produce a drop-in replacement for CPython. It's only an "incomplete solution" in the sense that it's alpha software, not that its goals are incompatible.
LOL. And GNU Hurd is only an "incomplete solution" in the sense that it should not be used in production and there are many missing features, not that its goals are incompatible with being with a usable unix-like system.
Furthermore, you think that this is the first time we hear about a project that is going to remove the GIL and be a drop-in replacement for CPython? /u/theseoafs is not misleading anyone. Once they have a decent beta version supporting all the features CPython has and some benchmarks showing that there is actually a performance improvement then we'll check it out.
That's fine, and this project may as well; my point is there is a distinction between incomplete in time and incomplete in scope. This project is only the former as far as I can tell.
... my point is there is a distinction between incomplete in time and incomplete in scope. This project is only the former as far as I can tell.
No. In the project web page at Github they state that it won't be a drop-in replacement for CPython 2.7. Also from the discussion on HN it is strongly implied (apart from the language features that won't be supported) that it won't support C extensions ever.
You said producing a drop-in replacement for CPython was in the scope of the project. It is not a dispute. I'm just correcting that statement.
The project's scope is more or less CPython without C extensions. The author is unambiguous.
The project's scope is to implement as much CPython 2.7 features as they can that at the same time don't sacrifice performance and without using a GIL. That's actually very far from implementing more or less CPython without C extensions that is different from what you originally stated before. The author is unambiguous. You are not.
What are you confused about?
Nothing. Hopefully you are not confused anymore about what the project is actually trying to achieve.
Considering this has been tried multiple times before with no success (including by, wait for it, Google), what are your reasons for thinking this announcement isn't premature other than "they haven't failed yet"?
I don't see anything critically wrong with the approach, other than the sheer work required to build a quirks-compatible CPython replacement. In particular, the runtime seems similar to CPython except for the GIL. Building on Go's runtime and concurrency primaries should help here as well.
You're awfully sarcastic for someone who doesn't understand "alpha"...
Oh, I don't understand "alpha". Here's the thing: we've seen this over and over again. Implementing a runtime that has better performance for a subset of the language and writing a blog post about it is the easy part. Heck, just a few weeks ago a guy presented a JS runtime that was way better than V8. But once people started to ask him if it implemented the whole language and that if it was actually the case then this was a scientific breakthrough shit did hit the fan.
So if you want to be optimistic about this project then go ahead. But don't come here telling us that "it is just alpha" and "the goals are not incompatible" history have showed us that they actually are and that's why it is a damn hard problem to solve. So when you say this:
... The features you mention are only missing because they haven't yet been implemented...
I just have to laugh hard. "Are only missing". Once those features are implemented is when performance graphs like this one stop being as pretty as they were in the past.
As someone who doesn't use Go or Python: what is the use for this?
vplatt 301 points · Jan 2017
To run Python code as a Go compiled executable. This allows them to avoid the Python Global Interpreter Lock (which severely limits Python's scalability within a single process) and run the Python code using Go modules as if they were Python modules.
Really, unless you're Google or another Python shop interesting in moving to Go, you probably don't have a use for this.
VodkaHaze 40 points · Jan 2017
Getting around the GIL is a huge step. Python's chief weakness is lack of parallel scalability, IMO. Most solutions around this (joblib, dask, etc.) don't feel like complete solutions. Not when, for example, in C++ or Julia you can slap a macro above a loop to make it instantly parallel.
[deleted] 36 points · Jan 2017
For the record, this is not a complete solution either. Grumpy is a tool that compiles a single Python file (no real module support) while eschewing most of Python's dynamic features and supporting very little of its standard library.
weberc2 21 points · Jan 2017
This is misleading. The features you mention are only missing because they haven't yet been implemented, but the intention is to produce a drop-in replacement for CPython. It's only an "incomplete solution" in the sense that it's alpha software, not that its goals are incompatible.
[deleted] 5 points · Jan 2017
OK, but it's not even particularly close to meeting that goal right now, so it's an incomplete solution.
weberc2 24 points · Jan 2017
Yes, it's incomplete in the "it's alpha software", sense; not in the "there's something fundamentally wrong with it" sense. No one is suggesting it's anything other than alpha software.
the_birds_and_bees 9 points · Jan 2017
Although /u/theseoafs didnt mention it, grumpy explicitly doesn't aim for support of the python C API. That's a big limitation and will rule it out for a big chunk of python users.
weberc2 1 points · Jan 2017
I understand and agree. The parallelism support might draw in even more users. Who knows? This project is super nascent.
Spider_pig448 0 points · Jan 2017
No, it's an incomplete implementation of a complete solution.
throwawayco111 -5 points · Jan 2017
LOL. And GNU Hurd is only an "incomplete solution" in the sense that it should not be used in production and there are many missing features, not that its goals are incompatible with being with a usable unix-like system.
Furthermore, you think that this is the first time we hear about a project that is going to remove the GIL and be a drop-in replacement for CPython? /u/theseoafs is not misleading anyone. Once they have a decent beta version supporting all the features CPython has and some benchmarks showing that there is actually a performance improvement then we'll check it out.
weberc2 9 points · Jan 2017
You're awfully sarcastic for someone who doesn't understand "alpha"... Better luck next time.
floodyberry 6 points · Jan 2017
Everyone else who tried this and failed made it to the "alpha" stage as well.
weberc2 1 points · Jan 2017
That's fine, and this project may as well; my point is there is a distinction between incomplete in time and incomplete in scope. This project is only the former as far as I can tell.
throwawayco111 2 points · Jan 2017
No. In the project web page at Github they state that it won't be a drop-in replacement for CPython 2.7. Also from the discussion on HN it is strongly implied (apart from the language features that won't be supported) that it won't support C extensions ever.
weberc2 1 points · Jan 2017
No one disputes this. The project's scope is more or less CPython without C extensions. The author is unambiguous. What are you confused about?
throwawayco111 1 points · Jan 2017
You said producing a drop-in replacement for CPython was in the scope of the project. It is not a dispute. I'm just correcting that statement.
The project's scope is to implement as much CPython 2.7 features as they can that at the same time don't sacrifice performance and without using a GIL. That's actually very far from implementing more or less CPython without C extensions that is different from what you originally stated before. The author is unambiguous. You are not.
Nothing. Hopefully you are not confused anymore about what the project is actually trying to achieve.
weberc2 1 points · Jan 2017
So you were confused by my abbreviated explanation of the scope. I'm glad you're caught up.
load 0 more replies →
floodyberry 1 points · Jan 2017
Considering this has been tried multiple times before with no success (including by, wait for it, Google), what are your reasons for thinking this announcement isn't premature other than "they haven't failed yet"?
weberc2 1 points · Jan 2017
I don't see anything critically wrong with the approach, other than the sheer work required to build a quirks-compatible CPython replacement. In particular, the runtime seems similar to CPython except for the GIL. Building on Go's runtime and concurrency primaries should help here as well.
load 0 more replies →
throwawayco111 0 points · Jan 2017
Oh, I don't understand "alpha". Here's the thing: we've seen this over and over again. Implementing a runtime that has better performance for a subset of the language and writing a blog post about it is the easy part. Heck, just a few weeks ago a guy presented a JS runtime that was way better than V8. But once people started to ask him if it implemented the whole language and that if it was actually the case then this was a scientific breakthrough shit did hit the fan.
So if you want to be optimistic about this project then go ahead. But don't come here telling us that "it is just alpha" and "the goals are not incompatible" history have showed us that they actually are and that's why it is a damn hard problem to solve. So when you say this:
I just have to laugh hard. "Are only missing". Once those features are implemented is when performance graphs like this one stop being as pretty as they were in the past.
EDIT: just checked the discussion at HN with some of the developers involved. Don't expect ever this be a drop-in replacement for CPython.