Grumpy: Go running Python! by Google
248 comments · 92% upvoted · source · other discussions
comments
semi_colon
gin_and_toxic
This is crazy awesome. Currently only works for Python 2.7 though.
A lot of good discussions on hacker news: https://news.ycombinator.com/item?id=13319904
[deleted]
[deleted]
Vorticity
Meaning that the Python community is about to get a bit smaller...
[deleted]
They are the ones who made Python popular though, Python got popular because people learned that it was used by Google. Now Google is shining their light to Go.
I tried Go, but I didn't like it that much. It feels like the only new thing it brings to the table it's the go routines. Also it feels like it's very simple (perhaps too simple) and very opinionated (e.g. indenting), or how code is stored. It seems like it was designed just for Google's needs.
kaeshiwaza
It was designed to replace C++ and finally replace Python.
To be simple, opinionated, with top batteries included, fast and easy to deploy. All of this together are quite new on the table ! It seems it was designed just for me :-) !
jrandomcoder
Don't worry, they don't actually use this in production.
spacebandido
That's what they all said about Node
ArmoredPancake
...yet.
recycled_ideas
No, because this won't do that for them. The code base is still going to be python 2.7.
What it'll do is allow them to get past the limitations of Python 2.7 without having to rewrite hundreds of thousands of lines of code.
[deleted]
If you want to rewrite a large monolithic code base from one language to another there's problem that you would have to finish the entire work before you can use the code in new language.
So what can you do? You write code to do the automatic conversion for you. The problem is that the generated code is not maintainable, but at least now you can make calls to it. It also allows you to rewrite module by module.
Google's Unladen Swallow project did not succeed, they also don't seem seem to move away from Python 2.7, for god's sake even this project is Python 2.7 only, and ticket about Python 3 support was closed with change in documentation that specifically says Grumpy targets Python 2.7 only. It can't be any more clearer that Google is moving away from Python to Go.
rjcarr
This is really cool. Seems they didn't like jython, or what was similarly available, and wrote what they wanted for go. Makes sense.
But in the beginning they say this:
but we always run up against the same issue: it's very difficult to make concurrent workloads perform well on CPython.
How is it even possible with the GIL? How can they handle millions of requests per second (as they say)?
Hshskwkk
From r/python : ""
Quoting the author of Grumpy on Hacker News:
" ""The idea is to continue to write code in Python. (...) ""That said, there is the possibility of rewriting bits and pieces in Go (e.g. performance critical stuff) and then call into it from Python. Sort of a hybrid approach.
"
Also: " I'd like to support 3.x at some point. "
However: " We do iteratively rewrite components as well. We are pursuing multiple strategies " ""
censored_username
edit: Thinking about it a bit more, they probably went with the locking per individual object route.
This seems cool and all, but couldn't they have just solved the problem by just not having the GIL and accepting data races in Cpython?
The GIL exists because python wanted to be free from data races. To achieve that, there were multiple options, each of which had to satisfy the constraint that no two threads would access a python object at the same time. The GIL achieves this in a simple manner. There's only one lock, and to access any python object you need to hold it. The problem with this is of course that it means that no two threads can access python objects at the same time even while these objects have nothing to do with each other.
Another solution is pyobject-level locking of course. Various attempts at this have been made, but they all had the shared problem that while they increase throughput at high amounts of threads, they significantly lower single-thread performance compared to the GIL.
Various other strategies have been tried, but none have been able to guarantee being data-race free without sacrificing single-thread performance. An often used strategy has been to move work that could be accelerated significantly into extension libaries, where the GIL could be dropped as long as the python objects weren't accessed. Unfortunately, this doesn't really work for web services.
Go doesn't have an answer to race conditions either. It's where Go's safety story falls a bit apart. So I'm a bit confused about what they've done. If they just transpile python code into go code, all they made is a python runtime with the possibility of data races. If they wanted that they could've just as well just ripped the GIL right out of CPython and called it a day.
What they could've also done is to build in individual locking for all objects that could be accessed by other threads. This would explain their pretty disastrous performance (only 50% performance compared to CPython when single threaded, and only gaining a 2.5x speedup when using 8 threads compared to CPython single thread. Keep in mind that this is compiled versus interpreted!).
If they did this, I'm left wondering why they didn't make these changes in CPython itself. This would have conserved the ability to call into C extension modules. If this is the case, it seems that the focus of the project is more being able to call into go from python.
nostrademons
The data races that the GIL protects against are primarily the refcounts of Python objects, which must be updated whenever a variable is assigned or a parameter is passed into a function. A data race there means either a memory leak or a double-free, as the refcount would no longer be in sync with the number of actual references. Either one would make Python basically unusable: you would have simple object assignments randomly corrupting memory as they try to objects that have already been freed.
Go uses a GC; while its data race story is a lot weaker than say Rust or Erlang, at least a data race there results in wrong answers and a panic, not silent memory corruption.
censored_username
The GIL doesn't only protect against refcount changes (it's indeed the most important thing though). It also protects against concurrent modification problems of dictionaries, lists, etc. which are also quite critical to python. They can have avoided this by making these objects all threadsafe which would explain the terrible single-thread performance.
While the GIL simplifies python's GC (refcounting with cycle detection) design (the GC can just acquire the GIL when it needs to clean up cycles). It is not necessary. If there was no GIL the refcounting would have to be atomic, but otherwise not much changes.
blablahblah
That's true. The main reason CPython hasn't gotten rid of the GIL is because no one has come up with a way to do it without breaking existing C extensions (which rely on the ref-counting) and without hurting single thread performance (since every variable assignment needs to acquire a lock).
funny_falcon
Go has no thread-safe dictionaries or lists. In fact, it may happily segfault if you try to modify map from different goroutines. You ought to protect concurrent modification by yourself.
I've heard, Java and C# also doesn't protect their default datastructures, but rather they have separate concurrent in a standard library.
Python can choose this way, but Guido just do not want. And it will break C extensions.
Damien0
Go's build tool has a race detector, and for calling into C from Go there is cgo. With that plus the ability to call into Go from Python in mind, I can see the benefit of this custom runtime.
Giggaflop
Last I saw, granted a while ago, cgo wasn't a recommended solution to calling C from go and was "considered harmful"
Damien0
Hmm, I wouldn't say harmful per se, but as is often quoted in the community "Cgo is not Go". As long as it's done carefully it can be useful. I think what happened is that early on, people treated it like a more fleshed out FFI solution and wrote less than optimal Go code as a result.
BillyBoyBill
It's hard to say without looking at the code, but I imagine they implemented thread-safe Go versions of the Python built in types. Your code that assumes Python data can be accessed concurrently from multiple threads still works. This is the moral equivalent of the per-object locks you mention (and, in fact, Grumpy performance matches what you say about that implementation).
This is obviously completely different than just ripping out the GIL.
I'm not sure what you mean about Go's thread safety falling apart, but it's not relevant here anyways --- C's built-in types are not thread safe, but you can build a safe Python implementation on top of them.
censored_username
This is obviously completely different than just ripping out the GIL.
True. As you probably noticed I changed a bit in opinion while writing that post. Originally based on their description I thought they were just straight transpiling code but looking at the performance they get it has to be composed from threadsafe types (aka per-object locking).
[deleted]
[deleted]
singron
Pypy also has a GIL, just like CPython, so it still can't execute python threads in parallel. This runs on the Go runtime, which uses lightweight threads (goroutines) and a concurrent garbage collector to run many python threads in parallel.
It's like if there was an ahead-of-time pypy combined with stackless python and it didn't have a GIL.
Another advantage specifically for Google is that it allows more interop between python and Go code while they transition.
[deleted]
Why cant python threads just access any object at the same time without the GIL or any per-object implicit locking, and then it's up to the developer to explicitly add locks/mutexes where needed, just like in every language?
Sure it would require a rewrite of applications to add this explicit locking, but it would still work well on regular GIL CPython (because there would be very little that you really need to explicitly lock). So you would have two python distributions really, one without the GIL to run parallel and explicitly locking code, and the regular one that can run code written for the first (but it wont really be parallel)
censored_username
Why cant python threads just access any object at the same time without the GIL or any per-object implicit locking, and then it's up to the developer to explicitly add locks/mutexes where needed, just like in every language?
Because this would make python an absolute pain in the ass to use due to its extreme dynamism.
Don't forget, in python functions are also just objects, and they can be mutated. So you'd need to lock for any function call (especially builtin functions or stdlib functions which could be used by another thread at the same time). Even accessing an attribute from a shared module could require locking. You'd basically have to lock at every point to do anything.
tonnynerd
I'm not sure I understand what you're saying about the GIL being about avoiding data races. What the threading module from the standard lib is for, then?
censored_username
the GIL (global interpreter lock) avoids data races in python object internals by making concurrent accesses of the same python objects impossible. In effect, it means that all actions by different threads in python will always happen after each other. It is an implementation detail of CPython, not part of the core language.
The threading module provides tools for spawning threads and higher level concurrency primitives (while the GIL means that accesses to the same python object won't coincide, there still need to be primitives to lock out sequences of multiple accesses).
[deleted]
Once we started going down the rabbit hole, Go seemed like an obvious choice of platform since its operational characteristics align well with our use case
"After lots of deep research, we decided to use that language that we made ourselves!"
sigma914
Well, yeh, they have one major problem, serving massive numbers of concurrent users. They've dealt with it in a number of ways, one of which was developing a language specifically for writing massively concurrent code. They've run into their usual problem again, so they're using the tool that was developed specifically for that task to do it.
Google has a lot of similarly shaped nails. They made a hammer that does a good job of hitting that particular shape of nail. They are now using that very specifically tuned hammer on more of the nails it's designed for.
myringotomy
I like how you completely ignored the content of the quote and then made up something and then pretended the author said it even though it has nothing to do with anything.
atomheartother
I really really should seriously learn Go
Edit 1 month later: I have learned go. It's really neat, you should do it too.
[deleted]
[deleted]
atomheartother
I just like learning and Go looks neat for small apps too. I also personally don't like python too much and go looks neater
SnowdensOfYesteryear
Meh, it's all right. I like the language, but it gets annoying when you want to do something that the language doesn't want you to do, like discarding buffered data in channels or timing on a channel receive.
Also the syntax is pretty irritating because it isn't like any other language. That felt like a change for the sake of it.
[deleted]
It is actually based on a language called Oberon, I think. Still I think it is close enough to C not to be alien and different from Rust or C++, you can look at any Go code and know what it does right away. It's almost a C with packages and without ; and variable types go after their names instead of before and you explicitly declare functions with "func".
immerc
Yet another example of "Go" being a terrible choice for a language name.
I had to go read the page to understand that they were talking about the Go language. Did they mean that I should "Go run python"?
[deleted]
This just makes me think: when is python going to get its shit together and get concurrency working properly?
stormcrowsx
Not every language needs to do everything. It'll be great if they improve concurrency but Python is in my opinion best used as a glue language or a language for rapid development. Other people probably have other great uses for it but no one sane uses Python for its performance. If concurrency is your thing, probably shouldn't be writing in Python, there's Go or Java that work great in that realm.
anedisi
when i need some service with concurrency and asynchronicity i write that in javascript and run it in nodejs.
it worked ok for now.
__eastwood
This breaks down when you need CPU bound concurrent tasks
iconoclaus
you could ask this of almost any dynamically typed language. btw, i think you meant parallelism, not concurrency.
renrutal
Dynamic typing and concurrency done right
Elixir
iconoclaus
that's actually a really good counter-example afaik
kirbyfan64sos
I would be grumpy if I had to use Go, too.
ReflectiveTeaTowel
I initially misread that as 'Go ruining Python!' and spent the first few paragraphs of the article really confused.
fatzombi
it would be awesome if i could run it as services for existing python projects!
Pharisaeus
They didn't write how compatible is it. There is already for example ShedSkin which translates Python to C, but it can handle only some subset of the language.
Also if they want to sell this as something "fast" the comparison should have been with PyPy and Shedskin and not with Cpython.
imhotap
Is there a chance of transpiling (parts of) certbot/letsencrypt to Go? I'm guessing not, because at least one C dependency (eg. aureas for parsing and manipulating Apache config files) is being used, but I don't know enough about Python to judge.
wotamRobin
Very cool. I'd like to see how it compares to Cython (not CPython).
As someone who doesn't use Go or Python: what is the use for this?
vplatt 305 points · Jan 2017
To run Python code as a Go compiled executable. This allows them to avoid the Python Global Interpreter Lock (which severely limits Python's scalability within a single process) and run the Python code using Go modules as if they were Python modules.
Really, unless you're Google or another Python shop interesting in moving to Go, you probably don't have a use for this.
semi_colon 31 points · Jan 2017
Thanks, great explanation.
VodkaHaze 34 points · Jan 2017
Getting around the GIL is a huge step. Python's chief weakness is lack of parallel scalability, IMO. Most solutions around this (joblib, dask, etc.) don't feel like complete solutions. Not when, for example, in C++ or Julia you can slap a macro above a loop to make it instantly parallel.
[deleted] 38 points · Jan 2017
For the record, this is not a complete solution either. Grumpy is a tool that compiles a single Python file (no real module support) while eschewing most of Python's dynamic features and supporting very little of its standard library.
weberc2 22 points · Jan 2017
This is misleading. The features you mention are only missing because they haven't yet been implemented, but the intention is to produce a drop-in replacement for CPython. It's only an "incomplete solution" in the sense that it's alpha software, not that its goals are incompatible.
[deleted] 6 points · Jan 2017
OK, but it's not even particularly close to meeting that goal right now, so it's an incomplete solution.
weberc2 23 points · Jan 2017
Yes, it's incomplete in the "it's alpha software", sense; not in the "there's something fundamentally wrong with it" sense. No one is suggesting it's anything other than alpha software.
the_birds_and_bees 9 points · Jan 2017
Although /u/theseoafs didnt mention it, grumpy explicitly doesn't aim for support of the python C API. That's a big limitation and will rule it out for a big chunk of python users.
load 1 more replies →
load 1 more replies →
load 15 more replies →
[deleted] 26 points · Jan 2017
It doesn't need GIL, because it offers subset of functionality that python has. It is not even capable of compiling all python standard library.
This code is for Google to move away from python. They can include python libraries in their go code and then one by one rewrite it in go.
weberc2 6 points · Jan 2017
The only design decision that prevents it from compiling the standard library is the lack of C support. The solution is to reimplement those modules in Go or Python. Anything else that doesn't compile is due to bugs in this alpha implementation (as far as I can tell, anyway).
[deleted] 2 points · Jan 2017
PyPy also doesn't natively support C extension and also needs GIL. C support is one of reason of GIL's existence but not the only one.
load 1 more replies →
sultry_somnambulist 10 points · Jan 2017
Speaking of this, what's the general trajectory of go? Is it worth dabbling into at this point? People seem to be very divided over the language.
[deleted] 11 points · Jan 2017
People seem to be divided, because while the language is not bad, it's nothing special (seems like the only significant thing it brings to the table is the go routines, but other languages are adding similar functionality), there's a bit of hype going on because it's backed by Google.
It also initially was trying to be marketed as a C replacement, but it is nowhere close to fit the niche that C fits. Rust seems like a language that has potential to replace C, but it is too early to say.
Go is more on level of Python, Java etc, although a bit lower level than them. Some people compared it to D, but I never used it, so can't comment. The goal of this language is to be a very simple and opinionated language, some people like that some don't.
If you are asking the question you can try and see if you like it, it has low learning curve, so it's not hard to learn it, and then you will have your opinion about it.
TankorSmash 2 points · Jan 2017
I've heard it's got a bit of boilerplate and written to be as easy to write as possible to help with the incredible amount of new talent Google gets out of schools.
Only a rumour though.
the_birds_and_bees 3 points · Jan 2017
Id say it's more about making it easy to read and maintain. The google code base is 2 billion+ LOC, so designing a language that eases the burden of maintaining a code base of that size seems like a sensible goal.
load 1 more replies →
[deleted] 2 points · Jan 2017
If you like it, then do it. No reason not to.
The community is sizable and there's a ton of tooling so it's not really like you're taking a risk.
weberc2 4 points · Jan 2017
I think so. If you have prior programming experience and a free afternoon, you should be able to start writing real (i.e., non-toy) programs. Check out http://tour.golang.org.
Go has a lot of quirks--most notably, no generics--which turn off a lot of folks. But if you want to build performant, scalable software quickly, Go is worth a look.
th3_pund1t 5 points · Jan 2017
Not entirely true. You should be familiar with pointer based languages like C. If you come from a python/ruby/Java experience you will spend a whole lot longer to get used to pointers.
Source: work had a hackathon. I decided to code in go. Spent a lot of time correcting my code related to pointers before it would work.
recycled_ideas 4 points · Jan 2017
I don't know about python and ruby, but you can actually run into a lot of really weird bugs in Java if you don't have at least a cursory understanding of how pointers and references work.
The big problem that companies like Google and Facebook and the like are constantly trying to solve is that folks who know dynamic languages like Python or PHP or Ruby are a dime a dozen, but these languages really don't scale well to even ordinary enterprise scales let alone the kind of volume these companies see.
That's why we see so many transpilers and why Twitter is written in PHP even though the general industry consensus is that it's probably one of the worst non joke languages in existence.
load 1 more replies →
load 3 more replies →
load 1 more replies →
load 12 more replies →
MightyCreak 2 points · Jan 2017
Does this mean that you can't interpret two Python scripts at the same time because of GIL, but once interpreted, they can run in parallel?
ShotgunToothpaste 23 points · Jan 2017
The global interpreter lock exists on a per-process basis in Python. Any two python scripts (including two instances of the same script) can be run in two different processes and do whatever. However, if one script starts multiple threads, only one thread will execute concurrently (the thread which has the interpreter lock).
This is a CPython implementation detail, and other Python runtimes need not have this limitation. The reason for GIL is because CPython's memory management is not thread-safe.
https://wiki.python.org/moin/GlobalInterpreterLock
btgeekboy 9 points · Jan 2017
To add to this, there is a project to remove the GIL from CPython. However, it's not currently as fast as with the GIL. More info: https://lwn.net/Articles/689548/
[deleted] 3 points · Jan 2017
[deleted]
load 3 more replies →
MightyCreak 2 points · Jan 2017
Thanks for this answer!
phpdevster 2 points · Jan 2017
So is it fair to say that this is just a way for them to give Python the performance characteristics of Go without having to actually rewrite into Go code?
vplatt 2 points · Jan 2017
Yes, more or less. Python is still dynamically typed, so there has to be some runtime overhead around that, but that's got to be insignificant compared to the advantages around getting away from the GIL.
load 6 more replies →
karma_vacuum123 34 points · Jan 2017
since this targets Google's particular version and feature constraints, it seems like a bulk migration tool by Google for Google. they don't seem to be selling it as a general purpose execution platform
still a neat accomplishment of course
weberc2 14 points · Jan 2017
I'm guessing it's an ambitious 20% time project that was open sourced, but this is just a hunch.
Uncaffeinated 4 points · Jan 2017
That's what I'd guess too. If this were an official project at Google, they would have actually finished it, instead of releasing what sounds like an experimental hobby project.
yes_or_gnome 11 points · Jan 2017
Google is minimizing it's use of Python and ramping up use of Go. However, they don't want to rewrite everything from Python to Go, so they created
grumpy. Without support for C extensions, exec, eval, etc., it's not likely going to all that useful for many people.google_you 3 points · Jan 2017
Compile node.js to python and then to native for full scale.
rjcarr 5 points · Jan 2017
Read the article. Python doesn't scale well under load but go does. Rather than go with an alternate python runtime, or rewrite everything in something else, they wrote a python runtime for go.
jdmulloy 3 points · Jan 2017
It's not an interpreter, it's a compiler that compiles Python code into a go executable. Once you've compiled it you have a binary you can stick on any system that has go libs installed but you don't have to have this project installed to run the binary
load 1 more replies →
load 9 more replies →
weberc2 2 points · Jan 2017
Concurrency seems to be one of the primary objectives, but also straightforward interop between Go and Python, which naturally enables a code base to iteratively move from one language to another.
[deleted] 1 points · Jan 2017
[removed]
semi_colon 2 points · Jan 2017
Ooh, I love Kraftwerk!