edit: Thinking about it a bit more, they probably went with the locking per individual object route.
This seems cool and all, but couldn't they have just solved the problem by just not having the GIL and accepting data races in Cpython?
The GIL exists because python wanted to be free from data races. To achieve that, there were multiple options, each of which had to satisfy the constraint that no two threads would access a python object at the same time. The GIL achieves this in a simple manner. There's only one lock, and to access any python object you need to hold it. The problem with this is of course that it means that no two threads can access python objects at the same time even while these objects have nothing to do with each other.
Another solution is pyobject-level locking of course. Various attempts at this have been made, but they all had the shared problem that while they increase throughput at high amounts of threads, they significantly lower single-thread performance compared to the GIL.
Various other strategies have been tried, but none have been able to guarantee being data-race free without sacrificing single-thread performance. An often used strategy has been to move work that could be accelerated significantly into extension libaries, where the GIL could be dropped as long as the python objects weren't accessed. Unfortunately, this doesn't really work for web services.
Go doesn't have an answer to race conditions either. It's where Go's safety story falls a bit apart. So I'm a bit confused about what they've done. If they just transpile python code into go code, all they made is a python runtime with the possibility of data races. If they wanted that they could've just as well just ripped the GIL right out of CPython and called it a day.
What they could've also done is to build in individual locking for all objects that could be accessed by other threads. This would explain their pretty disastrous performance (only 50% performance compared to CPython when single threaded, and only gaining a 2.5x speedup when using 8 threads compared to CPython single thread. Keep in mind that this is compiled versus interpreted!).
If they did this, I'm left wondering why they didn't make these changes in CPython itself. This would have conserved the ability to call into C extension modules. If this is the case, it seems that the focus of the project is more being able to call into go from python.
Go's build tool has a race detector, and for calling into C from Go there is cgo. With that plus the ability to call into Go from Python in mind, I can see the benefit of this custom runtime.
Hmm, I wouldn't say harmful per se, but as is often quoted in the community "Cgo is not Go". As long as it's done carefully it can be useful. I think what happened is that early on, people treated it like a more fleshed out FFI solution and wrote less than optimal Go code as a result.
edit: Thinking about it a bit more, they probably went with the locking per individual object route.
This seems cool and all, but couldn't they have just solved the problem by just not having the GIL and accepting data races in Cpython?
The GIL exists because python wanted to be free from data races. To achieve that, there were multiple options, each of which had to satisfy the constraint that no two threads would access a python object at the same time. The GIL achieves this in a simple manner. There's only one lock, and to access any python object you need to hold it. The problem with this is of course that it means that no two threads can access python objects at the same time even while these objects have nothing to do with each other.
Another solution is pyobject-level locking of course. Various attempts at this have been made, but they all had the shared problem that while they increase throughput at high amounts of threads, they significantly lower single-thread performance compared to the GIL.
Various other strategies have been tried, but none have been able to guarantee being data-race free without sacrificing single-thread performance. An often used strategy has been to move work that could be accelerated significantly into extension libaries, where the GIL could be dropped as long as the python objects weren't accessed. Unfortunately, this doesn't really work for web services.
Go doesn't have an answer to race conditions either. It's where Go's safety story falls a bit apart. So I'm a bit confused about what they've done. If they just transpile python code into go code, all they made is a python runtime with the possibility of data races. If they wanted that they could've just as well just ripped the GIL right out of CPython and called it a day.
What they could've also done is to build in individual locking for all objects that could be accessed by other threads. This would explain their pretty disastrous performance (only 50% performance compared to CPython when single threaded, and only gaining a 2.5x speedup when using 8 threads compared to CPython single thread. Keep in mind that this is compiled versus interpreted!).
If they did this, I'm left wondering why they didn't make these changes in CPython itself. This would have conserved the ability to call into C extension modules. If this is the case, it seems that the focus of the project is more being able to call into go from python.
Damien0 13 points · Jan 2017
Go's build tool has a race detector, and for calling into C from Go there is cgo. With that plus the ability to call into Go from Python in mind, I can see the benefit of this custom runtime.
Giggaflop 2 points · Jan 2017
Last I saw, granted a while ago, cgo wasn't a recommended solution to calling C from go and was "considered harmful"
Damien0 4 points · Jan 2017
Hmm, I wouldn't say harmful per se, but as is often quoted in the community "Cgo is not Go". As long as it's done carefully it can be useful. I think what happened is that early on, people treated it like a more fleshed out FFI solution and wrote less than optimal Go code as a result.
[deleted] 1 points · Jan 2017
[deleted]
Damien0 1 points · Feb 2017
Oops, missed this. Slow compared to what? I'm curious if there are benchmarks floating around