Lua written in modern C++ (>= 14)?
I work on Factorio (written in C++) and we use Lua as the script language for mods. So, from time to time I interact with the C implementation of Lua and my experience with it hasn't been the most pleasant.
It got me wondering - has anyone re-written it in modern C++? Everything I find online is "how to use Lua in C++" or "C++ wrapper for Lua" and they all use the basic C implementation provided from the official Lua website.
I've had quite a lot of experience with the internals of the C Lua implementation and from what I've seen it wouldn't be hard to re-write it using modern C++ - maybe 2~ weeks time. But from what I can find nobody has done it - which also makes me wonder if I'm missing something.
Part of me wants to do it just to say I did it and the other part of me wants to do it to see if I can make it faster than the C version: I think I can - not because "C++ is faster" but because I can mutate/experiment with the C++ version far easier and know changes I make are safe with how the rest of the implementation operates.
127 comments · 96% upvoted · other discussions
Hey, I just wanted to let you know that I understand perfectly where you are coming from. I was the lead programmer for Starbound, which has an extensive Lua modding API, and interacting with the Lua C API has... an enormous impedance mismatch, to put it mildly. No, LuaJIT does not help, and not every platform allows you to JIT.
It would take me a long time to put into words all of the issues I have with the Lua C API, but you've mentioned a lot of them. Stack manipulation incurs a lot of real overhead, and I find myself just wanting to hold onto actual internal pointers. It makes interacting with the garbage collector and holding values outside of the Lua state difficult, requiring constructs like the registry, which incurs even more overhead.
Also, to the extent that you care about safety, Lua's error handling strategy brings in even more problems. I currently do most of my work in Rust, and making a Rust bindings layer for Lua that is safe has taken far more work than it should, and the result is far slower than I would like. The Lua C API, ESPECIALLY with the whole __gc madness, will just longjmp on you at the drop of a hat! Also, you have to constantly call lua_checkstack, and (surprise!) luaL_xxxx calls will use stack space that they 1) don't document, and 2) require that you call lua_checkstack to protect against. I'm POSITIVE I could do a better job speed wise with rlua, but I question HOW much of a better job is possible, if you limit yourself to an actually safe API.
I do, however, think that the Lua C API is ultimately very clever. Ultimately, it IS quite a lot simpler than other language's APIs, and I'm sure if you're writing nothing but C and the occasional setjmp / longjmp is not a problem for you, it's quite nice even. However, when you care about safety, or hit an optimization wall because of the overhead of the API, it starts to.. lose its charm somewhat.
I haven't interacted with the Python C API in quite a while, but as I remember it, it was much more the style that you would want when binding a large complex API to it. You get pointers and manually inc / dec the reference count on them, and this allowed you to, with very low overhead, maintain handles to internal Python objects. Also, the error handling strategy is NOT based on setjmp / longjmp, but rather error codes and a global error state like unix errno.
I'm ultimately not sure whether the fact that the Lua API is C is the problem, or whether it's the specific choices that went into it that are the problem. I could see, at least theoretically, that a C API in the style of Python's could be quite low overhead, if only because you can more or less take the C++ API you're describing (or any C++ API really) and, with great effort, write something that's the moral equivalent of it in C.
If such a Lua interpreter existed, and had a C API, I might use it!
(Full disclosure, I have actually seriously considered writing my own Lua interpreter in Rust for precisely the reasons you describe, and am STILL seriously considering it.)
smog_alado 3 points · Feb 2018
The reasons why Lua's API are the way it is are pretty much why you guessed.
Not exposing pointers to C is primarily about garbage collection and secondarily about isolating C from Lua. Lua doesn't use reference counting so it cannot use a system similar to the one used by Python and Perl. (switching the original GC to reference counting could also have a negative performance impact, since reference counting has nontrivial overhead while the program is running). And if you don't want to have reference counting things start to look less apealing. One option is to have the GC inspect the C stack like Ruby does but this will result in a conservative GC and would also need nonportable assembly code. Another option would be to have multiple different reference types, like Java's native interface but that adds a lot of complexity as well.
As for the stack manipulation, I think the main reason for that is it avoids needing to store function arguments and result lists in garbage-collected tables. I agree with you that lua_checkstack is super annoying though...
As for the problem of speed, one avenue I am currently exploring is to have a third programming language to sit between Lua and C. The compiler for this language is allowed to mess with the Lua internals directly (which can be good for speed) but the programmer only ever interacts with the new high level language and never messes with the Lua internals and garbage collection themselves (which is good for simplicity, safety and version compatibility). But at this point it is still in very early prototype stage...