Mod Support #2 – Embedding Lua into C++

This is part of a series of posts revolving around mod support, the introduction and links to the other posts can be found here.

Our current game in development, Solitude, is a cooperative multiplayer action survival game. Since its a coop game we use a game server that is bundled with the client to allow players to host their own servers and it’s also used for the single player experience. From the very beginning we wanted to give players the maximum mod support we were able to so we decided both the game client and server would be moddable. As our game server is written in C++ that meant embedding and integrating Lua into C++. I’ll write about integrating Lua into Unity as a separate post.

 

A Faster Lua – LuaJIT

Lua is the scripting language we decided to integrate into our game’s scripting layer for code logic control. The aim is for the scripting layer to control all gameplay logic while the core, C++, layer will control generic non-gameplay game logic and time critical algorithms. To gain the speed increases discussed in the last post we decided to go with LuaJIT, which is a Just-In-Time compiler for Lua.

LuaJIT is distributed as source code so to gain a binary version you need to compile it against your compiler of choice. Our server is developed using the C++ compiler MinGW x64 and since we want it to be as cross platform as possible we use the POSIX threads version. What this means is that the threading library used is cross platform instead of using the default Windows threading library when on Windows.

LuaJIT can have some trouble compiling if you’re using the wrong variation of MinGW. For LuaJIT to compile correctly you need to use the SEH and not the SJLJ variation. SEH and SJLJ are two different types of exception handling systems and if you try using the SJLJ version it just won’t compile at all. After that’s compiled you need to link your C++ project against LuaJIT and run some examples to test everything is working. If you are already using core Lua then it’s a simple drop-in replacement.

 

Bringing Object-Orientation to Lua

Lua is a powerful, fast, lightweight, embeddable scripting language but Lua is not an object-oriented language.

The above is a pretty powerful statement, especially since today so many languages are object-oriented. Ideally we would like to use all the advantages of Lua but still have object-orientation. So… is that possible? Yes, by using Lua’s flexible table data structure and some clever libraries. To help us save time we use a library called classes.lua. It allows you to construct classes in Lua that can be passed around and used in a way that feels natural to anyone used to object-orientation.

For an example, at the top of your intended lua class file you’d add:

Door = classes.class();

Provide a constructor:

function Door:init() 
	-- code
end

and then when you want to create an object from the class you’d call:

local door = classes.class(Door);

and it’s as easy as that. You can then call methods on the object like you would in any object-oriented language but with the syntax of:

door:open();

Without classes.lua you wouldn’t be able to do this in Lua (unless you coded it yourself). By bringing in object-orientation it allows us to better bind our core and scripting layers together and make an easy to use and flexible modding framework. I’d highly recommend you look at classes.lua, or the many other OO libraries, for Lua.

 

Binding Lua and C++: The API

Once you have LuaJIT and object-orientation working in your project you need to consider how you want modders to interact with your game. Generally it’s a very good idea to create a level of abstraction so to make things easier for modders. Make C++ do the heavy lifting where possible and allow it to present the results to the scripting layer when requested or required.

When you want to start coding your API you quickly find yourself asking questions like “How on Earth do I get Lua to understand C++ objects? Can my Lua objects be sent back into C++ for processing?”. Welcome to the world of Lua / C++ binding.

Lua does not understand your C++ classes unless you code it to understand it. Binding, in this example, is the process of exposing the C++ classes to Lua. It allows Lua to understand your C++ code and there are two ways to bind C++ and Lua together.

Manual Binding

Manual binding means you write the entire binding yourself. There is a pretty infamous and painfully sharp post about why you should take this approach instead of automatic binding. You can read it here but to summarise they paint automatic binding in a bad light. I happen to disagree with the post but it does raise some valid points. At the end of the day, manual binding is harder and takes much more time to complete. If you’re a small team there will always be the cost vs. benefit argument. For us, manual bindings didn’t bring any real advantages.

Automatic Binding / Glue

Automatic binding / glue means the binding is automatically created for you. Most binding libraries simplify the process massively and provide various advantages depending on the library used. I won’t go into all the libraries as there are too many out there but each take a slightly different approach to binding. There will be a performance hit for using a library but a lot of libraries take performance very seriously and do their best to keep things as fast as possible.

For Solitude we used an automatic binding library called LuaBridge. After a lot of research and evaluation I found that this worked best for us and was the right balance of sensible syntax, functionality and performance. When using LuaBridge, you link it to your C++ project then register the classes you wish to expose to Lua. For example,

void ScriptAPI::registerEventManager(lua_State* state) {
	getGlobalNamespace(state)
			.beginNamespace("event")
				.beginClass<event::EventManager>("EventManager")
					.addFunction("registerEventHandler", &event::EventManager::lua_registerEventHandler)
					.addFunction("registerEvent", &event::EventManager::lua_registerEvent)
					.addFunction("triggerClientEvent", &event::EventManager::lua_triggerClientEvent)
					.addFunction("triggerEvent", &event::EventManager::lua_triggerEvent)
				.endClass()
			.endNamespace();
}

The above code registers the Solitude C++ EventManager so it’s accessible to Lua. You can pick and choose what methods you expose on the class so it’s flexible if you only want to expose select functionality to Lua. You specify the method names and how you’d like to access them from Lua. The method signature is automatically extrapolated by the library. This extrapolation can cause some issues with method overloading but there are workarounds for that. From Lua, you’d access the EventManager as follows:

-- Door was opened so trigger the door opened event
EventManager:triggerEvent("ON_DOOR_OPENED");

Even from this simple example you can already see how powerful and easy it is to bind C++ and Lua together with LuaBridge. Things can become even more powerful when you expose your custom objects and actually create them in Lua instead of C++, then feed them back into C++ for processing. That’ll be for a later blog post though.

After LuaJIT, classes.lua and LuaBridge are in your project you can slowly create your scripting API. Knowing what should be in the core layer and what should be in the scripting layer takes time as there is not always a clear boundary. Hopefully this helps create a clearer understanding of what it takes to embed Lua into C++. Thanks for reading. Feel free to get in contact with me on the comments, email or grab me on Twitter at @CWolf.

 

Links

Lua – Scripting language
LuaJIT – Just-In-Time version of Lua
classes.lua – Object-orientation library for Lua
LuaBridge – C++/Lua binding library
MinGW-w64 C++ Compiler – C++ compiler

Mod Support #1 – Scripting Languages

This is part of a series of posts revolving around mod support, the introduction and links to the other posts can be found here.

For our current game, we’ve decided to provide as much mod support as possible. One large part that will contribute to this is providing a scripting language both on the server and client.

 

Scripting Language

So… what exactly is a scripting language? A scripting language is a programming language used to control another application. In the case of games, and depending on how they are implemented, they can be embedded and used to control the game logic layer. There are advantages and disadvantages of embedding a scripting language in your game, these are:

 

Good: Mod Support

By pushing as much as possible of the game logic into the scripting layer you allow for a mod community to grow around your game. Over the last few years mod support in games has been very lacking and if you provide mod support you’ll be greatly increasing your game’s shelf life and differentiate yourself from the crowd. Modders can become a very loyal and technically helpful addition to your community. Also, who wouldn’t like to see what other people can do with the world you’ve provided them? You make a game because you love the idea and world (hopefully!). Seeing what others can craft with the tools you give them is a great feeling and can even rekindle your original feelings if you’ve unluckily burnt yourself out.

Good: Fast Development Iterations

When developing, if you can do something faster without sacrificing quality – you’ll want to do that. By pushing the game logic out to the scripting layer you’ll be able to reload the scripts without restarting, or recompiling, the game. This is especially useful for developing artificial intelligence and if you’re working with C++. Recompiling your project for every minor edit is exhausting and, after the thousandth time, can send a developer to the edge of insanity and then past the gates of Oblivion.

Bad: Initial Integration

The initial research, selection, embedding and binding of the scripting language to your game core can take a while and is not the easiest thing to do. It may seem like too much effort for developers who are on a very tight schedule and are not experienced enough to get through the process.

Bad: Slower Processing

Any embedded scripting language will almost certainly run slower than the core language you’re running it from. This can eventually have an effect on the end game system requirements but there are steps that can be taken to mitigate the worst cases. Any heavy duty, performance critical code should be run in the core language and then this can be provided to the scripting language as an API call. By doing this, you’re correctly breaking the core logic and the game logic apart and keeping them in their respective places too.

Bad: Exposure of Source

Whilst some scripting languages allow you to pre-compile your scripts to shield your code from prying eyes – you may actually want to share the uncompiled scripts with your community and modders. Your scripts can be the best teaching tool for how to use your scripting API and provide a great start to mods. This matter really depends on how you view your game logic source code and if you’re willing to provide it for the greater good of your game and the modding community, or not.

Bad: Security

If you expose a way for the community to mod your game you will always get someone who will try and do malicious things with it. You will need to sandbox your scripting language runtime and / or whitelist libraries to prevent this. It’s very hard to maintain a complete sandbox but in the end you have to try your best. No one likes a rogue mod deleting save files… or worse.

 

Prerequisites

After deciding that we want to use a scripting language for the above reasons, and feel the above disadvantages aren’t too bad, we need to decide on what language to use. There are lots of choices out there for game scripting languages. For a good game scripting language a few prerequisites need to be met.

 

Performance

Since we aim to provide extensive mod support we are developing as much of the game logic in the scripting layer whilst keeping any processing intensive code and generic client / server logic in the core layer. Due to this the scripting layer needs to be as fast as possible. This is a very important point as stacked inefficiencies will eventually bite you.

Ease of Use

We want modders to be able to get up and running creating mods as quickly as possible. For this we need a scripting language that won’t take new users too long to get into so the lowest possible barrier to entry is required. My early days of development was modding newly released Fallout 1 and 2 so I’d like to give people an easier ride than what I had to do with installing compilers and editing huge files of identifiers.

Security

Since we plan for mods to be client and server side then we need to ensure nothing terribly bad can happen when running mods. Crashing games from bad mods can’t usually be avoided, however, preventing unrestricted access to the end user’s computer is very important. Either a sandbox or whitelisting of functionality is a must here.

Power / Extensibility

Game logic can become very complex so we want to use a language that is either powerful enough to do what we want or can be extended to do what we want. There is no reason to handicap ourselves.

Easy to Embed

At the end of the day, Rogue Vector is a two person company and adding mod support is a huge undertaking that can equally provide huge benefits. We need a language that will be fast for us to embed without spending months trying to integrate and bind it.

 

Language Choice

With the prereqs all agreed on, we need to select which language will be a good fit for us. There are too many scripting languages for me to go into them all, however, they fall into two main categories: dynamically or statically typed. Here are a few languages and the considerations to take into account.

 

Statically Typed

As a rough generalisation, statically typed languages are high level languages that, at compile time, execute extensive checks to ensure type safety and are bound to a data type and an object (or null). Compiler optimisations are also applied in statically typed languages.

mono

Mono is a .NET framework runtime that supports embedding and running of many other languages, even those not from the .NET family such as Java. While Mono is rather large to include in your game it’s surprisingly nimble when it comes to the benchmarks and code execution. In many of the benchmarks it comes out as the fastest scripting system of them all. The main issue is that to use Mono embedded in a commercial product you have to pay a licence fee. This was a deal breaker for us so we stopped investigating it but it has been used amazingly well by Unity and Second Life as their scripting base.

angelscript

AngelScript is a mature language that is very close to C++ in syntax. It hides and handles pointers and memory automatically so that reduces the complexity, however, for a scripting language to remain so similar to C++ may raise the barrier to entry for modders. On the other side, it allows for developers to swap between the scripting and core language without any worries on having to mentally swap development modes. If you want to have C++ style scripting and your core game is C++ then AngelScript may be your winner here. It has a very clean integration and so that reduces the need for a binding library, which is a big plus. It’s used successfully in games such as SOMA, Amnesia and Overgrowth.

Thanks to Ian Thomas (@wildwinter) from Frictional Games and GameDevWales for feedback on AngelScript.

 

Dynamically Typed

Again as a rough generalisation, dynamically typed languages are high level languages that, at runtime, execute many common programming behaviors that statically typed languages perform during compilation. They are not bound to a data type but only to the object (or null).

lua

Lua is generally seen as the games industry scripting language of choice. It’s very lightweight, fast and very portable. Lua becomes even faster when using the JIT compiler called LuaJIT. While it’s only a procedural language it does have a very flexible data structure that allow for object oriented programming techniques to be used. Extension libraries can bring that support even closer to full object orientation. The language itself has a few things to get used to, such as indexes starting at 1 and not 0 and missing expected keywords like ‘continue’, but it’s not too bad to get used to. All in all, it’s easy to see why Lua is used frequently for embedded game scripting and has been used in games such as World of Warcraft, Freelancer, Garry’s Mod and Natural Selection 2.

python

Python is another popular procedural language similar to Lua. It’s a lot heavier due to it including all the nuts and bolts that Lua does not. It also runs slower than Lua so this is something to keep in mind. I have an issue with the white space nature of Python but that’s more a personal dislike more than anything else. In the end, Python’s focus is wider than Lua for better or worse. Python has been successfully used for game scripting, like in Civilisation 4 and Battlefield 2, so that speaks for itself.

v8

Google’s V8 Javascript-based scripting language seems to be a faster choice than core Lua, however, it’s slower than LuaJIT according to the sources I’ve found. I haven’t been able to confirm this with hard benchmarks as I don’t have time so I’d just say it’s pretty fast. Binding support seems to be lacking and it’s an uphill struggle to get it embedded and working the way you want, especially if you want to pass complex objects between the core language and V8. I couldn’t find any games that currently use V8 as their scripting language but, on the brightside, a lot of people know Javascript syntax.

 

Our Conclusion

We decided to back Lua (LuaJIT) for our scripting language of choice. The main reasons are already mentioned above but also we needed a scripting language that is consistent between the server and client. While Unity has no native scripting support, there are assets that help integrate LuaJIT in nicely. Other scripting languages would be harder to integrate into our game client. At the end of the day most of the scripting languages highlighted have been used very successfully so personal preference is also a key deciding factor.

For a more indepth look at some of these languages and a set of benchmarks see the blog post at upvoid.com. A wider set of benchmarks are found here and here. My next post will go into more details of LuaJIT embedding and binding into a C/C++ game core.

Thanks for reading. Feel free to get in contact with me on the comments, email or grab me on Twitter at @CWolf.

Interfacing with UI #4 – Coherent UI

February 5th, 2016

This is part of a series of posts revolving around user interface design and development, the introduction and links to the other posts can be found here. Last I wrote about user interfaces I discussed the new Unity UI system and I wrote about our process of porting from Daikon Forge to it. That was a year and a half ago and a lot has changed since then. To keep things interesting we decided to move from Unity UI (yet another move?!) to Coherent UI and I’ll explain why we did it. Why Move… Again?!... (read more)

@SolitudeGame: Status update: Fuel reserves low. Asteroid mining facility detected on sensors. No response to our communications. On approach station appears to be abandoned and running on emergency power only. Away mission approved. Mission objective: Search and salvage - fuel is a priority.

23/03/2022 @ 11:00am UTC

@RogueVec: And so it begins! #RebootDevelop

19/04/2018 @ 8:05am UTC

@RogueVec: We'll be at @RebootDevelop this year. We can't wait! If you want to hang out just give us a shout! #RebootDevelop2018 #GameDev

16/04/2018 @ 12:06pm UTC

@SolitudeGame: Fullscreen terminals allow you to hook into your ship's guns for fine control! Moddable gun modules, terminals and UI! https://t.co/B5N01jrA70 #GameDev

8/12/2017 @ 4:58pm UTC

@CWolf: Woo! And, now we have a cross-compiled (nix --> win64) @SolitudeGame server in our build and deploy pipeline #RV #GameDev

28/11/2017 @ 3:39pm UTC