Tencent open source mobile hot update program, Lua programming under Unity3D

Write in front

xLua is a Lua programming solution under Unity3D. Since it was promoted in early 2016, it has been used in more than ten Tencent self-developed games, and has been widely acclaimed for its good performance, ease of use, and expandability. Tencent has now opened xLua to GitHub.

At the end of December 2016, xLua has just achieved a new breakthrough: the full platform supports the use of Lua to fix C# code bugs.

At present, most of Lua's hot update programs under Unity are required to be updated by the Lua language from the very beginning. The inadequacies are:

1. The access cost is high, and some projects have been written in C#. At this time, the access needs to be re-implemented with Lua where it needs to be hot.
2. Even if it is accessed from the beginning, there are also problems in developing both languages ​​at the same time;
3. Lua performance is inferior to C#;

The xLua Hotfix technical support replaces a C# implementation (function, operator, property, event, or entire class) with a Lua implementation at runtime, meaning that you can:

1. Usually use C# development;
2. Running is also C#, performance spikes Lua;
3. Where there is a bug, a Lua script fix is ​​issued. The Lua implementation can be replaced with the correct C# implementation when the whole update is next. Even when the update is completed, the game can be restarted without restarting.

This new feature iOS, Android, Window, Mac passed the test, is currently doing some ease of use optimization.

So, what kind of technology is xLua of Tencent's open source? Why is it so designed? Even more interesting is the performance of xLua? With these questions, InfoQ interviewed the author and compiled the content.

Guest Profile

Che Xungsheng graduated in 2005, worked for Huawei for 6 years, followed several years in two game startup companies, and entered the Tencent Mutual Entertainment Public Component Center in 15 years. Currently focusing on the development of some game common components.

technical background

Tencent has been working on mobile games. For the projects I know, most game engines are Unity3D, and a few use coco2d.

What games did the xLua plugin actually use? Although xLua completed its first version in March 2015, due to the fact that the project team's warmer consciousness was not very common and the demand was not very strong, xLua's development resources were transferred to more urgent projects. Until the end of the year 15 was officially integrated into our apollo mobile game development framework, the first project of xLua was ushered in. So far, there are more than a dozen projects we have applied xLua, some of which are heavyweight IP, or products built according to star standards.

Prior to xLua, in the face of iOS's inability to hot-update issues, there were ulua's, useful slua's, and project-based scripting languages. However, there weren't many projects that were updated at the time.

Hot update process

The hot update process of the mobile game is very simple. It is just to check if there is a new version of the file at startup. If there is, the old file will be overwritten and then started.

If the downloaded file is a picture, the model is no problem, but if Unity's native code logic, whether it is the previous Mono AOT or later il2cpp, is compiled into a native code, iOS can not run.

The solution is one, do not use native code, do not use jit, parse the implementation on it. All hot-update support programs, including xLua, implement code logic thermal updates through "parse execution."

Hello world from xLua

(1) Three lines of code running lua script

A complete example requires only 3 lines of code:

After downloading xLua and unzipping it into the Unity Project Assets directory, create a MonoBehaviour and drag it to the scene. Add these three lines to Start:

XLua.LuaEnv luaenv = new XLua.LuaEnv();
luaenv.DoString("CS.UnityEngine.Debug.Log('hello world')");
luaenv.Dispose();

Run to see the console print hello world.

1. The first and third lines respectively create and destroy LuaEnv. The so-called LuaEnv can be understood as a lua virtual machine, and often the entire project can be a virtual machine:
2. DoString can be any legal lua code. In the example, the UnityEngine.Debug.Log interface is used to print a log (C#'s static functions are available directly under CS).

(2) C# calls lua system function math.max

xLua supports binding a Lua function to a C# delegate.

We first declare a delegate and add the CSharpCallLua tag to it:

[XLua.CSharpCallLua]
Public delegate double LuaMax(double a, double b);

Then add these two lines in the above example (before luaenv is destroyed):

Var max = luaenv.Global.GetInPath("math.max");
Debug.Log("max:" max(32, 12));

It is so simple, after binding lua's math.max to C#'s max variable, the call is similar to a C# function call, and, most importantly, after executing "XLua/Generate Code", max(32 12) The call is not generated (C#) gc alloc, both elegant and efficient! (For more details, see the XLua\Doc documentation.)

xLua global view

(1) Ease of use: No need to generate code to support all features under the editor

The ease of use of xLua is not only reflected in programming, but also in the details of all aspects, even taking into account the team work flow.

xLua only has two menu choices, namely generating code and clearing generated code. Outside of the menu, you can even just execute "Generate Code" before building the mobile version (this also has an API that can be integrated into the project's automated packaging process).

This is one of xLua's special features: there is no need to generate code to support all features under the editor.

The reason why this function is done is because of some project feedback, "generating code" is too far away for planning art, teaching for a long time or forgotten; there is also a big project feedback that due to a lot of code, Unity3D is generated after each code generation. It takes a long time to turn.

(2) Extensibility : It is better to grant it to fish than to give it to fish.

We often use a lot of things in development, such as using PB and background interactions, parsing json format configuration files, and so on. Although we can all find the corresponding library in C#, then use xLua to use these libraries, but this is inefficient, and it is better to have corresponding Lua libraries.

Many programs directly integrate some commonly used Lua libraries, but this brings some new problems: These libraries do not necessarily use, but increase the installation package; integrated library does not necessarily meet the project habits: json parsing someone likes rapidjson, someone Love with cjson, the so-called difficult to adjust; for some projects, these libraries are still not enough, still have to think about ways to increase;

The design principle of the Tencent team is to teach them to fish instead of teaching them to fish. Therefore xLua:

• Provides interfaces, tutorials, developers can add libraries based on personal preferences without modifying xLua code;

• Cross-platform compilation through cmake, you can choose to compile along with xLua, modify a makefile, and get to compile on each platform.

• In addition to facilitating the addition of third-party Lua plug-ins, xLua's build engine supports secondary development and can generate plug-ins, generate some of the code and configuration that you need.

(3) Guarantee of performance

The performance of the game is a concern, so any module changes need to be as low as possible or even to tune the overall performance of the game. The xLua design principle is to ensure the development efficiency as far as possible under the premise of ensuring the operating efficiency.

For performance, there are several crucial versions:

The first version 1.0.0 was released in March of 2005. At that time, delegate, interface was the most important C# access to Lua's settings, and boxing, unboxing, and gc alloc were avoided from the interface level. This is a good starting point. As a general-purpose component, it is known that the problems caused by the unreasonable design of the interface are difficult to solve. Others have already used it, and even they have become accustomed to it, and it is difficult to correct it. Ps: Speaking of this habit, some from the other Lua plug-in to use xLua's children's shoes, first used to call lua LuaFunction.Call (xLua also retains this interface, can be used for occasions where performance requirements are not high), they It was painful in the later period, and it had to be changed back one by one.

The second and most important version is 2.0.0 (released in March 2006). The main goal of this version is to optimize performance because there was a project with extremely stringent performance requirements that wanted to use lua. How severe is it? They feel that C# performance is not assured that the battle system is going to be written in C. In that version, we switched the virtual machine to luajit, added lazyload technology, optimized the line-by-line statements, and even used containers provided by C# in key areas, and wrote them for ourselves (4 times higher than the measured performance of the Dictionary). . . It can be assumed that we remade an xLua. In the end, their selection test concluded that they chose xLua.

Later exchanges with some projects found that the project team is very concerned about gc alloc this indicator, and even more important than interoperability between lua and C# indicators. So with the 2.1.0 version (released in July 2006), this version of the main goal is to optimize the gc, we rewrite the reflection, reflection of the call to reduce the gc to a fraction of the original, performance increased by about 3 times. We designed a brand-new support scheme for complex value types that supports more types (as long as the fields of the struct are all value types), including user-defined structs (not supported by other solutions), and more Memory (Vector3, for example, uses only 30% of memory for other scenarios). However, there are also disadvantages. For example, if you call some methods on Vector3, it will be worse than ulua or slua, because the latter two reimplement Vector3 with Lua. This kind of time-consuming operation is direct compared to Lua and C#. Adaptation costs are much smaller, and it is more cost-effective to do it directly in lua, but the gap is limited to the few ulua and slua reimplemented classes.

The above are just three major nodes. We feel that performance is a point that needs constant attention : when thinking of a good idea at ordinary times, it will be changed. Under the test, there will be promotion when it is added; establishing a performance baseline to prevent the addition of a new function. The modification of the bug corrected the performance.

xLua built-in Lua code profiler; support for real machine debugging. Currently lua profiler is just a gadget, so there is no graphical interface. A typical report is as follows:

There are similar tools on the web. The advantage of ours is the support for C# functions and the accuracy of luajit.

The real machine debugging support is the same for all lua plugins. It is just to compile the luasocket libraries that need to be used in Zero BraneStudio debugging, and there is nothing worthy of introduction.

Technical implementation details

(1) Generics

Generic types are supported in addition to dynamic instantiation at runtime, and runtime dynamic instantiation requires jit support, iOS down the line. For example, if you have generated a code for Dictionary<int, string>, that type is available, but if you newly update the lua code and want to use a Dictionary<int, double>, this type was not generated before. Code, and there is no use anywhere in C#, which is not supported. Statically instantiated generics are in fact no different from non-generic types.

(2) The entrusted event package

Delegate encapsulation is based on the delegated interface to generate a piece of code that manipulates the lua stack as a delegate implementation. For example, it's easy to understand. For example, for delegate: delegate double Add (double a, double b), we generate the following code:

Public double SystemDouble(double a, double b)
{
        RealStatePtr L = luaEnv.L;
        Int err_func =LuaAPI.load_error_func(L, errorFuncRef);
        LuaAPI.lua_getref(L, luaReference);
        LuaAPI.lua_pushnumber(L, a);
        LuaAPI.lua_pushnumber(L, b);
        Int __gen_error = LuaAPI.lua_pcall(L, 2, 1, err_func);
    If (__gen_error != 0)
        luaEnv.ThrowExceptionFromError(err_func - 1);
        Double __gen_ret = LuaAPI.lua_tonumber(L, err_func 1);
        LuaAPI.lua_settop(L, err_func - 1);
        Return __gen_ret;
}

This code transfers the call to the lua function, which is called by the calling delegate.

All other schemes have delegate support. They are generally only used to pass/set a lua function to C# on the lua side. The xLua support is more complete. For example:

• Support C # initiative to use the delegate to refer to a lua function. The advantage of calling lua with a delegate instead of an interface like object[] Call(params object[] args) is to avoid boxing/unboxing when the value type is passed, as well as an array of arguments and a gc alloc of the array of returned values;

• Support for delegate's delegate can correspond to Lua's higher-order functions;

As an extension of this technique, xLua supports using a c# interface to reference a lua table. This feature, in conjunction with some IOC frameworks, allows for no perception between C# and Lua (modules are coupled via interfaces and then assembled by the framework).

(3) Seamless support for generating code and reflections

Generating code is important, and it is already standard in all major programs.

Reflective programs clearly do not support, but from the project's feedback, it is also very important: some project code is many, is close to Apple's 80M Text segment restrictions, for them, the amount of code is related to whether or not Released, the performance of the reflection mode is not as good as the generated code, but the impact on the installation package is small.

This seamless has two meanings:

1. The two are consistent in the characteristics of the support and the use of the features. Switching between the two methods, the business logic code does not need to be modified, change the configuration can be;
2. The two work together seamlessly. For example, on an inheritance chain, any class can choose to generate code or reflection. For example, a subclass selects a generated code, a parent class selects a reflection because it is not commonly used, or can call a parent class on a subclass object. method;

For il2cpp stripping, xLua is also taken into account. As long as you configure ReflectionUse for a class, Unity's link.xml configuration file is automatically generated and the type is listed as not trimmed.

List of other Lua plug-ins

In addition to xLua, there are other Lua plugins such as uLua, SLua, C#light, and so on.

(1) The  ulua application project is the largest, and since open source is early, its reputation is also the largest, which is its great advantage. Tencent also has projects using ulua. The issue with more feedback is the compatibility of its versions:

• Ulua was originally a Unity port called the LuaInterface open source library. It was replaced by cs2lua in early 2015 and replaced by tolua c# in early 2016. It was only because it was considered from an API perspective as three different products. It is difficult to upgrade between them, and each time it is replaced, the previous version is completely unmaintained, which brings great trouble to the project.

• The first version of ulua is purely reflective and is not practical. It has faded out of the market and most of the existing applications are in the latter two versions. Cstolua version of the interface is confusing: it retains the first version of the ulua interface, engage in a new set of interfaces, the two sets of interfaces are not orthogonal, nor is the latter completely replace the former, making people a bit at a loss. By the tolua c# version, this problem was solved, but at the same time the reflection feature (old interface) was also scrapped. But overall, ulua is walking in the right direction.

(2) The  slua code quality is much better than cstolua (many people chose slua for the time) and partially supports reflection. Performance by our test case overall slightly lower than tolua c #, and the code quality compared to tolua c # has not formed a clear advantage.

(3)  C#light , personally feel that there are two major deficiencies:

• According to its implementation principle, performance will not be reliable, and it will not be practical for mobile phones.

• Due to incomplete support for C#, it is essentially just another language called C#light (C# like? name is very appropriate), the two codes are also complex to match, and even it can do more complex than C# and lua

The facts also prove that C# light is basically faded out of the market and can be ignored.

(4)  LSharp is the follow-up work of the author of C# light. Instead, I can expect some of them to be implemented from the il level. These two problems are expected to improve. Unfortunately, there is no following (not maintained).

In contrast, when Tencent designed xLua, it realized more complete functions. This "full" embodiment shows that the features of C# are more fully supported, the lua version of the virtual machine is more complete, and it is easier to use, for example, the editor does not generate Code; In addition, performance is not worse than they are.

Speaking of more complete functionality, some people may complain that there is no pb, json, sqlite and so on. In fact, people who are a little familiar with lua know that it is just a compilation of some ready-made lua extensions, it is not that it does these functions. The benefits of pre-integration are convenience, and the downside is that there is no room for choice. Things that don't fit in will take up space, and what you can use is not necessarily your favorite library.

xLua lua library based on cmake compiler, to add these libraries is very low threshold, there are tutorials, change a Makefile to get each platform compiled. An api is also provided in C# to initialize these libraries. All in all, xLua's principle is to teach it to fish.

Inspired by xLua

The xLua project started by looking at all the options that were available at that time and analyzed the pros and cons of each solution to determine the features of the first version, which was based largely on NLua plus code generation. Introducing NLua, the author of NLua is the author of LuaInterface. NLua can be thought of as an upgraded version of LuaInterface, and as mentioned earlier, the first version of uLua is a Unity ported version of LuaInterface, and it cannot be considered as original.

Because it was "stands" at the time the generated code had seen cstolua's implementation (at the time no ulua was hanging), it felt that it was not very well maintained through hard-coded string splicing, and it was done using templates. Feel this step is right, subsequent generation code adjustment is relatively simple, this is very good for performance tuning.

After more than a dozen versions of iterations, optimization, now the shadow of NLua is relatively light (NLua only supports reflection, and xLua's reflection is completely rewritten in version 2.1.0), leaving the C# reference type object in the lua expression The idea has not changed.

In addition, when we encounter a bug that needs to be adjusted, we will also look at whether the same kind of plug-in has already been resolved. It is more appropriate to compare their revision plans with ours.

R&D and Team behind xLua

xLua is currently iterating over a dozen versions, beginning with the first project and averaging one version per month.

R&D team personnel currently have a full-time development. The test uses the public resources of Tencent Mutual Entertainment. It is standardized: There is a set of continuously supplemented functional automation use cases, and the performance test also establishes a baseline to ensure that performance is not affected by functional iteration. Tencent Mutual Entertainment has a dedicated client compatibility testing lab. We will submit them to the top 100 model for compatibility testing.

As for lua, luajit's update follow-up, first luajit it, luajit little change, the first time I used luajit is 11 years, then support to lua5.1, now also lua5.1, in the middle is only a few bug fixes , performance optimization, or new platform support, we do not do much. The difference between the lua versions is still quite large, but the Chinese version does not change very frequently. It took 6 years from 5.1 to 5.2, 3 years from 5.2 to 5.3, and 5.3 was released in early 2015. I personally feel that the next time The Chinese version will change for a long time, no less than the time span of even greater than 5.1 to 5.2 (5.2 individuals think it is just a transitional version).

The minor version generally changes the bug, so after the stability is upgraded directly, it does not need to do a lot of things. The lua version of xLua currently uses the latest version 5.3.3 of Lua.

Talk about C# and talk about Lua

C # has a good balance between development efficiency and operating efficiency, and it has relatively complete language features. Personally, I think it is a very good language. The main drawback of Unity3D is that its mono version is too low. Some very old bugs, such as the well-known foreach performance problem, have not been solved in many versions. New features such as await do not support it.

In addition, in the mobile platform iOS does not allow applications to download native code to run, jit, just to block the application of the hot update to die, if the mono virtual machine can do like luajit, jit no way to use interpret mode, in fact, no lua Or what's wrong with other hot update programs.

Lua is known as the king of game scripts. It is widely used in the game field. It has been designed into the embedded domain since the beginning of the design. For example, compared to the features it provides, it is very small, and starting a vm is not a lot of resources. Performance is also the best in the script.

Lua is relative to C#, the first is that it supports parsing execution, which in turn supports hot updates. Compiler-free compilation is also very large for development efficiency, especially for larger projects.

Lua's dynamic type has its advantages and disadvantages. Good is that there is no type checking during compile-time, and rapid development has more advantages, especially in the game field where the demand changes for three days. The downside is that it takes a lot of testing to ensure that robust software is built, and that because of runtime checks, performance is lower than in statically typed languages.

Lua's major feature is the support of language-level coroutines. It is much better than Unity3D's generator-based simulation coroutines. It is helpful for complex asynchronous business logic writing. Examples of xLua's companion examples are (ps, Unity3D. The mono version is a more ideal asynchronous solution when it is upgraded to support await.

As for how C# and Lua fit together, everyone may have different views, but at least one thing is certain: changes in demand are expected to be more likely to require hot spots. Of course, you can also try the latest development model, all C# development, lua fix bug.