Tencent Open Hand Mobile Hot Update: Introduction to XLua under Unity3D

Write in front

　　xLua is a Lua programming solution under Unity3D . Since it was promoted in early 2016 , it has been used in more than ten Tencent self-developed games, and has been widely acclaimed for its good performance, ease of use, and expandability. Tencent has now open sourced xLua to GitHub .

　　At the end of December 2016 , xLua has just achieved a new breakthrough: the full platform supports Lua to fix C# code bugs .

　　At present , most of Lua's hot update programs under Unity are required to be updated by the Lua language from the very beginning . The inadequacies are:

1. The cost of access is high. Some projects have been written in C# . At this time, access needs to be re-implemented with Lua .

2. Even if it is accessed from the beginning, it is difficult to develop two languages ​​at the same time.

3, Lua performance is inferior to C# ;

　　The xLua Hotfix technical support replaces a C# implementation (function, operator, property, event, or entire class) with a Lua implementation at runtime , meaning that you can:

1, usually with C# development;

2, the operation is C # , performance spike Lua ;

3, there is a bug in the distribution of a Lua script fix , the next time the overall update can be replaced by Lua 's implementation of the correct C # implementation, even when the update can not do to restart the game;

　　This new feature, iOS , Android , Window , and Mac, all passed the test and is currently being optimized for ease of use.

　　So, what kind of technology is xLua of Tencent's open source ? Why is it so designed? What is more interesting is the performance of xLua ? With these questions, InfoQ interviewed the author and compiled the content.

Guest Profile

　　Car male students, 05 graduated, worked in Huawei 6 years, followed by two games in start-up companies has spent the past few years, 15 years into the public Tencent mutual entertainment center components. Currently focusing on the development of some game common components.

technical background

　　Tencent has been working on mobile games. For the projects I know, most game engines are Unity3D , and a few use coco2d .

　　What games did the xLua plugin actually use? Although xLua is 2015 Nian 3months to complete the first version, but because the project team thermal more sense is not very common, demand is not very strong, xLua development resources are transferred to a more urgent project. Until the end of the year 15 was officially integrated into our apollo mobile game development framework, the first project of xLua was ushered in. So far, we have more than a dozen projects that we have applied to xLua , some of which are heavyweight IP , or products built according to star standards.

　　Prior to xLua , in the face of iOS's inability to hot-update issues, there were ulua 's, useful slua 's, and project-based scripting languages. However, there weren't many projects that were updated at the time.

Hot update process

　　The hot update process of the mobile game is very simple. It is just to check if there is a new version of the file at startup. If there is, the old file will be overwritten and then started.

　　If the downloaded file is a picture, the model is no problem, but if Unity's native code logic, whether the previous Mono AOT or later il2cpp , are compiled into native code , iOS can not run.

　　The solution is one, do not use native code , do not use jit , parse the implementation on it. Including xLua all hot update support program, it is designed by " analysis execution " to implement the code logic hot update.

From xLua of Hello world

( 1 ) Three lines of code running lua script

　　A complete example requires only 3 lines of code:

　　After downloading xLua and unzipping it into the Unity Project Assets directory, create a MonoBehaviour and drag it to the scene. Add these three lines to Start :

1 2 3

XLua.LuaEnv luaenv = new XLua.LuaEnv(); luaenv.DoString("CS.UnityEngine.Debug.Log('hello world')"); luaenv.Dispose();

　　Run to see the console print hello world.

1, the first and third lines respectively LuaEnv creation and destruction, the so-called LuaEnv can be understood as lua virtual machine, often the whole project can be a virtual machine:

2, DoString inside can be any legal lua code, the example calls UnityEngine.Debug.Loginterface to print a log ( C# 's static function is directly available under CS );

( 2 ) C# calls lua system function math.max

　　xLua supports binding a Lua function to a C# delegate .

　　We first declare a delegate and add the CSharpCallLua tag to it :

1 2	[XLua.CSharpCallLua] public delegate double LuaMax(double a, double b);

　　Then add these two lines in the above example (before luaenv is destroyed):

1 2	var max = luaenv.Global.GetInPath("math.max"); Debug.Log("max:" + max(32, 12));

　　It's that simple, after binding lua 's math.max to C# 's max variable, the call is similar to a C# function call, and, most importantly, after executing "XLua/Generate Code" , max(32 12) The call is not generated ( C# ) gc alloc , both elegant and efficient! (For more details, see the XLuaDoc documentation.)

xLua global view

( 1 ) Ease of use: No need to generate code to support all features under the editor

　　The ease of use of xLua is not only reflected in programming, but also in the details of all aspects, even taking into account the team work flow.

　　xLua only has two menu choices, namely generating code and clearing generated code. In addition to the menu, or even just need to build the implementation of some former mobile version of the "Generate Code" can (which is also API automation project can be integrated into the packaged process).

　　This is one of xLua 's special features: there is no need to generate code to support all features under the editor.

　　The reason why this function is done is because of some project feedback. "Generating code " is too far away for planning arts. It has been taught for a long time or it has been forgotten; there is also a big project feedback that because of the large amount of code, Unity3D will generate code every time . It takes a long time to turn.

( 2 ) Extensibility: It is better to grant it to fish than to give it to fish.

　　We often use a lot of things in development, such as using PB and background interactions, parsing json format configuration files, and so on. Although we can all find the corresponding library in C# , then use xLua to use these libraries, but this is inefficient, and it is better to have corresponding Lua libraries.

　　Many programs directly integrate some commonly used Lua libraries, but this brings some new problems: These libraries do not necessarily use, but increase the installation package; integrated library is not necessarily consistent with the project habits: json parsing someone likes rapidjson , someone Love with cjson , the so-called difficult to adjust; for some projects, these libraries are still not enough, still have to think of ways to increase their own;

　　The design principle of the Tencent team is to teach them to fish. It is better to teach them to fish, so xLua :

　　 Provides interfaces, tutorials, developers can add libraries based on personal preferences without modifying xLua code;

　　Cross-platform compilation through cmake , you can choose to compile along with xLua , modify a makefile , and get to compile on each platform.

　　In addition to facilitating the addition of third-party Lua plugins, xLua 's buildengine supports secondary development, and can generate and generate plugins that generate some of the code and configuration they need. 

( 3 ) Guarantee of performance

　　The performance of the game is a concern, so any module changes need to be as low as possible or even to tune the overall performance of the game. The xLua design principle is to ensure the development efficiency as far as possible under the premise of ensuring the operating efficiency.

　　For performance, there are several crucial versions:

　　The first version 1.0.0 on 05 Nian 3 released in March, when the delegate , interfaceas the most important C # to access Lua settings from the interface level, avoiding Boxing , Unboxing , gc alloc , this is a good starting point. As a general-purpose component, it is known that the problems caused by the unreasonable design of the interface are difficult to solve. Others have already used it, and even they have become accustomed to it, and it is difficult to correct it. Ps: Speaking of this habit, some from the other Lua plug-in to use xLua 's children's shoes, used LuaFunction.Call to call lua at first ( xLua also retains this interface, can be used for occasions with low performance requirements), they It was painful in the later period, and it had to be changed back one by one.

　　The second version is a very important 2.0.0 ( 06 Nian 3 Yue release), this version of the main objectives to optimize performance, when there was a very demanding performance requirements of the project would like to use Lua , harsh and to what extent? They feel that C# performance is not assured that the battle system is going to be written in C++ . In that version, we switched the virtual machine to luajit , added lazyload technology, optimized line-by-line statements, and even used containers provided by C# in key areas , and wrote them for ourselves ( 4 times higher than the measured performance of Dictionary ). . . Think of us as redoing a xLua . In the end, their selection test concluded that they chose xLua .

　　Later exchanges with some projects found that the project team is very concerned about gc alloc this indicator, and even more important than interoperability between lua and C# indicators. So with 2.1.0 version ( 06 Nian 7 Yue release), this version is the main objective gc optimization, we rewrite the reflection, reflection call gc reduced to a fraction of the original, performance improved 3 or so times. We designed a brand-new support scheme for complex value types that supports more types (as long as the fields of the struct are all value types), including user-defined structs (not supported by other solutions), and more Memory ( Vector3 as an example, memory usage is only 30% ofother schemes ). But there are also disadvantages. For example, if you call some methods on Vector3 , it will be worse than ulua or slua , because the latter two reimplement Vector3 with lua . This kind of time-consuming operation is compared to lua and C#.The direct adaptation cost is much smaller, and it is more cost-effective to do it directly in lua , but the gap is limited to the few re-implemented classes of uluaand slua .

　　The above are just three major nodes. We feel that performance is a point that needs constant attention : when thinking of a good idea at ordinary times, it will be changed. Under the test, there will be promotion when it is added; establishing a performance baseline to prevent the addition of a new function. a bug modified to change the bad performance.

　　xLua built-in Lua code profiler ; support for real machine debugging. Currently lua profiler is just a gadget, so there is no graphical interface. A typical report is as follows:

　　There are similar tools on the web. The advantage of ours is the support for C#functions and the accuracy of luajit .

　　The real machine debugging support is the same for all lua plugins. It is just to compile the luasocket libraries that need to be used in Zero BraneStudio debugging , and there is nothing worthy of introduction.

Technical implementation details

(1) Generics

　　Generic types are supported in addition to dynamic instantiation at runtime, and runtime dynamic instantiation requires JIT support. iOS does not work downstream. For example, if you are equipped with a pair Dictionary generated code, then this type can be used, but if you are new updates lua code and want to use a Dictionary , did not generate the code before this type, and C # inside nor anywhere However, this does not support it. Statically instantiated generics are in fact no different from non-generic types.

(2) Encapsulation of delegated events

　　Delegate encapsulation is based on the delegated interface to generate a piece of code that manipulates the lua stack as a delegate implementation. For example, it's easy to understand. For example, for delegate: delegatedouble Add(double a, double b), we generate the following code:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18

public double SystemDouble(double a, double b) {         RealStatePtr L = luaEnv.L;         int err_func =LuaAPI.load_error_func(L, errorFuncRef);                                   LuaAPI.lua_getref(L, luaReference);                                   LuaAPI.lua_pushnumber(L, a);         LuaAPI.lua_pushnumber(L, b);                                   int __gen_error = LuaAPI.lua_pcall(L, 2, 1, err_func);         if (__gen_error != 0)             luaEnv.ThrowExceptionFromError(err_func - 1);                                   double __gen_ret = LuaAPI.lua_tonumber(L, err_func + 1);         LuaAPI.lua_settop(L, err_func - 1);         return  __gen_ret; }

　　This code transfers the call to the lua function, which is called by the calling delegate.

　　All other schemes have delegate support. Generally, they are only used to pass /set a lua function to C# on the lua side . The xLua support is more complete. For example:

　　Support C # initiative to use the delegate to refer to a lua function. The advantage of calling lua with a delegate instead of an interface like object[]Call(params object[] args) is to avoid boxing/unboxing when passing value types , as well as parameter arrays and gc alloc of return value arrays .

　　Supports the delegate 's delegate return , which corresponds to Lua 's higher-order functions;

　　As an extension of this technique, xLua supports using a c# interface to refer to a lua table . This feature works in conjunction with some IOC frameworks to achieve a sense of insensitivity between C# and Lua (modules are coupled via interfaces and then assembled by the framework).

(3) seamless support for generating code and reflections

　　Generating code is important, and it is already standard in all major programs.

　　Reflective programs clearly do not support, but from the project's feedback, it is also very important: some project code is many, is close to Apple's 80M Text segment restrictions, for them, the amount of code is related to whether or not Released, the performance of the reflection mode is not as good as the generated code, but the impact on the installation package is small.

　　This seamless has two meanings:

1, the two in the support of the characteristics and the use of features are consistent, switching between the two methods, business logic code without modification, change the configuration can be;

2, the two seamlessly cooperate, such as an inheritance chain, any one class can choose to generate code or reflection, such as subclasses choose to generate code, the parent class is not commonly used to select the reflection, or can call the parent on the subclass object The method of the class;

For il2cpp of stripping , xLua also taken into account, as long as you configure for a class ReflectionUse , will automatically generate the Unity of link.xml configuration file, the type classified as not trimmed.

List of other Lua plug-ins

　　In addition to xLua , there are other Lua plugins such as uLua , SLua , C#light, and so on.

(1) The  ulua application project is the largest, and since open source is early, its reputation is also the largest, which is its great advantage. Tencent also has projects using ulua . The problem with more feedback is the compatibility of its versions:

　　Ulua was originally a Unity port called the LuaInterface open source library. It was replaced by cs2lua in early 2015 , and it was replaced by tolua c# in early 2016. Theonly reason to say “ change ” is because it can be considered as three different products from an API perspective. It is difficult to upgrade between them, and each time it is replaced, the previous version is completely unmaintained, which brings great trouble to the project.

　　The first version of ulua is purely reflective and is not practical. It has faded out of the market and most of the existing applications are in the latter two versions. Cstoluaversion of the interface is confusing: it retains the first version of the ulua interface, engage in a new set of interfaces, the two sets of interfaces are not orthogonal, nor is the latter completely replace the former, making people a bit at a loss. By the tolua c#version, this problem was solved, but at the same time the reflection feature (old interface) was also scrapped. But overall, ulua is moving in the right direction.

(2) The  slua code quality is much betterthan cstolua (many people chose slua for the time) and partially supports reflection. Performance by our test case overallslightly lowerthan tolua c # , and the code quality compared to tolua c # has not formed a clear advantage.

(3)  C#light , personally feel that there are two major deficiencies:

　　According to its implementation principle, performance will not be reliable, and it will not be practical for mobile phones.

　　Due to incomplete support for C# , it is essentially just another language called C#light ( C# like ? name is very appropriate), the two codes are also complex to match, and even it can do more complex than C# and lua .

　　The facts also prove that C# light is basically faded out of the market and can be ignored.

(4)  LSharp isthe follow-up work of the author of C# light . Instead, I can expect some of them to beimplementedfrom the il level. These two problems are expected to improve. Unfortunately, there is no following (not maintained).

　　In contrast, when Tencent designed xLua , it realized more complete functions. This " full " is reflected in the more complete support of the features of C# , and the luaversion of the virtual machine is more fully supported; it is easier to use, such as not generating an editor. Code; In addition, performance is not worse than they are.

　　Speaking of more complete functionality, there may be complaints that there are no functions such as pb , json , sqlite, etc. In fact , people who are a little familiar with lua know that it is just a compilation of some ready-made lua extensions , it is not that it does these functions. The benefits of pre-integration are convenience, and the downside is that there is no room for choice. Things that don't fit in will take up space, and what you can use is not necessarily your favorite library.

　　xLua lua library based on cmake compiler, to add these libraries is very low threshold, there are tutorials, change a Makefile to get each platform compiled. An apiis also provided in C# to initialize these libraries. All in all, xLua 's principle is to teach it to fish.

Inspired by xLua

　　The xLua project started by looking at all the options that were available at that time and analyzed the pros and cons of each solution to determine the features of the first version, which was based largely on NLua plus code generation. Introduce NLua , NLua author is LuaInterface author, NLua can be considered LuaInterface upgraded version, but also said earlier, the first edition uLua is LuaInterface of Unity portable version, can not be considered original.

　　Because it was " stands " at the time the generated code had seen cstolua 's implementation (at the time no ulua was hanging ), it felt that it was not very well maintained through hard-coded string splicing, and it was done using templates. Feel this step is right, subsequent generation code adjustment is relatively simple, this is very good for performance tuning.

　　After more than a dozen versions of iterations, optimization, the shadow of NLua is now relatively light ( NLua only supports reflection, and xLua 's reflection has been completely rewritten in version 2.1.0 ), leaving the C# reference type object in the luaexpression The idea has not changed.

　　In addition, when we encounter a bug that needs to be adjusted , we will first look at whether the same kind of plug-in has already been resolved. Compare it to their modification plan and ours, the selection is more appropriate.

R&D and Team behind xLua

　　xLua is currently iterating over a dozen versions, beginning with the first project and averaging one version per month.

　　R&D team personnel currently have a full-time development. The test uses the public resources of Tencent Mutual Entertainment. It is standardized: There is a set of continuously supplemented functional automation use cases, and the performance test also establishes a baseline to ensure that performance is not affected by functional iteration. Tencent Mutual Entertainment has a dedicated client compatibility testing lab. We will submit them to the top 100 model for compatibility testing.

　　As for lua , luajit 's update follow-up, first luajit it, luajit little change, the first time I used luajit is 11 years, then support to lua5.1 , now also lua5.1 , in the middle is only a few bug fixes , performance optimization, or new platform support, we do not do much. The lua in the difference between the versions is still quite large, but not the version change frequently, from 5.1 to 5.2 with a 6 -year, from 5.2 to 5.3 with a 3 -year, 5.3 is the 2015 release of the year, I personally think that the next time The Chinese version will change for a long time, no less than a time span of even greater than 5.1 to 5.2 ( 5.2 individuals think it is just an interim version).

　　The minor version of the general change bug , such as a direct upgrade after the stability on it, do not need to do a lot of things, the current lua version of LuLu is used Lua 's latest version 5.3.3 .

Talk about C# and talk about Lua

　　C # has a good balance between development efficiency and operating efficiency, and it has relatively complete language features. Personally, I think it is a very good language. The main drawback of Unity3D is that its mono version is too low, and some very old bugs , such as the well-known foreach performance problem, have not been solved in many versions. New features such as await do not support it.

　　Also in the mobile platform iOS does not allow applications to download native code runs, the JIT , just the mono thermal applications to update blocked, if the monovirtual machine can do as luajit that, the JIT nowhere to use interpret mode, in fact, no lua Or what's wrong with other hot update programs.

　　The lua known as the king of the game script, used widely in the field of the game, it was designed with consideration to the embedded field, such as the relative properties it offers, it is very small and start a vm accounts are not many resources Performance is also the best in the script.

　　Lua is relative to C# , the first is that it supports parsing execution, which in turn supports hot updates. Compiler-free compilation is also very large for development efficiency, especially for larger projects.

　　Lua's dynamic type has its advantages and disadvantages. Good is that there is no type checking during compile-time, and rapid development has more advantages, especially in the game field where the demand changes for three days. The downside is that it takes a lot of testing to ensure that robust software is built, and that because of runtime checks, performance is lower than in statically typed languages.

　　lua is a major feature of the language level of coroutines ( the coroutine ) support than Unity3D based generator simulation coroutine much better for complex business logic asynchronous write helpful, xLua supporting examples are examples ( PS look, Unity3D of The mono version is a more ideal asynchronous solution when it is upgraded to support await .

　　As for how C# and Lua fit together, everyone may have different opinions, but at least one thing is certain: changes in demand are expected to be more likely to require hot spots, with Lua . Of course, you can also try the latest development model, full C#development, lua fix bug .

Write last

　　xLua should also have shortcomings. We will modify it the first time we discover it. The Tencent xLua team very much welcomes everyone to provide feedback after discovering the insufficiency.