How to understand Lua virtual machine

Time:2020-2-7

Welcome to Tencent cloud + community to get more Tencent technology practice dry goods~

Author: Zheng Xiaohui, senior engineer of Tencent game client development

Write in the front: all the words in this article are manually typed by me, and the demo code shared later in this article is line by line by me. Before me, many predecessors have studied Lua virtual machine, so many ideas in this article must be on the shoulders of these giant people.

The title of this article is “in depth and in simple terms Lua virtual machine”, in fact, the focus is on the two words. After all, the author’s technical level is limited. But it’s said that the loser’s article with a name needs to be read, so it gets its name.

This article is dedicated to those who are interested in Lua virtual machine. I hope that this paper can achieve a good result.

Lua’s execution process:

The whole process of lua Code:

As shown in the figure below: programmer encodes Lua file – > syntax lexical analysis generates Lua bytecode file (corresponding to luac.exe of lua tool chain) – > Lua virtual machine parses bytecode and executes instruction set – > output result.

How to understand Lua virtual machine

The blue and green parts are what this article tries to talk about.

Lexical analysis:

I’m not going to talk about all the lexical analysis processes of lua. After all, if I waste too much time writing this, I will plan for a while and ask my classmates how the development progress of my needs is. So to make a long story short, I will analyze it with a specific example based on my understanding of lua:

Lua code block:

If a < b then a = c end

We programmers can understand this sentence, but the computer is just like the beautiful wife in charge of some male programmers’ home, only knowing that this is a string of meaningless strings spelled with English characters.

In order to make the computer understand this sentence, the first thing we have to do is participle: since you can’t understand it. I’ll break a sentence into one word first, and I’ll tell you what each word means.

The result of participle is about as follows:

Participle result type (meaning)

    If type if (if keyword)

    A Type Var (this is a variable)

    < type < opless (this is a less than sign)

    B type? Var (this is a variable)

    Then type [then (then keyword)

    A Type Var (this is a variable)

    =Type [opequal (this is an equal sign)

    C type? Var (this is a variable)

    End type? End (end keyword)

Okay. Now the computer finally understands. It turns out that there are nine words in the code you wrote, and I understand the meaning of each word. So now the question is, does the computer understand this sentence?

Computers still don’t understand. Just like the sentence “eat”, the computer understands that “eat” is a verb that opens its mouth. “Rice” is a noun, which means rice. But when you put your meals together, the computer doesn’t know that it means “open your mouth, put your meal in your mouth, and swallow it in your stomach.”. Because the computer only knows “open mouth” and “rice” two things, what is the connection between these two things, the computer can not understand. Some people will say: simple: eat + other words, this kind of structure makes the computer generally understand the meaning of putting the things represented by the latter word into the mouth? This situation is suitable for the word “eat”, but how do you make the computer understand the word “surprised”? So this leads to the next topic: semantic analysis.

For semantic parsing, if you want to know more about it, you can learn about ast (abstract syntax tree). However, for our example, we can simulate it in a simple way to understand.

For Lua, each keyword has its own special structure. Therefore, the key words of lua will become the focus of semantic analysis. Now we are referring to the example of if: we can simply express the parsing process in pseudo code:

We can abstract the if statement into this structure:

If condition then dosth end

So the pseudo code for parsing if statement block is as follows:

ReadTokenWord();
       If(tokenWord.type == Type_If) then
          Readcondition() // read condition expression
          Readthen() // read the keyword then
          Readcodeblock() // read logical code block
          Readend() // read keyword end
      End

So in order for the computer to understand, we still have to turn this thing into a data structure.

Because I’m just doing a demo, I’ve used prior knowledge. That is to say, I assume that the logical structure of our if statement block is as follows:

If is less than the conditional expression then the assignment expression end

So in my demo, I converted to C + + data structure, which is ifstatement

How to understand Lua virtual machine

OK, so now we have finished the whole lexical analysis. But the real Lua virtual machine can’t execute our ifstatement. The implementation in Lua source code is similar to this tokentype and structured if statement while statement, etc., and Lua does not generate a complete syntax tree. In the implementation of lua source code, it parses some statements, generates temporary syntax trees, and translates them into instruction sets. It doesn’t wait for all statements to be parsed before translation. Semantic parsing and translation into instruction set is a parallel process. Paste a partial implementation of semantic parsing in the source code:

How to understand Lua virtual machine

OK, now we have turned the Lua code that our programmers input into a data structure (the computer can read it). Next, we need to turn this data structure into something that Lua virtual machine knows. This thing is Lua instruction set!

As for the process of transformation, for our example, it is roughly as follows:

    If a < b then a = c end

First understand the condition a < B: a register based instruction design is roughly as follows:

a. B is variable. Suppose our available register index value starts from 10 (registers 0-9 are already occupied): and suppose we have a constant index table: constant 0: character ‘a’, constant 1: String ‘B’. Then a < B can be translated as follows:

  • Loadk 10, 0: load_g [constvar [0]] into register 10: R [10] = _g [“a”]
  • Loadk 11, 1: load_g [constvar [1]] into register 11: R [11] = _g [“B”]
  • LT 10,11: compare whether R [10] < R [11] is valid. If it is, skip the next instruction (+ + PC), otherwise execute the next instruction. The instruction that follows LT must be the JMP instruction. That is, if R [10] < R [11] is set up, JMP will not be executed, and an instruction (the instruction set corresponding to a = C statement block) will be executed directly after JMP. Otherwise, the following statement block will be skipped directly (the assignment process of a = C will be skipped).

    Similarly, continue to translate a = C and so on.

    So if a < B then a = C end is translated into the demo I wrote:

How to understand Lua virtual machine

OK, now we’ve probably figured out how to turn Lua code into an instruction set.

Now let’s take a look at the instruction set of lua5.1:

Lua’s instruction set is fixed length, and each instruction is 32-bit, which is about the same length:

How to understand Lua virtual machine

The lower six bits of each instruction are the instruction code of the instruction, such as 0 for move, 12 for add. Lua has a total of 37 instructions

MOVE,LOADK,LOADBOOL,LOADNIL,GETUPVAL,GETGLOBAL,GETTABLE,SETGLOBAL,SETUPVAL,SETTABLE,NEWTABLE,SELF,ADD,SUB,MUL,DIV,MOD,POW,UNM,NOT,LEN,CONCAT,JMP,EQ,LT,LE,TEST,TESTSET,CALL,TAILCALL,RETURN,FORLOOP,TFORLOOP,SETLIST,CLOSE,CLOSURE,VARARG.

We found that there are IABC, iabx, iasbx on the graph. This means that some instruction formats are opcode, a, B, C, some are opcode a, BX, some are opcode a, sbx. The difference between sbx and BX is that BX is an unsigned integer, while sbx represents a signed number, that is, sbx can be a negative number.

I’m not going to go into every instruction in detail. Let me give you an example:

How does instruction code 0x 0000441 resolve

0x4041 = 0000 0000 0000 0000 0100 0000 0100 0001

The lower six bits (0-5) are opcode: 000001 = 1 = loadk instructions (0-37 correspond to 38 instructions listed above. In order, 0 is move, 1 is loadk, 2 is loadtool….. 37 is vararg). The loadk instruction format is the IABC (C is useless, only AB is useful) format. So let’s continue to read ab.

A = low 6-13 bits is 00000001 = 1, so a = 1

B = the lower 14-22 bits are 000000001 = 1, so B = 1

So 0x4041 = loadk 1, 1

How to parse the instruction code is also written in the demo. The code is as follows:

How to understand Lua virtual machine

So what is the Lua bytecode generated by the compiled luac file? Besides the instruction set, what is in the Lua bytecode file? Of course, it’s not as mentally retarded as the demo I used to parse. So let’s talk about the structure of lua bytecode file:

Lua bytecode file (*. Lua. Bytes) contains: file header + top level function:

File header structure:

How to understand Lua virtual machine

Top level functions and other ordinary functions have the same structure:

How to understand Lua virtual machine

So we can easily write our own code to parse. In the demo source code provided later, I have also implemented the byte code file parsing.

The example in demo is the Lua source code involved and the information obtained from the final parsing of bytecode

How to understand Lua virtual machine

How to understand Lua virtual machine

OK, the last thing left in this article is: how does Lua virtual machine execute these instructions?

It’s like this:

While (instruction is not empty)
       Execution instruction
       Take down an instruction to execute
    End

How should each instruction be executed??? If you still have the impression, we will transfer the instruction set after the semantic analysis of the previous article is as follows:

a < b

  • Loadk 10, 0: load_g [constvar [0]] into register 10: R [10] = _g [“a”]
  • Loadk 11, 1: load_g [constvar [1]] into register 11: R [11] = _g [“B”]
  • LT 10,11: compare whether R [10] < R [11] is valid. If it is, skip the next instruction (+ + PC), otherwise execute the next instruction. The instruction that follows LT must be the JMP instruction. That is, if R [10] < R [11] is set up, JMP will not be executed, and an instruction (a = C Block) will be executed directly after JMP. Otherwise, a statement block (a = C assignment process) will be skipped directly.

    Of course, the text behind the instruction has described the execution logic of the instruction in detail.

    In order to truly execute, we need to design the data structure as follows: 1. Register: 2. Constant table: 3. Global variable table:

In order to execute the example in our demo:

I implemented all the instructions involved in this code

insExecute[(int)OP_LOADK] = &LuaVM::LoadK;
insExecute[(int)OP_SETGLOBAL] = &LuaVM::SetGlobal;
insExecute[(int)OP_GETGLOBAL] = &LuaVM::GetGlobal;
insExecute[(int)OP_ADD] = &LuaVM::_Add;
insExecute[(int)OP_SUB] = &LuaVM::_Sub;
insExecute[(int)OP_MUL] = &LuaVM::_Mul;
insExecute[(int)OP_DIV] = &LuaVM::_Div;
insExecute[(int)OP_CALL] = &LuaVM::_Call;
insExecute[(int)OP_MOD] = &LuaVM::_Mod;
insExecute[(int)OP_LT] = &LuaVM::_LT;
insExecute[(int)OP_JMP] = &LuaVM::_JMP;
insExecute[(int)OP_RETURN] = &LuaVM::_Return;

Take add for example:

bool LuaVM::_Add(LuaInstrunction ins)
{
    //R(A):=RK(B)+RK(C) :::
    //Todo: necessary parameter validity check: throw exception if there is any problem  
    //The result of adding the data represented by ins.bvalue and the data represented by ins.cvalue is assigned to the register whose index value is ins.avalue
    luaRegisters[ins.aValue].SetValue(0, GetBK(ins.bValue) + GetBK(ins.cValue));
    return true;
}

The following is a screenshot of the operation effect of the program:

How to understand Lua virtual machine

Looking at the whole process, we can actually think about this question: why is the execution efficiency of lua far lower than that of C program?

Personal folly:

  1. True and false register: the register involved in Lua instruction set is an analog register, in fact, it is a data in memory. The access speed depends on the CPU’s access speed to memory. In the end, C program can be executed with Win32 instruction set or arm instruction set. The registers ebx, esp, etc. involved in this are NAND gates on the CPU, and their access speed = the frequency of the CPU (compared with the speed of the CPU accessing memory, it’s almost one sky and one earth).
  2. Instruction set running platform: Lua instruction set running platform is Lua virtual machine. C program instruction set is directly supported by hardware.
  3. The data in C directly corresponds to the memory address. The data in Lua corresponds to a data structure that describes the data. So after such a layer, the efficiency is greatly reduced.
  4. For example, Lua’s GC operation and so on are all things that C program doesn’t need to do….

OK, finally, I will present the source code of this demo: this source code was written by me when I was at home on Qingming Festival. That is to say, the code has not been patiently sorted out, and someone asked me to go out for a drink on Qingming Festival, which led me to be absent-minded for a long time, “I’m going to go out for a drink when I’m finished coding quickly.” so some coding formats and structural design can see random examples everywhere ~ after all, it’s just a demo. Life in the world, to have Buddha nature, good luck! If you really want to understand more about Lua virtual machine, I recommend you to read the source code of lua virtual machine if you have time and patience~

Finally, I sincerely thank all the students who saw the last sentence. Thank you for your patience after reading a long piece of technical nonsense.

This article has been released by Tencent cloud + community authorized by the author. For students who need source code, please click: https://cloud.tencent.com/dev

Questions and answers

#What do you mean in Lua?

Related reading

Lua performance analysis

Using Lua tips

Openresty best case | introduction to Lua

How to understand Lua virtual machine