Deep understanding function of go language

Time:2021-12-1

Deep understanding function of go language

concept

In computer programming, function is actually an abstract concept and a programming interface; Through abstraction, the complex system can be decomposed into various opaque interfaces wrapped with complex algorithms, which is convenient for calling each other, layering, scalability, convenience and so on.

Specifically, function generally refers to an independent and reusable program logic fragment, which is used to facilitate other function calls; The English name is function, sometimes called method and routine.

The compiler finally compiles the function into machine instructions and saves them in an executable file.

In the memory space of a process, a function is just a continuous memory area containing machine instructions; Just in terms of structure, it is no different from array.

In go language, function is a first-class citizen, not only a code fragment, but also a data type; Like other data types, it has its own type information.

Function type

There are many definitions of function types, which are equivalent.

It is defined in the Runtime / type.go source file as follows:

  1. typefunctypestruct{
  2. typ_type
  3. inCountuint16
  4. outCountuint16
  5. }

The definitions in the source files of reflect / type.go and internal / reflectlite / type.go are as follows:

  1. //funcTyperepresentsafunctiontype.
  2. //
  3. //A*rtypeforeachinandoutparameterisstoredinanarraythat
  4. //directlyfollowsthefuncType(andpossiblyitsuncommonType).So
  5. //afunctiontypewithonemethod,oneinput,andoneoutputis:
  6. //
  7. //struct{
  8. //funcType
  9. //uncommonType
  10. //[2]*rtype//[0]isin,[1]isout
  11. //}
  12. typefuncTypestruct{
  13. rtype
  14. inCountuint16
  15. outCountuint16//topbitissetiflastinputparameteris…
  16. }

As you can see from the comments of the functype structure, the information of the function type is actually very complex.

In fact, the complete function type definition is shown in the following pseudo code:

  1. typefuncTypestruct{
  2. Rtype / / basic type information
  3. Incountuint16 / / number of parameters
  4. Outcountuint16 / / number of returned values
  5. Uncommonuncommontype / / method information
  6. Intypes [incount] * rtype / / parameter type list
  7. Outtypes [outcount] * rtype / / return value type list
  8. Methods [uncommon. Mcount] method / / method list
  9. }

Uncommontype and method are defined in the reflect / type.go source file and are used to store and resolve method information of types.

  1. typeuncommonTypestruct{
  2. Pkgpathnameoff / / package path name offset
  3. Mcountuint16 / / number of methods
  4. Xcountuint16 / / number of public export methods
  5. Moffeint32 / / methods offset from the starting address of the object
  6. _uint32//unused
  7. }
  1. //Methods of non interface types
  2. typemethodstruct{
  3. Namenameoff / / method name offset
  4. Mtyptypeoff / / method type offset
  5. Ifntextoff / / address offset when calling through the interface; Interface types are not described in this article
  6. Tfntextoff / / address offset during direct type call
  7. }
  1. typenameOffint32//offsettoaname
  2. typetypeOffint32//offsettoan*rtype
  3. typetextOffint32//offsetfromtopoftextsection
  • Nameoff is the offset from the starting address of the. Rodata section.
  • Typeoff is the offset from the starting address of the. Rodata section.
  • Textoff is the offset from the start address of the. Text section.
  • For an introduction to reflect.name, please read integers in memory.

Function type structure distribution diagram

The complete function type information structure distribution is shown in the following figure:

Deep understanding function of go language

Each function has its own type information, but some functions are simple and some functions are complex. Not every function type contains all the fields in the figure above.

The distribution of simple function type information structure may be as shown in the following figure:

Deep understanding function of go language

perhaps

Deep understanding function of go language

Note: the light gray blocks in the above diagram represent the filling of memory alignment and do not store any data.

Of course, functions may also have parameters without return values, and functions may also have return values without parameters. Their type information structure will be a little different. Imagine, it’s just a simplified structure.

Through the memory analysis of this article, we will understand every detail of function types.

environment

  1. OS:Ubuntu20.04.2LTS;x86_64
  2.  
  3. Go:goversiongo1.16.2linux/amd64

statement

Different operating systems, processor architectures and go versions may lead to differences in register values, memory addresses, data structures, etc. at runtime after compiling the same source code.

This paper only includes the research and analysis of 64 bit executable program under 64 bit system architecture.

This paper only guarantees the accuracy and effectiveness of the analysis data in the learning process in the current environment.

This paper only discusses ordinary functions and declared function types, not interfaces, implementations, closures and other knowledge points.

Code list

  1. packagemain
  2.  
  3. import(
  4. “errors”
  5. “fmt”
  6. “reflect”
  7. )
  8.  
  9. //Declare function type
  10. typecalcfunc(a,bint)(sumint)
  11.  
  12. //Private method – > packagescope
  13. //go:noinline
  14. func(fcalc)foo(a,bint)int{
  15. returnf(a,b)+1
  16. }
  17.  
  18. //REE public export method – > publicscope
  19. //go:noinline
  20. func(fcalc)Ree(a,bint)int{
  21. returnf(a,b)-1
  22. }
  23.  
  24. funcmain(){
  25. //Ordinary function
  26. Print(fmt.Printf)
  27. //Function type instance
  28. varaddcalc=func(a,bint)(sumint){
  29. returna+b
  30. }
  31. fmt.Println(add.foo(1,2))
  32. fmt.Println(add.Ree(1,2))
  33. Print(add)
  34. //Anonymous function
  35. Print(func(){
  36. fmt.Println(“helloanonymousfunction”)
  37. })
  38. //Methods; closure
  39. f:=errors.New(“helloerror”).Error
  40. Print(f)
  41.  
  42. }
  43.  
  44. //go:noinline
  45. funcPrint(iinterface{}){
  46. v:=reflect.ValueOf(i)
  47. FMT. Println (“type”, v.type(). String())
  48. Fmt.println (“address”, V)
  49. fmt.Println()
  50. }

Operation effect

The above code list mainly prints out the types and memory addresses of the four functions.

Compile and run, and the output is as follows:

Deep understanding function of go language

In the process of memory analysis in this paper, there are many operations to calculate memory address by offset.

It mainly involves two sections. Text and. Rodata. In this program, their information is as follows:

Deep understanding function of go language

Ordinary function

Take fmt.printf, a common function, as an example to study the type information of ordinary functions.

As can be seen from the above run output results, the string representation of fmt.printf function type is:

  1. func(string,…interface{})(int,error)

Dynamic debugging

Set a breakpoint at the entry of the print function to view the type information of the fmt.printf function.

Deep understanding function of go language

The type information of fmt.printf function is plotted as follows:

Deep understanding function of go language

  • rtype.size = 8
  • rtype.ptrdata = 8
  • rtype.hash = 0xd9fb8597
  • rtype.tflag = 2 = reflect.tflagExtraStar
  • rtype.align = 8
  • rtype.fieldAlign = 8
  • rtype.kind = 0x33
  • rtype.equal = 0 = nil
  • rtype.str = 0x00005c90 => *func(string, …interface {}) (int, error)
  • rtype.ptrToThis = 0
  • funcType.inCount = 2
  • funcType.outCount = 0x8002
  • funcType.inTypes = [ 0x4a4860, 0x4a2f80 ]
  • funcType.outTypes = [ 0x4a41e0, 0x4a9860 ]

constant pointer

The size of the function object is 8 bytes (rtype. Size) and contains 8 bytes of pointer data (rtype. Ptrdata), so we can treat the function object as a pointer.

In other words, fmt.printf is actually a pointer, but the pointer is an immutable constant. This is consistent with C / C + +. The function name is a pointer constant.

Type name

rtype.tflag = 2 = reflect.tflagExtraStar

The fmt.printf function has its own data type, but the type has no name.

Data category

The data category (kind) is calculated as follows:

  1. constkindMask=(1<<5)-1
  2.  
  3. func(t*rtype)Kind()Kind{returnKind(t.kind&kindMask)}

0x33 & 31 = 19 = reflect.Func

Variable parameters

The number of parameters (functype. Incount) of fmt.printf function is 2, and the number of return values is also 2. Why is the value of functype.outcount 0x8002?

The reason is that the functype.outcount field not only needs to record the number of returned values of the function, but also needs to mark whether the last parameter of the function is a variable parameter type; If yes, set the highest bit of the functype.outcount field value to 1.

In the reflect / type.go source file, the method to judge variable parameters is as follows:

  1. func(t*rtype)IsVariadic()bool{
  2. ift.Kind()!=Func{
  3. panic(“reflect:IsVariadicofnon-functype”+t.String())
  4. }
  5. tt:=(*funcType)(unsafe.Pointer(t))
  6. returntt.outCount&(1<<15)!=0
  7. }

The returned value quantity is calculated as follows:

  1. outCount:=funcType.outCount&(1<<15-1)

It is curious how the variable parameter tag is not saved in the functype.outcount field.

Parameter and return value types

In the fmt.printf function definition, the types of parameters and return values are:

  • string
  • …interface{}
  • int
  • error

In the function type information of memory, the type pointers of parameters and return values are saved; View their type information through these pointers as follows:

Deep understanding function of go language

From the memory data, you can see that the data category (kind) of the parameters and return values of fmt.printf function are as follows:

  • reflect.String
  • reflect.Slice
  • reflect.Int
  • reflect.Interface

For a detailed introduction to integers and their types, please read integers in memory.

For a detailed introduction to strings and their types, please read strings in memory.

In go language, error is special. It is both a keyword and an interface definition. As for interface types, special articles will be published for in-depth analysis and will not be introduced for the time being.

About slice, the article slice in memory once introduced [] int in detail.

Obviously, the second parameter of the fmt.printf function is not [] int. let’s see what kind of slice it is from the memory data.

Deep understanding function of go language

As can be seen from the figure above, the compiler compiles the variable parameter type… Interface {} in the source code into [] interface {}, thus turning the variable parameter into a parameter.

This way of handling variable parameters is very similar to the Java language.

Through in-depth analysis and understanding of the types of fmt.printf functions, we can easily understand the function related interfaces in the reflection package; If you are interested, you can take a look at the source implementation. I believe it is relatively simple to compare the type information of fmt.printf function.

  1. typeTypeinterface{
  2. … / / omit irrelevant interfaces
  3. IsVariadic()bool
  4. NumIn()int
  5. NumOut()int
  6. In(iint)Type
  7. Out(iint)Type
  8. … / / omit irrelevant interfaces
  9. }

Declared function type

In go language, any data type can be defined through the type keyword, which is very powerful.

In the code listing of this article, we use the type keyword to define the calc type, which is obviously a function type.

type calc func (a, b int) (sum int)

Is this type different from the fmt.printf function type? Using the same method as above, let’s study it in depth.

Dynamic debugging

Deep understanding function of go language

It can be seen from the memory data that the add variable of calc type points to an anonymous function, which is named main.main.func1 by the compiler.

The type information of calc is very complex, with 128 bytes in total. The chart is as follows:

Deep understanding function of go language

  • rtype.size = 8
  • rtype.ptrdata = 8
  • rtype.hash = 0x405feca1
  • rtype.tflag = 7 = reflect.tflagUncommon | reflect.tflagExtraStar | reflect.tflagNamed
  • rtype.align = 8
  • rtype.fieldAlign = 8
  • rtype.kind = 0x33
  • rtype.equal = 0 = nil
  • rtype.str = 0x00002253 => *main.calc
  • rtype.ptrToThis = 0x0000ec60
  • funcType.inCount = 2
  • funcType.outCount = 1
  • funcType.inTypes = [ 0x4a41e0, 0x4a41e0 ]
  • funcType.outTypes = [ 0x4a41e0 ]
  • uncommonType.pkgPath = 0x0000034c => main
  • uncommonType.mcount = 2
  • uncommonType.xcount = 1
  • uncommonType.moff = 0x48
  • method[0].name = 0x000001a8 => Ree
  • method[0].mtyp = 0xffffffff
  • method[0].ifn = 0x00098240
  • method[0].tfn = 0x00098240
  • method[1].name = 0x000001f6 => foo
  • method[1].mtyp = 0xffffffff
  • method[1].ifn = 0x000981e0
  • method[1].tfn = 0x000981e0

Type name

The rtype.tflag field contains the reflect.tflagnamed tag, indicating that the type has a name.

The name of the calc type is Calc, and the acquisition method is defined in the reflect / type.go source file:

  1. func(t*rtype)hasName()bool{
  2. returnt.tflag&tflagNamed!=0
  3. }
  4.  
  5. func(t*rtype)Name()string{
  6. if!t.hasName(){
  7. return””
  8. }
  9. s:=t.String()
  10. i:=len(s)-1
  11. fori>=0&&s[i]!=’.'{
  12. i–
  13. }
  14. returns[i+1:]
  15. }
  16.  
  17. func(t*rtype)String()string{
  18. s:=t.nameOff(t.str).name()
  19. ift.tflag&tflagExtraStar!=0{
  20. returns[1:]
  21. }
  22. returns
  23. }

Type pointer

  1. rtype.ptrToThis=0x0000ec60

This value is the offset from the program. Rodata section. In this program, the starting address of the. Rodata section is 0x49a000.

The pointer type of calc type is * Calc, and the type information is saved at the address 0x49a000 + 0x0000ec60. Pointer types are not discussed further in this article.

Parameters and return values

The calc type has two parameters and one return value, and they are all int types (the information is saved at the 0x4a41e0 address).

Type method

Methods are essentially functions.

In a tour of go( https://tour.golang.org/methods/1 )In, the function is defined as:

  1. Amethodisafunctionwithaspecialreceiverargument.

Calc is a function type. It is a clever design that function types can actually have their own methods.

The rtype.tflag field of calc type contains the reflect.tflaguncommon tag, indicating that its type information contains uncommontype data.

The size of the uncommontype object is 16 bytes. The calc type has three parameters and return values, and the three type pointers account for 24 bytes. Therefore, the offset of the [mcount] method from the uncommontype object is 16 + 24 = 40 bytes.

The following results are obtained through calculation:

Deep understanding function of go language

The REE method of calc type is renamed as main.calc.ree, and the memory address is 0x00098240 + 0x401000 = 0x499240. It is an export function, so reflect. Name. Bytes [0] = 1.

The foo method of calc type is renamed as main.calc.foo, and the memory address is 0x000981e0 + 0x401000 = 0x4991e0.

It can be seen from the memory analysis results that if multiple methods are defined for a data type, and some public export methods with names beginning with uppercase letters and some private methods with names beginning with lowercase letters, the compiler will sort the public export method information first and the private method information second, and then save it in its data type information. Moreover, this conclusion can be confirmed by the two methods defined in the reflect / type.go source file:

  1. func(t*uncommonType)methods()[]method{
  2. ift.mcount==0{
  3. returnnil
  4. }
  5. return(*[1<<16]method)(add(unsafe.Pointer(t),uintptr(t.moff),”t.mcount>0″))[:t.mcount:t.mcount]
  6. }
  7.  
  8. func(t*uncommonType)exportedMethods()[]method{
  9. ift.xcount==0{
  10. returnnil
  11. }
  12. return(*[1<<16]method)(add(unsafe.Pointer(t),uintptr(t.moff),”t.xcount>0″))[:t.xcount:t.xcount]
  13. }

In this example, you can also see that the method.mtyp field value corresponding to both REE and foo methods is 0xFFFFFF, that is – 1.

From the comments of the resolvetypeoff function in the Runtime / type.go source file, – 1 indicates that there is no corresponding type information.

That is, although the REE and foo methods of calc type are also functions, they have no corresponding function type information.

Therefore, the go compiler does not generate the corresponding type information for each function, but only when necessary, or the runtime generates it as needed.

Anonymous function

In the code listing, the third call to the main. Print function outputs the type information of an anonymous function. This anonymous function does not form a closure, so it is relatively simple.

Deep understanding function of go language

Sort the memory data into a chart as follows:

Deep understanding function of go language

This function has no parameters, return values and methods, so its type information is very simple. I believe there is no need for further introduction.

summary

Through step-by-step memory analysis, I have an in-depth understanding of the functions of go language, learned a lot of knowledge and solved many doubts. I believe that I will be able to handle it easily in practical development and avoid some small pits.

Recommended Today

Datawhale pandas punch in – Chapter 4 grouping

Today, I’m learning Chapter 4 – grouping, which I think is a very important and useful knowledge of pandas. This chapter mainly introduces the application of AGG, transform, apply and other functions based on the growpy function.The textbook summarizes the three operations of grouping: aggregation, transformation and filtering. 1. Polymerization: The groupby object has defined […]