Remember a memory leak of golang

Time:2020-10-29

Program function

The main function of this program is to import the data in the file into the Clickhouse database.

[problem description]

Server memory runs out every once in a while

[problem analysis]

Because it is developed with go language, pprof, a popular tool in the industry, is used.

reference resources URL:https // cizixs.com/2017/09/11/profiling -golang-program/

Tool use and ideas:
1) Modify the source code first
2) Installation tool observation
3) According to the phenomenon of tool grabbing
4) Fix the memory defect code, and then fix the memory leak according to the analysis results
5) Release code for re tracking analysis

1) Modification code:
Before using this tool, you need to write a few lines of code in the code so that you can use the tool to collect data.

1 // reference pprof
2 import "net/http"
3 import_ "net/http/pprof"
4
5 // add a port monitor program to the main function
6 // since my code is originally a daemons, a new monitoring protocol is used to prevent blocking
7 func main(){
8 go func(){
9 http.ListenAndServe("0.0.0.0:80", nil)
10 }()
11 // other codes
12 ...
13 }

After the above source code transformation, re deploy to the server to observe the memory condition;
Memory can still be consumed again and continuously without releasing memory.

2) Install golang pprof program on the server to collect data.

Installation method: Yum install golang pprof

3) The advantage of this tool is that it can generate PDF or PNG directly after dump

1 [[email protected] ~]# go tool pprof /root/clickhouse_runner/clickhouse_mssql_e
tl http://0.0.0.0:80/debug/pprof/heap
2 Fetching profile over HTTP from http://0.0.0.0:80/debug/pprof/heap
3 Saved profile in /root/pprof/pprof.clickhouse_mssql_etl.alloc_objects.all
oc_space.inuse_objects.inuse_space.012.pb.gz
4 File: clickhouse_mssql_etl
5 Type: inuse_space
6 Time: Feb 5, 2020 at 4:15pm (CST)
7 Entering interactive mode (type "help" for commands, "o" for options)
8 (pprof) pdf
9 Generating report in profile003.pdf
10 (pprof) quit
11 [[email protected] ~]



Through the analysis of the above heap, it is obvious that the main memory usage in the code is in the driver of the Clickhouse, and the part calling the Clickhouse is not released during the creation of the memory (after a careful analysis of the memory GC logic of golang, it is due to the lag of GC speed, and the creation speed of the import program is very fast, so GC is getting slower and slower).

4) Find the source of the memory leak and start modifying the code
Source code before modification:

1 connect, err := sql.Open("clickhouse", connstring)
2 if err != nil {
3 return err
4 }
5 load_start := time.Now()
6 tx, err := connect.Begin()
7 if err != nil {
8 log.Println(full_filename, "begin err", err)
9 return err
10 }
11 stmt, err := tx.Prepare("insert ... values....")
12 if err != nil {
13 log.Println(full_filename, "preare err", err)
14 return err
15 }
16 _, er := stmt.Exec(...)
17 if er != nil {
18 log.Println("err", er)
19 }
20 er2 := tx.Commit()
21 if er2 != nil {
22 log.Println(db_view, "err", er2)
23 }
24 stmt.Close()
25 connect.Close()

//Through the analysis of our own code and Clickhouse driver code, we can conclude that there are two ways to improve memory leak
Leakage:
a. Modify the driver code in the Clickhouse and reset the memory immediately after executing the code, instead of waiting for GC to process:

1 func (stmt *stmt) Close() error {
2 stmt.ch.logf("[stmt] close")
3 // add and recycle the memory data again
4 if stmt.ch.block != nil {
5 stmt.ch.block.Reset()
6 }
7 return nil
8 }

b. Release stmt objects directly and use GC to recycle them automatically (after consideration, it is more reasonable to use this method)

1 stmt.Close()
2 connect.Close()
3 // add the stmt and connect objects to nil directly
4 //clear mem
5 stmt = nil
6 tx = nil
7 connect = nil

Complete code after modification:

1 connect, err := sql.Open("clickhouse", connstring)
2 if err != nil {
3 return err
4 }
5 load_start := time.Now()
6 tx, err := connect.Begin()
7 if err != nil {
8 log.Println(full_filename, "begin err", err)
9 return err
10 }
11 stmt, err := tx.Prepare("insert ... values....")
12 if err != nil {
13 log.Println(full_filename, "preare err", err)
14 return err
15 }
16 _, er := stmt.Exec(...)
17 if er != nil {
18 log.Println("err", er)
19 }
20 er2 := tx.Commit()
21 if er2 != nil {
22 log.Println(db_view, "err", er2)
23 }
24 stmt.Close()
25 connect.Close()
26
27 //***** clear mem for gc ******
28 stmt = nil
29 tx = nil
30 connect = nil
31 //////////////////////////////////////////////////////////////////////////////////

5) After publishing the modified code, we observed that the system memory can be recovered and released normally

[Conclusion]

After the debugging of golang, the real reason is that the GC memory is not released in time and there is lag (through other servers, it is found that when the pressure is small, the memory can be released normally).
Therefore, when it comes to using large objects or frequently creating memory in golang, the best practice is to set the object to obj = nil, and tell GC that I really don’t use this memory block anymore, so that GC can recycle quickly and reduce iterative GC.
In addition, this method can be applied to languages such as Java, C ා, which have similar problems.

Recommended Today

PHP 12th week function learning record

sha1() effect sha1()Function to evaluate the value of a stringSHA-1Hash. usage sha1(string,raw) case <?php $str = “Hello”; echo sha1($str); ?> result f7ff9e8b7bb2e09b70935a5d785e0cc5d9d0abf0 sha1_file() effect sha1_file()Function calculation fileSHA-1Hash. usage sha1_file(file,raw) case <?php $filename = “test.txt”; $sha1file = sha1_file($filename); echo $sha1file; ?> result aaf4c61ddcc5e8a2dabede0f3b482cd9aea9434d similar_text() effect similar_text()Function to calculate the similarity between two strings. usage similar_text(string1,string2,percent) case […]