Remember the performance optimization of 18 times at a time

Time:2022-5-6

background

Recently, a self-developed Dubbo registration center in charge often received alarms about CPU utilization, so it carried out a wave of optimization with good results. Therefore, it plans to share the thinking and optimization process, hoping to help you.

I’ll draw a sketch of what the Dubbo registration center is. Just feel it a little. It doesn’t matter if you can’t understand it. It doesn’t affect the subsequent understanding.

image

  • The service discovery requests (registration, logout and subscription) of the consumer and provider are sent to the agent, which is the sole agent
  • Registry and agent maintain a long grpc link. The purpose of the long link is to push the change of the provider to the corresponding consumer in time. In order to ensure the correctness of the data, a push-pull mechanism is adopted. The agent will pull the subscribed service list from the registry at regular intervals
  • Agents and business services are deployed on the same machine, similar to the idea of service mesh, to minimize business intrusion, so that they can iterate quickly

Back to today’s focus, the CPU utilization of this registry has been at a medium high level for a long time. Occasionally, applications are released, and the CPU will even be full when there is a large amount of push.

I didn’t feel it before because there are not many connected applications. In recent months, more and more applications have been connected, and slowly reached the alarm threshold.

Find optimization points

Since this project was written by go (it doesn’t matter if you don’t understand go. This article focuses on algorithm optimization, not on the use of tools), it’s still very simple to find out where to consume CPU: open pprof and go online to collect for a period of time.

For specific operation, please refer to my previous article. The knowledge and tools used in today’s article can be found in this article.

image

The CPU profile cuts part of the diagram. Others are not very important. You can see that it consumes more CPUAssembleCategoryProvidersMethod, which is directly related to

  • 2 redis related methods
  • One is calledassembleUrlWeightMethod of

For a little explanation, the assemblycategoryproviders method is to construct the URL that returns the Dubbo provider. Because some processing will be done on the URL when it is returned (such as adjusting the weight, etc.), it will involve the parsing of the Dubbo URL. In addition, due to the push-pull mode, the more online service users, the larger the QPS of this process, so it is not surprising that it occupies most of the CPU.

These two redis operations may be that serialization takes up CPU. The larger part is in assemblyurlweight, which is a little unclear.

Next, let’s analyze how to optimize assemblyurlweight. Because it occupies the most CPU, the optimization effect must be the best.

The following is the pseudo code of assemblyurlweight:

func AssembleUrlWeight(rawurl string, lidcWeight int) string {
	u, err := url.Parse(rawurl)
	if err != nil {
		return rawurl
	}

	values, err := url.ParseQuery(u.RawQuery)
	if err != nil {
		return rawurl
	}

	if values.Get("lidc_weight") != "" {
		return rawurl
	}

	endpointWeight := 100
	if values.Get("weight") != "" {
		endpointWeight, err = strconv.Atoi(values.Get("weight"))
		if err != nil {
			endpointWeight = 100
		}
	}

	values.Set("weight", strconv.Itoa(lidcWeight*endpointWeight))

	u.RawQuery = values.Encode()
	return u.String()
}

The parameter rawurl is the URL of the Dubbo provider, and lidcweight is the weight of the computer room. According to the configured machine room weight, recalculate the weight in the URL to realize the distribution of multi machine room traffic according to the weight.

This process involves the resolution of URL parameters, the calculation of weight, and finally restore to a URL

Dubbo’s URL structure is consistent with the ordinary URL structure, which is characterized by many parameters and no#The following fragment.

image

The CPU is mainly consumed in these two parsing and the final restore. We see that the purpose of these two parsing is to get the information in the URLlidc_weightandweightParameters.

url. Parse and URL Parsequery is an official library provided by go and implemented in various languages. Its core is to parse the URL into an object to easily obtain all parts of the URL.

If you understand the concept of information entropy, in fact, you probably know that it must be optimized.Shannon (Shannon)Referring to the concept of thermodynamics, the average amount of information excluding redundancy is calledInformation entropy

image

url. Parse and URL Parsequery must have redundancy in parsing in this scenario, which means that the CPU is doing redundant things.

Because there are usually many parameters for a Dubbo URL, we only need to take these two parameters, and the URL Parse parses all the parameters.

For example, given an array, find the maximum value. If you sort the array first and then get the maximum value, it is obvious that there is redundant operation.

The sorted array can not only take the maximum value, but also take the second largest value and the third largest value Minimum value, there is redundancy in information, so sorting first is definitely not the optimal solution to find the maximum value.

optimization

Optimize the performance of obtaining URL parameters

The first idea is not to parse all URLs, but only take the corresponding parameters, which is very similar to the algorithm problem we wrote, such as obtaining the weight parameter. It can only be in these two cases (it doesn’t exist #, so it’s much simpler):

  • dubbo://127.0.0.1:20880/org.newboo.basic.MyDemoService?weight=100&…
  • dubbo://127.0.0.1:20880/org.newboo.basic.MyDemoService?xx=yy&weight=100&…

Either&weight=, or?weight=, end or&, or go directly to the end of the string, and the code is easy to write. First write an algorithm for parsing parameters:

func GetUrlQueryParam(u string, key string) (string, error) {
	sb := strings.Builder{}
	sb.WriteString(key)
	sb.WriteString("=")
	index := strings.Index(u, sb.String())
	if (index == -1) || (index+len(key)+1 > len(u)) {
		return "", UrlParamNotExist
	}

	var value = strings.Builder{}
	for i := index + len(key) + 1; i < len(u); i++ {
		if i+1 > len(u) {
			break
		}
		if u[i:i+1] == "&" {
			break
		}
		value.WriteString(u[i : i+1])
	}
	return value.String(), nil
}

The original method of obtaining parameters can be extracted:

func getParamByUrlParse(ur string, key string) string {
	u, err := url.Parse(ur)
	if err != nil {
		return ""
	}

	values, err := url.ParseQuery(u.RawQuery)
	if err != nil {
		return ""
	}

	return values.Get(key)
}

First benchmark these two functions:

func BenchmarkGetQueryParam(b *testing.B) {
	for i := 0; i < b.N; i++ {
		getParamByUrlParse(u, "anyhost")
		getParamByUrlParse(u, "version")
		getParamByUrlParse(u, "not_exist")
	}
}

func BenchmarkGetQueryParamNew(b *testing.B) {
	for i := 0; i < b.N; i++ {
		GetUrlQueryParam(u, "anyhost")
		GetUrlQueryParam(u, "version")
		GetUrlQueryParam(u, "not_exist")
	}
}

Benchmark results are as follows:

BenchmarkGetQueryParam-4          103412              9708 ns/op
BenchmarkGetQueryParam-4          111794              9685 ns/op
BenchmarkGetQueryParam-4          115699              9818 ns/op
BenchmarkGetQueryParamNew-4      2961254               409 ns/op
BenchmarkGetQueryParamNew-4      2944274               406 ns/op
BenchmarkGetQueryParamNew-4      2895690               405 ns/op

You can see that the performance has been improved by more than 20 times

The newly written method has two small details. First, whether the parameter exists is distinguished in the return value, which will be used later; The second is the string operationstrings.Builder, which is also the result of the actual test+perhapsfmt.SpringfThe performance is not as good as this. If you are interested, you can test it.

Optimize URL write parameter performance

After calculating the weight, write the weight into the URL. Here is the optimized code:

func AssembleUrlWeightNew(rawurl string, lidcWeight int) string {
	if lidcWeight == 1 {
		return rawurl
	}

	lidcWeightStr, err1 := GetUrlQueryParam(rawurl, "lidc_weight")
	if err1 == nil && lidcWeightStr != "" {
		return rawurl
	}

	var err error
	endpointWeight := 100
	weightStr, err2 := GetUrlQueryParam(rawurl, "weight")
	if weightStr != "" {
		endpointWeight, err = strconv.Atoi(weightStr)
		if err != nil {
			endpointWeight = 100
		}
	}

	if err2 !=  Nil {// weight does not exist in URL
		finUrl := strings.Builder{}
		finUrl.WriteString(rawurl)
		if strings.Contains(rawurl, "?") {
			finUrl.WriteString("&weight=")
			finUrl.WriteString(strconv.Itoa(lidcWeight * endpointWeight))
			return finUrl.String()
		} else {
			finUrl.WriteString("?weight=")
			finUrl.WriteString(strconv.Itoa(lidcWeight * endpointWeight))
			return finUrl.String()
		}
	}Else {// weight exists in URL
		oldWeightStr := strings.Builder{}
		oldWeightStr.WriteString("weight=")
		oldWeightStr.WriteString(weightStr)

		newWeightStr := strings.Builder{}
		newWeightStr.WriteString("weight=")
		newWeightStr.WriteString(strconv.Itoa(lidcWeight * endpointWeight))
		return strings.ReplaceAll(rawurl, oldWeightStr.String(), newWeightStr.String())
	}
}

It is mainly divided into two cases to discuss whether there is weight in the URL:

  • If the weight parameter does not exist in the URL itself, a weight parameter should be spliced directly after the URL. Of course, pay attention to whether it exists?
  • If the weight parameter exists in the URL itself, the string is replaced directly

Careful, you must have found it again, whenlidcWeight = 1When, return directly becauselidcWeight = 1In fact, the following calculations do not work (Dubbo weight is 100 by default). Simply do not operate and save CPU.

After all optimization, make a benchmark:

func BenchmarkAssembleUrlWeight(b *testing.B) {
	for i := 0; i < b.N; i++ {
		for _, ut := range []string{u, u1, u2, u3} {
			AssembleUrlWeight(ut, 60)
		}
	}
}

func BenchmarkAssembleUrlWeightNew(b *testing.B) {
	for i := 0; i < b.N; i++ {
		for _, ut := range []string{u, u1, u2, u3} {
			AssembleUrlWeightNew(ut, 60)
		}
	}
}

The results are as follows:

BenchmarkAssembleUrlWeight-4               34275             33289 ns/op
BenchmarkAssembleUrlWeight-4               36646             32432 ns/op
BenchmarkAssembleUrlWeight-4               36702             32740 ns/op
BenchmarkAssembleUrlWeightNew-4           573684              1851 ns/op
BenchmarkAssembleUrlWeightNew-4           646952              1832 ns/op
BenchmarkAssembleUrlWeightNew-4           563392              1896 ns/op

The performance is improved by about 18 times, and this may still be relatively poor. If the incoming lidcweight = 1, the effect is better.

effect

After the optimization, the corresponding unit test was written for the changed method. After confirming that there was no problem, the system went online for observation, and the CPU idle rate increased by more than 10%

image

last

In fact, this article shows a very routine performance optimization of a go program, which is relatively simple. After reading it, you may have some questions:

  • Why parse URLs when pushing and pulling? Can’t you calculate it in advance and save it?
  • Why only this point is optimized? Can other points also be optimized?

For the first question, in fact, this is a historical problem. When you take over the system, it is like this. If there is a problem with the program, you can change the whole mechanism, which may take a long period and be prone to problems

image

The second question, in fact, has just been answered in passing. In this way, the change is the smallest and the income is the largest. Other points are not so easy to change. In the short term, the income is the most important. Of course, we plan to reconstruct the system later, but before reconstruction, such optimization is enough to solve the problem.