Practice and exploration of graffiti intelligent Dubbo go 100 million class traffic

Time:2020-10-27

Practice and exploration of graffiti intelligent Dubbo go 100 million class traffic

Author pan Tianying, GitHub ID @ pantianying, an open source enthusiast, works in graffiti intelligence

Dubbo is a High-Performance Lightweight RPC framework based on Java. Dubbo provides rich service governance functions and excellent extensibility. Dubbo go provides a unified service capability and standard between Java and golang, which is the main problem that needs to be solved at present. This paper is divided into two parts: practice and fast access, sharing the practical experience of doodle intelligent Dubbo go. The purpose is to help users quickly access Dubbo go RPC framework, hoping that we can avoid detours. In addition, the test code is based on Dubbo go version v1.4.0.

Practice of Dubbo go gateway

Practice and exploration of graffiti intelligent Dubbo go 100 million class traffic

The usage of Dubbo go in graffiti intelligence is shown in the figure above. Next, we will introduce the landing details in detail. We hope that the experience summarized in the production environment can help you.

1. Background

In graffiti intelligence, Dubbo go has been the preferred RPC framework for golang services to connect with the original Dubbo cluster. Among them, the representative open gateway system (hereinafter referred to as gateway, see the open source version https://github.com/dubbogo/dubbo-go-proxy )。 The gateway dynamically loads the internal Dubbo interface information and exposes it in the form of HTTP API. The gateway is intended to address the following pain points of the previous generation gateway.

  • Through the page configuration Dubbo interface opening rules, the steps are cumbersome and the authority is difficult to control;
  • The interface is not restful and not friendly to external developers;
  • Heavy dependence and high risk of upgrading;
  • Concurrent performance issues.

2. Architecture design

In view of the above pain points, we immediately prepare to design a new gateway architecture. The first is language selection. Golang’s coprocess call model makes golang very suitable for building IO intensive applications, and its application deployment is simpler than Java.

After investigation, we decided to use golang as the proxy coding language, and used Dubbo go to connect the Dubbo provider cluster. The business application on the provider side configures API configuration information in the form of annotations by using java plug-ins. The plug-in will update the configuration information and Dubbo interface metadata to the metadata registry (redis in the figure below). In this way, the configuration is transferred from the administration background page to the program code. When coding, developers can easily see the API description of Dubbo interface, without configuring the usage of API from another management background.

Practice and exploration of graffiti intelligent Dubbo go 100 million class traffic

3. Practice

As can be seen from the above figure, the gateway can dynamically load the Dubbo interface information, and calling the Dubbo interface is based on the Dubbo generalization call. In Dubbo go, the client does not need to build the interface code of the provider config.SetConsumerService And hessian.RegisterPOJO This makes it possible for the client to add and modify the interface dynamically. Demonstration code with generalization call in Apache / Dubbo sample / golang / generic / go client.

func test() {
    var appName = "UserProviderGer"
    var referenceConfig = config.ReferenceConfig{
        InterfaceName: "com.ikurento.user.UserProvider",
        Cluster:       "failover",
        Registry:      "hangzhouzk",
        Protocol:      dubbo.DUBBO,
        Generic:       true,
    }
    referenceConfig.GenericLoad(appName) // appName is the unique identification of RPCService
    time.Sleep(3 * time.Second)
    resp, err := referenceConfig.GetRPCService().(*config.GenericService).
        Invoke([]interface{}{"GetUser", []string{"java.lang.String"}, []interface{}{"A003"}})
    if err != nil {
        panic(err)
    }
}

The implementation of generalization calls is actually quite simple. Its function works in the filter layer of Dubbo. Generic filter has been added to the Dubbo filter chain as a default enabled filter. Its core logic is as follows:

func (ef *GenericFilter) Invoke(ctx context.Context, invoker protocol.Invoker, invocation protocol.Invocation) protocol.Result {
    if invocation.MethodName() == constant.GENERIC && len(invocation.Arguments()) == 3 {
        oldArguments := invocation.Arguments()
        if oldParams, ok := oldArguments[2].([]interface{}); ok {
            newParams := make([]hessian.Object, 0, len(oldParams))
            for i := range oldParams {
                newParams = append(newParams, hessian.Object(struct2MapAll(oldParams[i])))
            }
            newArguments := []interface{}{
                oldArguments[0],
                oldArguments[1],
                newParams,
            }
            newInvocation := invocation2.NewRPCInvocation(invocation.MethodName(), newArguments, invocation.Attachments())
            newInvocation.SetReply(invocation.Reply())
            return invoker.Invoke(ctx, newInvocation)
        }
    }
    return invoker.Invoke(ctx, invocation)
}

Generic filter transforms the structure parameters requested by users into map in a unified format (struct2mapall in the code), and changes the forward and de serialization operations of classes (struct in golang) into positive and de serialization operations of maps. This eliminates the need for POJO descriptions to be hard coded into the hessain library.

As can be seen from the above code, there are four contents that need to be built dynamically in the generalization call: interfacename required in referenceconfig, method in parameters, parametertypes, and actual input parameter requestparameters.

So how do you get these parameters from the HTTP API match?

The plug-in for collecting metadata from the provider mentioned above will be used here. After the plug-in is introduced, the application will scan the Dubbo interface to be exposed at startup, and associate Dubbo metadata with HTTP API. The usage of the plug-in is roughly as follows. Here, a few simple configurations are adjusted as an example. During actual production, the annotation content will be more.

Practice and exploration of graffiti intelligent Dubbo go 100 million class traffic

The final Dubbo metadata is as follows:

{
    "key": "POST:/hello/{uid}/add",
    "interfaceName": "com.tuya.hello.service.template.IUserServer",
    "methodName": "addUser",
    "parameterTypes": ["com.tuya.gateway.Context", "java.lang.String", "com.tuya.hello.User"],
    "parameterNames": ["context", "uid", "userInfo"],
    "updateTimestamp": "1234567890",
    "permissionDO":{},
    "voMap": {
        "userInfo": {
            "name": "java.lang.String",
            "sex": "java.lang.String",
            "age": "java.lang.Integer"
        }
    },
    "parameterNameHumpToLine": true,
    "resultFiledHumpToLine": false,
    "protocolName": "dubbo",
  .......
}

Gateway can match an API request to a Dubbo interface by subscribing to the above information from the metadata configuration center. Then grab the parameters from the API request as input parameters. This function completes the flow closed loop.

For the above contents, you should have a clear understanding of the project topology of the gateway. I then shared the problems and tuning experience of the project in using Dubbo go. At the beginning of 19, the Dubbo go project was only in the early stage of construction, and there was no user experience. I was also involved in community development while coding the company’s internal Gateway project. After solving the problems of hessain serialization and zookeeper registry, the project finally ran through the closed loop. However, as a core application, there is still a long way to go before running through the closed loop, especially the new framework with stability to be tested at that time. The whole test and functional completion took a whole quarter of time until the project became stable and the pressure test effect was good. A single gateway machine (2C 8G) full link simulated real environment pressure test up to 2000 QPS. Due to the introduction of heavy business logic (an average of three Dubbo interfaces are called by a single request), the pressure test results are in line with or even exceed the expectations.

This paper summarizes some experience of Dubbo go parameter configuration optimization, mainly network related configuration.

When you run a demo, you should see that there are a bunch of configurations at the end of the configuration file. However, if you are not familiar with the Dubbo go underlying network model, it is difficult to understand the meaning of these configurations. At present, the network layer of Dubbo go takes Getty as the underlying framework to realize the separation of read and write and the management of the coprocess pool. Getty exposes the concept of session to the public. Session provides a series of network layer method injection implementations, because this article is not a source code parsing document, so it is not discussed here. Readers can simply think that Dubbo go maintains a Getty session pool and a session maintains a TCP connection pool. For each connection, Getty will be accompanied by a read coroutine and a write coroutine to separate the read and write. Here, I try to use popular comments to help you sort out several configuration meanings that have a greater impact on performance

protocol_conf:
  #Here is the protocol independent configuration. Under the Dubbo protocol, most of the configurations are related to Getty session.
  dubbo:
      #A session always guarantees the connection_ Number the number of TCP connections. The default value is 16,
    #However, it is recommended that you configure a relatively small value. The general system does not need such a large number of connections.
    #Every reconnect_ Interval time, check the number of connections, if less than connection_ number,
    #Just make a connection. The default value of 300ms is 0 or none
    reconnect_interval: 0
    connection_number: 2
    #Interval between sending heartbeat by client
    heartbeat_period: "30s"
    #The timeout time of the session in oncron exceeds the session_ Timeout close session without return
    session_timeout: "30s"
    #Each client of Dubbo interface will maintain a maximum value of pool_ Session pool of size.
    #Each request selects one from the session pool. So the real number of TCP is the number of sessions * connection_ number,
    #And pool_ Size is the maximum number of sessions. The test concludes that four TCP connections are enough for the general program.
    pool_size: 4
    #The timeout time of session keeping alive, i.e. exceeding the session_ If the session is not used by the timeout time, the session will be closed
    pool_ttl: 600
    #The size of the coroutine pool that processes the return value
    gr_pool_size: 1200
    #The length of the buffer queue in the read data and coroutine pool is currently obsolete. Do not use buffered queues
    queue_len: 64
    queue_number: 60
    getty_session_param:
      compress_encoding: false
      tcp_no_delay: true
      tcp_keep_alive: true
      keep_alive_period: "120s"
      tcp_r_buf_size: 262144
      tcp_w_buf_size: 65536
      pkg_wq_size: 512
      tcp_ read_ Timeout: "1s" ා the timeout time for each packet read
      tcp_ write_ Timeout: "5S" ා the timeout time for each packet write
      wait_timeout: "1s" 
      max_ msg_ Len: 102400 ා maximum data transmission length
      session_name: "client"

Dubbo go fast access

The previous article has shown the practical results of Dubbo go in graffiti intelligence. Next, we introduce the way to quickly access Dubbo go.

Step 1: Hello World

The Dubbo go usage example is currently consistent with Dubbo and is placed in the Apache / Dubbo samples project. In the Dubbo sample / golang directory, users can choose the feature directory they are interested in to quickly test the code effect.

tree dubbo-samples/golang -L 1
dubbo-samples/golang
├── README.md
├── async
├── ci.sh
├── configcenter
├── direct
├── filter
├── general
├── generic
├── go.mod
├── go.sum
├── helloworld
├── multi_registry
└── registry

Let’s take Hello world as an example, according to Dubbo samples / golang/ README.md Start server and client respectively. You can try golang calling Java, Java calling golang, golang calling golang, Java calling Java. Dubbo go supports interworking with Dubbo in protocol.

Let’s start go server as an example. The registry uses zookeeper by default. First, make sure that the local zookeeper is working properly. Then execute the following command, and then you can see the log of the normal startup of your service.

export ARCH=mac
export ENV=dev
cd dubbo-samples/golang/helloworld/dubbo/go-server
sh ./assembly/$ARCH/$ENV.sh
cd ./target/darwin/user_info_server-2.6.0-20200608-1056-dev/
sh ./bin/load.sh start

Step 2: use Dubbo go in your project

Above, we run the use cases through community maintained test code and startup scripts. Next, we need to embed the Dubbo go framework into our own code. Many friends often encounter problems in this step, here I sort out some common problems, hoping to help you.

1) Environment variable

At present, Dubbo go has three environment variables to configure:

  • CONF_ CONSUMER_ FILE_ Path: the path of the configuration file on the consumer side, which is required when using consumer;
  • CONF_ PROVIDER_ FILE_ Path: provider side configuration file path, which is required when using provider;
  • APP_ LOG_ CONF_ File: log file path, required;
  • CONF_ ROUTER_ FILE_ Path: File router rule configuration file path, required when using file router.

2) Code Notes

  • Injection service: check whether the following code is executed
#Client
func init() {
  config.SetConsumerService(userProvider)
}
#Server
func init() {
  config.SetProviderService(new(UserProvider))
}
  • Injection serialization Description: check whether the following code is executed
hessian.RegisterJavaEnum(Gender(MAN))
  hessian.RegisterJavaEnum(Gender(WOMAN))
  hessian.RegisterPOJO(&User{})

3) Correct understanding of configuration files

  • The key under references / services, such as “userprovider” in the following example, needs to be consistent with the return value of service reference(), which is the key to identify the interface.

 

references:
"UserProvider":
  registry: "hangzhouzk"
  protocol : "dubbo"
  interface : "com.ikurento.user.UserProvider"
  cluster: "failover"
  methods :
  - name: "GetUser"
    retries: 3
  • If there is only one registry cluster, only one registry cluster needs to be configured. Multiple IPS are separated by commas, as follows:
registries :
"hangzhouzk":
  protocol: "zookeeper"
  timeout    : "3s"
  address: "172.16.120.181:2181,172.16.120.182:2181"
  username: ""
  password: ""

4) Java and go

Case of interaction between go and Java: in order to adapt to the hump format of Java, golang will automatically change the method and attribute initials to lowercase when calling Java services. Many students intentionally write java code as a parameter definition that adapts to golang, capitalizes the first letter, and finally fails to serialize and match.

Step 3: expand functions

Both Dubbo go and Dubbo provide rich extension mechanisms. You can implement custom module instead of Dubbo go default module, or add some new functions. For example, cluster, filter and router are implemented to meet the business requirements. These injection methods are exposed in Dubbo go / common / extension, allowing users to call and configure.

Alibaba programming summer phase 2 is in hot registration!

On May 25, 2020, the second phase of Alibaba summer of code (hereinafter referred to as ASoC) was officially launched, and the scale of the project was upgraded again. As many as 20 star open-source projects from the open source community, such as Apache Dubbo, Apache rocketmq, dragonfly and Nacos, have been launched There are 32 technical experts from Alibaba group, core members of open source community and Apache incubator tutors. The field involves micro services, containers and AI It aims to build a sincere, fair and fair open source internship platform with open source community, and make the Chinese open source community and developer elite recognized worldwide by Alibaba’s open source technology.

Click to learn more: https://developer.aliyun.com/topic/summerofcode2020

“Alibaba cloud original concerns micro service, Serverless, container, Service Mesh and other technology areas, focusing on cloud native technology trends, cloud native large-scale landing practice, do the best understanding of the official account of cloud native developers.”