By Zhang Yuanzheng
Source|Alibaba cloud official account
Reading guideDubbo as a distributed micro service framework, many companies in practice based on Dubbo for distributed system architecture. After restarting open source, we not only seeDubbo 3.0 the latest “roadmap” releaseIn addition, we can also see that Ali is starting to promote its e-commerceIntegration of Dubbo and internal HSF, and began to use Dubbo 3.0 on double 11. This paper is the sharing of ICBC’s financial micro service architecture based on Dubbo, mainly about the coping strategies and achievements of service discovery, the subsequent release of ICBC’s practice of large-scale service monitoring and governance, and how to develop Dubbo from the perspective of enterprises. Welcome to pay attention.
Background and overview
The traditional business system of ICBC is generally based on Jee single architecture. Facing the development trend of online and diversified financial business, the traditional architecture has been unable to meet the needs of business. Therefore, starting from 2014, ICBC chose a business system for service-oriented attempt, verified, evaluated and compared several distributed service frameworks at that time, and finally chose Dubbo, which is relatively perfect and has been used by many domestic companies. At the same time, ICBC also made enterprise customization for Dubbo to help this business system complete the implementation of service, and also received very good results after it went online.
In 2015, ICBC began to expand the implementation scope of service architecture. On the one hand, it helped traditional business system to carry out architecture transformation, on the other hand, it gradually precipitated super large scale service groups similar to middle platform to support the combination and reuse of fast service of business system. With the accumulation of experience, ICBC has continuously optimized Dubbo iteratively and customized it, and gradually built a perfect service ecosystem around service.
In 2019, ICBC’s Micro service system will be officially upgraded to one of the key capabilities of the core banking system of ICBC’s open platform, helping ICBC’s IT architecture to achieve a real distributed transformation.
The composition of ICBC’s Micro service system is as follows:
- In terms of infrastructure, both the service nodes of the business system and the work nodes of the micro service platform have been deployed on the cloud platform of ICBC.
- In the aspect of service registration discovery, in addition to the conventional service registration center, the metadata center is also deployed to realize the service registration discovery by node.
- In the aspect of service configuration, through the external distributed configuration center, the unified management and distribution of various dynamic parameters can be realized.
- In the aspect of service monitoring, it realizes the unified collection and storage of various service operation indicators, and connects with the enterprise monitoring platform.
- In the aspect of service tracking, it is mainly used to track the whole link of the service in real time to help the business system quickly locate the fault point and accurately evaluate the impact scope of the fault.
- In order to meet the needs of traditional business system to access services, the service gateway realizes the capabilities of automatic discovery, automatic subscription and protocol conversion (HTTP protocol to RPC Protocol) of new services and new versions on top of Dubbo service subscription and RPC capabilities, and realizes 7 × 24-hour uninterrupted operation.
- Service governance platform provides a one-stop management, monitoring and query platform for operation and maintenance personnel and development and testing personnel to improve the efficiency of daily service governance.
The biggest challenge
After years of practice, this paper summarizes the following two major challenges:
- Performance and capacityAt present, the number of online services (that is, the number of service interfaces in the Dubbo concept) has exceeded 20000, and the number of provider entries in each registry (that is, the cumulative number of all providers of each service) has exceeded 700000. According to the assessment, in the future, we need to support 100000 level services and 5 million level providers per registry.
- High availabilityThe goal of ICBC is: any node failure of the micro service platform can not affect online transactions. The business system of the bank runs 7 × 24 hours. Even in the time window of version launch, the launch time of each business system is staggered. The nodes of the platform need to be upgraded. How to avoid the impact on online transactions, especially the version update of the registry itself.
This article will first share the coping strategies and effects of ICBC from the aspect of service discovery.
Difficulties and optimization of service discovery
In Dubbo, the registration, subscription and invocation of service is a standard paradigm. The service is registered when the service provider initializes, and the service consumer subscribes to the service and gets the full list of providers when the service consumer initializes. During the run time, when the service provider changes, the service consumer can get the latest provider list. The point-to-point RPC call between consumers and providers does not go through the registry.
As for the choice of registration center, ICBC chose zookeeper in 2014. Zookeeper has large-scale applications in various scenarios of the industry, and supports cluster deployment. Data consistency between nodes is guaranteed by CP mode.
Inside zookeeper, Dubbo will set up different nodes according to services. Under each service node, there are four word nodes: Providers, consumers, configurations and routers
- Providers temporary node: record the list of service providers. The offline sub nodes of the provider are automatically deleted. Through the watch mechanism of zookeeper, consumers can know that the list of providers has changed at the first time.
- Consumers temporary node: record the list of consumers, which is mainly used to query consumers during service governance.
- Configurations persistent node: mainly save the service parameters that need to be adjusted during service governance.
- routers: the child node is a persistent node, which is mainly used to configure the dynamic routing policy of the service.
In the online production environment, zookeeper sub data center has deployed multiple clusters, each cluster has five election nodes and several observer nodes. Observer node is a new node type introduced in zookeeper version 3.3.3. It does not participate in the election, only listens to the voting results, and other capabilities are the same as follower node. The observer node has the following advantages:
- Shunt network pressure: with the increase of service nodes, if the clients are connected to the voting nodes, it will consume a lot of CPU to process network connections and requests. However, the election node can not be expanded at any level. The more election nodes, the longer the transaction voting process, which is unfavorable to high concurrent write performance.
- Reduce cross city and cross DC subscription traffic: when there are 100 consumers who need to subscribe to the same service across cities, the observer can uniformly handle this part of cross city network traffic to avoid pressure on inter city network bandwidth.
- Client isolation: Several observer nodes can be specially assigned to a key application to ensure its network traffic isolation.
- problem analysis
Based on the sad history of using online zookeeper in recent years, ICBC summarized the problems faced by zookeeper as a service registration center
- With the increase of the number of services and service provider nodes, the amount of data pushed by services will grow explosively. For example, a service has 100 providers. When the provider starts up, because of the CP feature of zookeeper, the consumer will receive an event notification for each online provider, read the list of all current providers of the service from zookeeper, and then refresh the local cache. In this scenario, theoretically, each consumer receives 100 event notifications and reads 100 service provider lists from zookeeper, 1 + 2 + 3 +… + 100, totaling 5050 provider data. This problem is particularly prominent in the peak period of business system production, which easily leads to the network of zookeeper cluster being full, resulting in extremely low service subscription efficiency and further affecting the performance of service registration.
- With the increase of the number of nodes written on zookeeper, the snapshot file of zookeeper keeps growing. Every time a snapshot is written to disk, there will be a disk IO rush. During the peak period of production, because of the large amount of transactions, the frequency of writing snapshot files is also very high, which brings great risks to the infrastructure. At the same time, the larger the snapshot file is, the longer the recovery time of zookeeper node after failure is.
- When the zookeeper election node is re elected, the observer node has to synchronize the full transaction from the new leader node. If it takes too long at this stage, it is easy to cause the client session connected to the observer node to time out, causing the corresponding providers to fail All the temporary nodes under the node are deleted, that is, from the perspective of the registry, these services are offline, and the consumer side will report an abnormal error without a provider. Then, these providers will re connect to zookeeper and re register the service. This phenomenon of registration reversal of a large number of services in a short time often brings more serious performance problems of service registration push.
To sum up, we can draw the conclusion that zookeeper is generally competent as a registry, but it needs to be further optimized in a larger scale of service scenarios.
- Optimization scheme
The main optimization measures of ICBC include the following aspects: Subscription delay update, multiple mode of registry, upgrade to registration by node, etc.
1) Subscribe to delayed updates
ICBC optimizes the zookeeper client component zkclient, and makes a small delay for consumers to obtain the list of providers after receiving the event notification.
When zkclient receives the one-time event of childchange, installwatch () recovers the monitoring of the node through eventthread. At the same time, getchildren () is used to read all the child nodes under the node, obtain the list of providers, and refresh the local service provider cache. This is the root of the 5050 data problem.
After zkclient received the event of childchange(), ICBC made a delay and asked installwatch() to do what it was supposed to do. In this waiting process, if the service provider changes, the childchange event will not be generated.
Some people will ask if this violates the CP model of zookeeper. In fact, it is not. The data of zookeeper server is strongly consistent, and consumers also receive event notification. They just delay to read the list of providers. When getchildren() is executed later, they will read the latest data on zookeeper, so there is no problem.
The internal pressure test results show that when the service providers go online on a large scale, before the optimization, each consumer receives a total of 4.22 million data from the provider nodes. After one second of processing delay, the data volume becomes 260000, and the number of childchange events and network traffic become 5% of the original After this optimization, we will be able to cope with the on-line and off-line of a large number of services during the peak period of production.
2) Multiple mode
ICBC adopted and optimized the SPI implementation of registry multiple in the new Dubbo version to optimize the service subscription in the multi registry scenario.
The original processing logic of service consumers in Dubbo is as follows: when there are multiple registries, consumers filter providers according to the corresponding invoker cache of the registry. If the cache of the first registry is not found, consumers will go to the cache of the second registry. If there is a usability problem in the first registry at this time, and the data pushed to consumers is missing or even empty, it will affect the filtering process of consumers, such as the exception of no provider, unbalanced call load, etc.
The multiple registry merges the data pushed by multiple registries and then updates the cache. Therefore, even if a single registry fails and the pushed data is incomplete or empty, as long as the data of any other registry is complete, the final combined data will not be affected.
In addition, the multiple registry mechanism is also used in heterogeneous registry scenarios. In case of problems, the registry can be offline at any time. This process is completely transparent to service calls of service nodes, which is more suitable for gray pilot or emergency switching.
Furthermore, there are additional benefits. The reference object on the consumer side takes up more memory in the JVM. Through the multiple registry mode, the consumer side can save half the overhead of the invoker object. Therefore, the multiple registry mode is highly recommended for multiple registry scenarios.
3) Register by node
ICBC reversely migrates the service discovery logic of Dubbo 2.7 and Dubbo 3.0, and uses the service registration discovery model of “registration by node”. Here is the iron triangle combination of configuration center, metadata center and registry center
- Configuration center: it is mainly used to store the dynamic parameters at the node level and the data of persistent nodes such as configurations and routers originally written on zookeeper.
- Metadata Center: storage node metadata, that is, the mapping relationship between the name of each service node (that is, applicaiton name) and the services it provides, as well as the class definition information of each service, such as the input and output parameter information of each method.
- RegistryAt this time, the registry only needs to store the relationship between the name of the service provider node and the actual IP port.
The change of this model has no effect on the service invocation of consumers. According to the relationship between “service node name” and “service” in the metadata center, and the relationship between “service node name” in the registry and the actual IP port, the consumer generates a service provider invoker cache compatible with stock mode.
Pressure test results show that registration by node can make the amount of data on the registry become the original 1.68%, which has no pressure on the online zookeeper. 100000 level service and 100000 level nodes can be easily supported.
Planning for the future
In the future, ICBC also hopes to have the opportunity to go out, deeply participate in the community, and contribute its good features on Dubbo, zookeeper server, and zkclient. For example, in addition to the above optimization points, ICBC has also done fine identification of RPC results on Dubbo, PAAS adaptation, multi protocol on the same port, self isolation and other capabilities, and is still in zookeeper At the same time, we are studying the synchronization mechanism of observer to avoid a series of problems caused by full data synchronization.
In addition, from the perspective of the development of micro services, mesh has become one of the current hot spots. The main pain point of ICBC is the upgrade of service SDK version. Istio is not competitive, and the life and death of MCP is uncertain. At present, there are preliminary plans on how to smoothly transition the existing Dubbo service to mesh architecture, but there are still many technical difficulties to overcome.
Welcome Dubbo’s practical students to discuss the problems and experience in large-scale scene, and work together to make Dubbo’s enterprise landing better!
This article is the original content of Alibaba cloud and cannot be reproduced without permission.