The origin and development of technology popularization service discovery and load balancing


The cause of the problem

In the stand-alone era, traditional software is mostly monolithic. People submit code to a code warehouse, which will lead to many problems, such as application expansion, difficult to understand and modify, limited expansion, and inability to scale on demand. How can single architecture solve the problem of multi person cooperation? Modularization, right, split by function, define programming interface (API) between modules, care about each other’s function but not the implementation.

The origin and development of technology popularization service discovery and load balancing

With the development of the times, stand-alone programs encounter the dual bottleneck of computing power and storage, and distributed architecture emerges as the times require. Single application can easily complete local function call through function name (ID). In distributed system, service (RPC / restful) can be used to call local function API) takes on a similar role, but service name alone is not enough to request service. Service name is only the identification of service capability (service type), and it also needs to indicate where the service is located in the network. Service instance IP deployed in the cloud is dynamically allocated, and expansion, failure and update make the problem more complicated. Static configuration of service instance can not adapt to new changes and needs to be more refined In order to solve or simplify this problem,As a basic capability, service discovery is abstracted and provided. It tries to make the request network service as simple and transparent as calling local function.

The origin and development of technology popularization service discovery and load balancing

Service is function. It is only when services are closely linked with the network that the term “network service” appears. Service providers publish services through the network, and service users request services through the network. Distributed systems break through the limitations of single machine computing power and storage, improve system stability, and make massive services with high concurrency and high availability possible. However, it also increases software complexity, New problems and challenges such as software layering, load balancing, microservice, service discovery / governance, distributed consistency are introduced.

Service discovery

Service is divided into service provider and service consumer If you want to provide massive service capabilities, a single service instance is obviously not enough. If you want to provide thousands of services, you need a place to record the mapping from service name to service instance list. Therefore, it is necessary to introduce a new role: service intermediary, which maintains a service registry The registry can be understood as a service dictionary, in which key is the service name and value is the service instance list; the service registry is a bridge between service providers and service consumers, which maintains the latest network location of service providers and other information, and is also the core part of service discovery.

When the service is started, put the service information into the service registry; when the service is terminated, remove the service information from the service registry.

When service consumers request services, they first go to the service registry to get the list of service providers by name, and then select a service instance from the list to request services from the instance.

This is the simplest service discovery model and the basic principle of service discovery. So far, it seems that everything is OK, but in fact, there are still several problems that have not been clarified.

Problems and Solutions

First question

If a service is not stopped normally, but is killed by the system, it will not have the opportunity to inform the service registry to delete its own service information. In this way, the registry will have an additional message pointing to the invalid service instance, but the service consumer does not know. What should be done? The solution is very simple: keep alive, and the service provider will send a message to the service intermediary regularly (for example, every 10 seconds) After receiving the keep alive message, the service intermediary updates the keep alive timestamp of the service instance. The service intermediary checks the timestamp regularly and removes the service instance from the registry if it is overdue.

Second question

How to inform service consumers of service instance list changes? There are only two methods, polling and pub sub. Polling is that the consumer actively asks whether the service list of the service intermediary has changed. If there is any change, the new service list will be sent to the consumer. If there are too many consumers, the service intermediary will have pressure to process polling messages. When there are many service categories and service lists, it may even become a bottleneck. Pub sub is a service intermediary that actively notifies service consumers. Compared with polling, it has better timeliness. The disadvantage is that it will occupy separate threads or connection resources.

The origin and development of technology popularization service discovery and load balancing

Third question

What should we do if the service intermediary is suspended? So we need to solve the single point problem. We usually use clusters to combat this vulnerability. There are many open source solutions for service registry, such as etcd / zookeeper / consumer. In essence, we use distributed consistent database to save registry information. It not only solves the problem of read-write performance, but also improves the stability and availability of the system.

Fourth question

If the service consumer needs to query the service intermediary to obtain the instance list and then request the service every time using the remote service, the efficiency is too low? The pressure on the service intermediary is not small? Usually, the client will cache the service instance list, so that there is no need to repeatedly query the service with the same name, which reduces the delay and the pressure on the service intermediary.

Fifth question

The above mentioned keepalive has interval. If the service instance is not available within this interval, the service consumer will not be able to perceive it. Therefore, it is possible to send the request to a remote network machine that can not provide service. Naturally, it is impossible to work. We can’t put an end to this situation fundamentally. The system needs to tolerate this kind of error, but we can also make some improvements, such as blacking out when a service request fails to an instance, so as to avoid multiple requests to the same invalid service instance.

Sixth question

How can a service consumer choose one from multiple service instances? How to ensure that multiple service requests of the same service consumer are assigned to a fixed service instance (sometimes it is necessary)? This is actually the problem of load balancing. There are many strategies, such as RR, priority, weighted random and consistent hash.

Service discovery model

There are two modes of service discovery: client side discovery and server side discovery.

Client discovery mode

The origin and development of technology popularization service discovery and load balancing

The client is responsible for querying the service instance list and deciding which instance to request service from, that is, the load balancing strategy is implemented on the client. The model includes two parts: registration and discovery.

The service instance calls the registration interface of the service intermediary to register the instance. The service instance renews the service by keeping alive, and the service intermediary removes the unavailable service instance by health check.

When service consumers request services, they first query the service instance list from the service registry. The registry is a service database. In order to improve performance and reliability, the client usually caches the service list (the cache is used to ensure that the registry can continue to work after it is suspended). After getting the instance list, the client selects an instance to send service requests based on the load balancing strategy.


  • Directly, the client can flexibly implement the load balancing strategy.
  • Decentralized, non gateway, effectively avoid single point bottleneck and reliability degradation.
  • Service discovery is directly integrated into the client SDK, which has a good degree of language integration, good program execution performance and convenient debugging.


  • The client is coupled with the service registry, and the service discovery logic needs to be developed for each language and framework used by the service client.
  • This kind of intrusive integration will lead to any change of service discovery, which requires the client application to be recompiled and deployed. Strong binding violates the principle of independence.
  • Service up and down will have an impact on the caller, resulting in service temporarily unavailable.

Server discovery mode

The origin and development of technology popularization service discovery and load balancing

Discovery: service consumers send service requests through the load balancer. The load balancer will query the service registry, select a service instance, and forward the request to the service instance.

Registration: service registration / logout can be consistent with the above client discovery mode, or can be completed through the built-in service registration and discovery mechanism of the deployment platform, that is, the container deployment platform (docker / k8s) can actively discover service instances and help them complete registration and logout.

Compared with the client discovery mode, the client using the server discovery mode does not save the local service instance list, and the client does not do load balancing. This load balancer not only takes on the role of service discovery, but also takes on the role of gateway, so it is often called API gateway server.

Because the load balancer is centralized, it must also be a cluster. A single instance is not enough to support high concurrent access. Service discovery and load balancing for the load balancer itself usually rely on DNS.

HTTP server, nginx and nginx plus are load balancers of this kind of server discovery mode.


  • Service discovery is transparent to service consumers, service consumers are decoupled from the registry, and the update of service discovery function is insensitive to the client.
  • Service consumers only need to send requests to the load balancer, and do not need to develop service discovery logic SDK for each service consumer’s programming language and framework.


  • Since all requests are forwarded by load balancer, load balancer may become a new performance bottleneck.
  • The load balancer (service gateway) is centralized, and the centralized architecture will have the worry of stability.
  • Because the load balancer forwards the request, the RT will be higher than the client direct connection mode.

Microservice and service discovery

Service mesh service grid is a configurable infrastructure layer serving microservice applications, which aims to handle a large number of network-based inter process communication between services.

The origin and development of technology popularization service discovery and load balancing

Service mesh service gateway decouples call and communication. In non mesh environment, the application needs to do the perception of protocol and service discovery method. After using mesh, it only needs to call. Mesh controls the data flow of application through control surface.

Mesh service discovery is actually an upgraded version of client discovery mode, which is based on sidecar and pilot. Sidecars, namely data plane, is responsible for discovering the address list of target service instances and forwarding requests. Pilots, namely control plane, is responsible for managing all service registration information of service registry.

Service registration mode

One option is the service instance self registration mode. Another option is to use other system components to manage the registration of service instances, that is, third party registration mode.

As mentioned earlier, self registration mode is simple enough and does not require third-party components. The disadvantage is that registration code must be implemented for each programming language and framework used in the service.

The third party registration service instance will not complete the registration and deregistration by itself. It is in the charge of another system component called service register, which will poll the deployment environment or track subscription events to sense the change of the service instance and help the service instance complete the automatic registration and deregistration.

The main advantage of the third party registration mode is that it decouples the service and the service registry. There is no need to implement service registration logic for every language and framework. Service instance registration is implemented by a dedicated service. The disadvantage is that in addition to being built into the deployment environment, it is also a highly available system component and needs to be started and managed.


If a service has many service instances, for example, in some head companies, a service name may correspond to tens of thousands of service instances. In this way, the query and comparison of service changes will be very slow, and the amount of IO will be much larger than expected. Usually, version num is used to solve this problem.

Click follow to learn about Huawei’s new cloud technology for the first time~

Recommended Today

Flutter synthetic games

Flutter synthetic gamesGitHub source address: init.gif To realize the function, we need to understand the two shutter controls draggable and dragtarget It can be seen from the literal meaning that draggable can be dragged, and dragtarget is the drag target, which is to receive the control of draggable ddd.png We can only receive controls that […]