Alibaba’s way of microservice open source


The great Xia is for the country and the people.

In Jin Yong’s novels, Guo Jing and Huang Rong are the models of “great Xia, for the country and the people”. They helped the song army guard Xiangyang for more than ten years with their clothes.

In the world of technology, there is no difference in size.

But when a group of programmers are transformed from internal service companies to social programmers, and they collaborate, improve and use technology in an open-source way with community developers, they are given greater responsibilities and expectations.

Alibaba cloud intelligent middleware programmers are working with community developers to tap the past and future of domestic microservice open source projects with a keyboard.

The first Apache community top project of non Hadoop ecosystem in China

In the 2016 double 11, rocketmq team applied the low latency storage solution to the support of double 11 for the first time, withstanding the traffic test. During the whole promotion period, 99.996% of the delay fell within 10ms, and completed the established goal of ensuring the stability of transactions.

For the distributed message engine with almost balanced read-write ratio, this technical breakthrough, even in the global scope, is absolutely commendable.

On the other hand, after three months of open source reconstruction, rocketmq team started to donate to the Apache Software Foundation, but it was not easy to take this step.

“At that time, the domestic open source atmosphere was not as active as it is now. After the open source, many design documents, code quality and community construction were not ideal. We have been expecting that domestic open-source projects and open-source communities can also play an eye-catching value on the world open-source stage, and we hope that more “made in China” open-source projects will become world-class open-source projects. “Feng Jia, a senior technology expert with Alibaba cloud intelligence, recalled.

After nearly a year’s efforts, on September 25, 2017, the Apache Software Foundation officially announced that rocketmq, an open-source project donated by Alibaba to the Apache community, graduated from the Apache community and became the Apache top project (TLP), which is the first Apache community top project in China that is not a Hadoop ecosystem.

It is worth mentioning that according to the statistics before graduation, 80% of rocketmq’s new features and ecological integration come from the contribution of the community.

More than a year after graduation, rocketmq has covered more than 60% of the information scenarios in Internet Finance and other fields, and has been applied to more than a dozen industries, including finance, power, logistics, games, e-commerce, and shared travel. However, with the in-depth promotion of cloud computing, big data, artificial intelligence and other technologies in the global scope, new application scenarios such as IOT, blockchain, AI, edge computing and so on have emerged. How to further evolve the architecture to better adapt to the new scenarios and serve the next decade is the problem to be solved by rocketmq 5.0.

Message domain milestones

When rocketmq graduated from the Apache community, another milestone event occurred in the field of messaging. The open source project of openmessaging, an international standard in the field of distributed messaging, officially entered the Linux foundation, which is the first international standard in the field of distributed computing launched globally in China.

Message communication has become a key part of modern data-driven architecture, but there are still two problems in the field of message in the world:

First, the lack of vendor neutral industry standards leads to high complexity and incompatibility of various message oriented middleware, which correspondingly leads to product inefficiency, confusion and vendor lock-in.

Second, the existing solution framework does not fit the Cloud Architecture well, that is, non cloud native architecture, so it cannot effectively provide technical support for emerging business needs such as big data, flow computing and the Internet of things.

This is also a problem often mentioned by developers and partners in the process of rocketmq open source: “in the field of message, there are various kinds of open source solutions in the market, which leads to higher access and maintenance costs for users. In order to ensure normal communication between various message engines, a lot of efforts should be devoted to compatibility.”

At this time, to establish a set of vendor neutral, language independent factual standards in the field of information becomes the common appeal of all community members.

Since then, Alibaba has launched the openmessaging project and invited Yahoo, Didi travel and streamlio to participate in it. A year later, there are as many as 20 enterprises participating in the openmessaging open source standard community, including Alibaba, datapipeline, Didi travel, Haojing technology, Jingdong Mall, iFLYTEK, Qingyun qingcloud, streamlio, vipkid, Weizhong bank, Yahoo China Mobile Suzhou R & D center, etc. (in alphabetical order), in addition, it has also obtained the support of Apache rocketmq, Apache pulsar and other top information open source manufacturers.

Compared with an open source distributed message project, a set of open source standards can be accepted by various manufacturers, which is a milestone event for the whole domestic open source field.

From microservice framework to microservice ecology

Dubbo is Alibaba’s open-source distributed service governance framework in 2012. It is one of the most influential and widely used open-source service frameworks in China. In 2016, 2017 and 2018, open source China launched the most popular open source software selection in China, which has been listed in the top 10 list for three consecutive years. Dubbo released 2.7.0 in February 2019, which will be used for the official graduation of Apache foundation. (already in the near graduation stage)

After graduation from Apache incubator, apart from having a reputation, what is the impact on the maintenance and development of the project?

“Graduation from an incubator is an honor, but it’s not the end, it’s the beginning. It’s a bit like going to school. Graduation does not mean the interruption of study, but the beginning of exerting greater social value. Graduation is also more like a rite of passage, which means that the Dubbo team has met Apache’s requirements for a mature open source project and has begun to have the ability to develop independently. “Beiwei, a senior Alibaba cloud intelligent technology expert, replied in an interview with the media.

Up to now, Dubbo has gained 2.5W + star, ranking in the top 10 of all Java projects in GitHub, and more and more enterprise users choose Dubbo as their own micro service governance framework. However, with the deepening of micro service, the capabilities provided by Dubbo are gradually unable to meet the needs of all aspects of micro service.

Wang Tao, an Alibaba cloud intelligent technology expert, said in a live broadcast“Dubbo is a microservice framework that helps developers quickly build high-performance microservice applications. However, in API gateway, fuse current limiting, distributed monitoring, distributed transaction and other aspects, there is a lack of a relatively complete solution around Dubbo, which is basically self-developed by each company, or needs to research and select various open-source frameworks outside, which has spent a lot of time and energy on this, but it is unable to form a systematic solution. “

Therefore, we have made a further evolution, that is, from the microservice framework to the microservice ecosystem.

Through integration with mature open-source solutions, a complete micro service ecosystem is formed to form Dubbo ecosystem. Developers can quickly develop micro service applications without making too many changes to the existing system.

The concept of Dubbo ecosystem can not be put forward without two open-source microservice components in summer 2018.

Midsummer night for techies

In the summer of 2018, there were two new members in the domestic open source field.

As two important open-source components under microservices and cloud native ecosystem, Nacos focuses on dynamic service discovery, configuration and service management in cloud native applications, while sentinel focuses on flow restriction and degradation.

Nacos and sentinel are generated from the precipitation of Ali’s core business scenarios in the past 10 years. Their open source is an effective supplement to the open source technology solutions in micro services and meta-native fields, and they also attach great importance to integrating into the open source ecology. Besides being compatible with Dubbo, they also support the ecology such as spring cloud and kubenetes to enhance their vitality.

“When Alibaba upgraded from the IOE centralized application architecture to the Internet distributed service architecture in 2007, it realized that in the distributed environment, such as distributed service governance, data source disaster recovery switching, remote multi activity, contingency plan and flow restriction rules, configuration changes are difficult, because in a large distributed system, you can’t integrate A distributed system stops to make a software, hardware or system upgrade. “Kun Yu, a senior technology expert of Alibaba cloud intelligence, shared on qcon site.

Compared with other open source solutions of service configuration center, Nacos started a little late, but in addition to configuration center,It also provides the functions of dynamic service discovery, service sharing and management, with better performance in large-scale scenarios, more convenient in ease of use, and more flexible in distributed deployment.

Nacos supports a variety of start-up modes. Users can freely choose from different functional modules, such as registration center and configuration center, to deploy separately or jointly, so as to fully support the growth of small start-ups to large enterprises and the evolution of micro service life cycle.

Up to now, more than 40 enterprises have deployed Nacos to the production environment. For example, Huya live is the first batch of typical users who introduced Nacos to the production environment on a large scale.

“Huya started to pay attention to Nacos from v0.2, and we also participated in the construction of the community. It can be said that Huya is a relatively early enterprise user. Before the introduction of Nacos, we also compared spring cloud config server, zookeeper and ectd. Based on our current microservice system and business scenarios, we decided to use Nacos as a solution for service registration and service discovery in service transformation. During the use, we found that with the continuous updating of the community version and the in-depth practice of Huya, the advantages of Nacos are far more than those found in the survey. “Zhang Bo, head of middleware team of Huya basic support department, said at a developer activity.

Coincidentally, on the one hand, Nacos announced open source and was listed in the CNCF cloud native panorama, and on the other hand, Netflix Eureka, the service registration and discovery component under the spring cloud ecosystem, announced that it stopped open source investment. The game of brave people is full of variables,But in the view of Nacos team, this game can go to the end by itself, because we are not alone in the battle. Nacos is just one of many open source projects in Ali, and then more open source projects will be fed back to the community to form an ecosystem, such as sentinel, a lightweight current limiting and degrading component.

On July 29, 2018, at aliwareopen source Shenzhen station, there were more than 700 developers who could only accommodate 400 people. Zijin, a senior intelligent technology expert of Alibaba cloud, announced the open source of sentinel, a lightweight current limiting and degrading component.

Sentinel has experienced the test of double 11 in 10 years, covering all the core scenes of Ali, and thus accumulated a large number of traffic consolidation scenes and production practices. Sentinel’s emergence is inseparable from the joint efforts of Alibaba’s previous high availability architecture teams.

“Capacity planning is one of the most important and challenging aspects in the preparation for the double 11. Since the first year, the 0:00 time of the double 11 represents the highest traffic volume in our history, which is usually dozens or even hundreds of times of daily traffic. Therefore, how to make a distributed site with continuous and complex technology and business to support the sudden traffic impact more smoothly is the problem we have been solving for the past 10 years. “You Ji, a senior technical expert of Alibaba cloud’s intelligent high availability architecture team, shared after a double 11 preparations.

In the past 10 years, capacity planning has experienced five stages: manual estimation, offline pressure measurement, online pressure measurement, full link pressure measurement, full link pressure measurement, isolation environment and elastic expansion. After the end of double 11 in 2013, the birth of full link voltage measurement solved the problem of capacity certainty. As an epoch-making technology, the realization of full link piezometry is a milestone event for the whole group.

! [in 2014, the high availability architecture team won the group CTO Award

Then, based on the whole link pressure test as the core, a series of supporting ecology related to capacity planning is created, which can not only improve the capacity, but also reduce the cost and improve the efficiency of the whole link. With the continuous evolution of capacity planning technology, from 2018, the high availability architecture team hopes to contribute the practice of these years in the generation environment to the community, and then there will be Sentinel’s open source.

Sentinel was listed in the cloud native panorama only two months after its open source, which is located in the quadrant of layout and management module. At the same time, sentinel is also listed in the cloud native panorama, which is an application high availability service AHAS that provides functions such as automatic detection of application architecture, fault injection high availability drill and one key application current limiting degradation.

Recently, sentinel contributedspring-cloud-circuitbreaker-sentinelThe module was formally incorporated into the spring cloud circuit breaker by the community. As a result, sentinel also joined the spring cloud circuit breaker club and became one of the official mainstream recommendation options of spring cloud.

Spring cloud officially recommends not only sentinel, but also spring cloud Alibaba

In 2018, a big event happened in China’s Java circle.

Spencer Gibb, the co-founder of spring cloud, announced on the blog page of spring official website that Alibaba open source spring cloud Alibaba and released the first preview version. Later, the official twitter of spring cloud also released this message.

It may be due to the impact of spring cloud Netflix’s reduced open-source investment. The popularity of spring cloud Alibaba after open-source exceeded the expectation of Ji Wang, a senior technical expert of Alibaba.

In an interview with opensource China, Ji Wang thought that “spring cloud Alibaba is a good news for Chinese Java developers, making up for the limitations of spring cloud’s native implementation in large-scale cluster scenarios. At present, there are many implementations of the spring cloud specification, such as Netflix’s own complete system, consul’s support for service registration and configuration management, zookeeper’s support for service registration, etc. However, each set of implementation has its own advantages and disadvantages more or less. Perhaps most spring cloud users are hard to realize the limitations of Netflix OSS and other implementations. Whether it’s service discovery, distributed configuration, or service invocation and fusing, they are not suitable for large-scale cluster scenarios. For example, we also encounter Eureka performance problems internally. Therefore, we will integrate our super large-scale cluster experience with a strong spring cloud ecosystem to achieve strong alliance, hoping to make some positive chemical changes to the industry. “

After the summer, the heat of open source continues

The advantage of efficiency is that we can focus our attention and time on the more creative things and do the more fulfilling things. For developers working in the field of engineering, they have a stronger sense of efficiency.

In September 2018, Ali will open source the java online diagnostic tool widely used internally, named Arthas. Maybe it hit the pain point of developers’ online troubleshooting. Within 147 days after the release of the first release after open source, Arthas obtained more than 1W stars, and more than 40 contributors participated in the open source contribution.

From this, we can not only see the popularity of Arthas in the developer community, but also find that more and more domestic developers are beginning to be good at using open source technology to accelerate business development, and we can’t help thinking that more domestic high-quality open source projects will get the attention and love of global developers in the future.

In the field of technology, the achievement of all milestones stems from a sense of technology.

Duanling, Alibaba cloud intelligent technology expert, recalled: “Arthas originated in 2015 in Alibaba. At that time, micro services were in the ascendant. On the one hand, our team focused on the implementation of spring boot to improve development efficiency. On the other hand, we hope to improve the ability and efficiency of the technical team in online troubleshooting. At that time, after selection and discussion, we chose greys (developed by Alibaba dukun @ oldanpushcart), a Java open-source online problem diagnosis tool, to provide a better application diagnosis experience. “

We have made a lot of improvements in the user experience: color UI, web console and Intranet one click diagnostics. Slowly, Arthas has become a necessary tool for many Alibaba technical colleagues to diagnose problems online. Although Arthas is widely praised in Ali, it is only a tool for its own use.Take it as an open source, use it as an open source, so on September 28, 2018, we officially opened Arthas, hoping to help Java developers improve diagnosis efficiency.

As more and more developers begin to use Arthas, many developer efficiency tools have built Arthas into their products, enriching the access and opening methods of Arthas, such as ide plug-in cloud toolkit.

Time has come to 2019.

Xuanhe, senior development engineer of Alibaba cloud intelligence, shares theDistributed event problem is always a technical itch in the process of application development. I can’t say it’s a pain point, because for a long time, the common coping strategy for distributed transaction problem is: don’t use it if you can, try to avoid it. However, with the general implementation of microservice architecture, the problem of distributed transactions is becoming more and more intractable. The solutions are not without, but they are either poor in performance, or highly intrusive and not easy to implement. In a word, it’s a little “unpleasant”.

This “unhappiness” is mainly reflected in the distributed transaction open-source middleware fescar. When Qing Ming, a senior development engineer of Alibaba cloud intelligence, announced on rocketmq meetup in January 2019 that the distributed transaction middleware fescar was officially open-source, within a week, fescar gained 3000 + stars and 58 issues discussed by the community. Then, the fescar project team sorted out and answered 13 questions that the developers focused on, such as the birth background, applicable scenarios, and the differences between fescar and other open source distributed transaction schemes.

Alibaba middleware team released TXC (Taobao transaction constructor) in 2014 and started to provide distributed transaction services for applications within the group. In 2016, after product transformation, TXC launched Alibaba cloud as GTS (global transaction service), becoming the only cloud distributed transaction product in the industry at that time. It was delivered to many external customers in the form of Alibaba cloud public cloud and proprietary cloud solutions, and was unanimously recognized by customers. In 2019, based on the technology accumulation of TXC and GTS, the middleware team launched the open-source project fescar (fast & easy commit and rollback, fescar) to build a distributed transaction solution together with the community.

TXC / GTS / fescar come down in one continuous line, which provides a unique answer to solve the distributed transaction problem under the microservice architecture. The vision of fescar is to make the use of distributed transactions as simple and efficient as the use of local transactions. The ultimate goal is to make fescar applicable to all distributed transaction scenarios.

Alibaba’s path to open source continues.

At the right time, Li Sanhong, a senior technical expert of Alibaba cloud intelligence, announced that Alibaba dragonwell, the only Chinese enterprise on the JCP supreme executive committee, will take a more active part in the maintenance of Java ecosystem.

Dragonwell means Longjing, a symbol of Chinese tea culture.

Author: middleware brother

Read the original text

This is the original content of yunqi community, which can not be reproduced without permission.