Openkruise: cloud native deployment base of Alibaba double 11 full link application

Time:2020-12-3

Introduction:Kruise is a homonym of cruise, ‘k’ for kubernetes, which means navigation and automatic patrol of applications on kubernetes. It is full of Alibaba’s best practices in large-scale application deployment, release and management for many years, as well as the precipitation of the demand of Alibaba cloud kubernetes for thousands of customers.

Source|Alibaba cloud official account

By Wang Siyu

Openkruise is a cloud native application automation engine opened by Alibaba cloud in June 2019. It is an application load project based on kubernetes standard, which can cooperate with native kubernetes It also provides more powerful and efficient capabilities for managing application containers, sidecars, and image distribution, so as to solve the problems of large-scale operation and large-scale station building of applications on kubernetes through automation in different dimensions, including deployment, upgrade, elastic capacity expansion, QoS adjustment, health check, migration and repair, etc.

Openkruise: cloud native deployment base of Alibaba double 11 full link application

Kruise is a homonym of cruise, ‘k’ for kubernetes, which means navigation and automatic patrol of applications on kubernetes. It is full of Alibaba’s best practices in large-scale application deployment, release and management for many years, as well as the precipitation of the demand of Alibaba cloud kubernetes for thousands of customers.

Openkruise: cloud native deployment base of Alibaba double 11 full link application

In the process of Alibaba economy’s overall cloud origin, Ali’s technical team has gradually developed a set of technical concepts and best practices that are close to the upstream community standards and adapt to the large-scale scene of the Internet. Among them, the most important is undoubtedly how to release, run and manage the application automatically. Therefore, the alicloud container team feeds these capabilities back to the community through openkruse, with a view to guiding the industry’s cloud original biochemical best practices and avoiding detours.

This year’s double 11, Alibaba has realized the comprehensive cloud original biochemistry of the core system. By 2020, Alibaba has run nearly 100000 openkruse workloads and managed millions of containers.

  1. More than 95% of the openkruse code running internally comes from the community repository

The following figure shows the relationship between the openkruse version and the open source version running inside Alibaba

Openkruise: cloud native deployment base of Alibaba double 11 full link application

As can be seen from the above figure: openkruse on GitHub is the upstream warehouse of our main body, while the internal downstream warehouse only implements a small number of internal coupling functions based on public interface (this part of code only accounts for less than 5%), that is to say, more than 95% of the code of openkruse running inside Alibaba is completely from the community warehouse.

Two points are worth explaining

  • All common capabilities will be developed and submitted directly based on the open source repository, and then synchronized to the internal environment.
  • Every line of code contributed by community members to openkruse will run in the internal environment of Ali.
  1. Automating application workload management on kubernetes

Students who do business at the upper level may lack the concept of “workload”. Here is a brief introduction. I don’t know if you’ve ever been curious about the implementation of each application scaling and publishing operation? In the cloud native environment, we describe the application deployment requirements (number of machines, mirror versions, etc.) in the end state oriented way, as shown in the following figure:

Openkruise: cloud native deployment base of Alibaba double 11 full link application

The application load mainly refers to the yaml definition and the corresponding controller

When the application expands or shrinks, PAAS (operation and maintenance platform) will modify the number of machines required in the yaml object mentioned above (for example, the number of machines to be expanded from 10 to 110, and that of 5 to 105). Then, the controller will adjust the actual number of pods (containers) according to the expected number of workload. When an application triggers a release or rollback, PAAS (operation and maintenance platform) will modify the mirror version and release policy in the yaml object mentioned above, and the controller will rebuild all managed pods (containers) into the desired version according to the specified release policy (these are just some simplified descriptions that are easy to understand, and the actual working mechanism will be much more complicated).

In other words, the application workload manages the lifecycle of all application containers. Not only does application expansion, reduction and release depend on workload, but workload is also responsible for maintaining the number of pods (containers) in the application runtime to ensure that there are continuous running instances that meet the expected number. If a host fails and the above application instance is expelled, workload will immediately expand a new container for the application.

  1. Double 11 core applications are fully deployed based on openkruse

With the cloud of Alibaba economy, the e-commerce business and middleware applications related to the double-11 entities are all migrated to the cloud native environment, and the application load capacity of openkruse is uniformly used for deployment. Openkruse provides a variety of different types of workload to support different deployment methods:

Openkruise: cloud native deployment base of Alibaba double 11 full link application

  • Cloneset: (stateless application) this is the largest part, and most of the pan e-commerce business is deployed and released through cloneset.
  • Advanced stateful set: (stateful application) is mainly used for middleware deployment in cloud native environment.
  • Sidecar set: (sidecar Lifecycle Management) the defined sidecar container can be dynamically injected into the new pod. The operation and maintenance container and mesh container on the cloud are all added to the service pod through this mechanism.
  • Advanced daemonset: the host level daemons are deployed to all nodes, including various basic components used to configure network and storage for business containers.

Therefore, we can see that from the upper e-commerce business to the middleware, to the O & M container and basic components, the whole upstream and downstream links are deployed and run by the workload provided by openkruse. Whether it’s the number of machines running the application, version management, or emergency expansion, release and other operations, openkruse is always maintaining it.

Imagine what happens if openkruse fails?

  • If only the controller is hung, the application scaling and publishing operations fail.
  • However, if there are significant bugs in the controller logic, such as the calculation error of quantity or version number, it may even cause the business container to be deleted or upgraded to the wrong version.

Of course, in view of the above high-risk situations, we have made a lot of protective measures to ensure the stability and availability of the business.

This is Alibaba’s cloud deployment base. Openkruise almost carries the deployment management and operation and maintenance responsibilities of the full-scale double 11 business.

  1. Key capabilities

Where does openkruse come from? Or what problems or requirements led to the birth of openkruse?

When the cloud becomes the trend and the cloud native gradually becomes the standard, we find that the workload capability provided by kubernetes can not meet the needs of Alibaba’s super large-scale business scenarios

  • When the application is released, all containers have to drift and rebuild: it is almost unacceptable to us. In the peak period of release, if Alibaba’s large-scale applications are being rebuilt on a large scale, it will be a disastrous pressure on the business itself or other schedulers, middleware and network / storage components.
  • Stateless application payload cannot grayscale upgrade container.
  • Stateful set cannot upgrade containers in parallel.
  • There are many more, not to be listed here

In this context, openkruse appeared. Through new development (cloneset, sidecarset), or enhanced compatibility (Advanced stateful set, advanced daemonset), we can finally successfully implement the cloud native business.

Openkruise is the “in place upgrade”. With this capability, we can finally make the application release without the need to drift and rebuild the container, but only upgrade the image that needs to be updated on the original pod in place. There are too many benefits:

  • The publishing efficiency has been greatly improved. According to incomplete statistics, in most business scenarios, in-situ upgrade can increase the release speed by at least 80% compared with the full reconstruction upgrade: it not only saves the time-consuming of scheduling, network allocation and remote disk allocation, but also benefits from the fact that there are old images on the node and only a few incremental layers need to be pulled.
  • Before and after the release, the IP remains unchanged, the upgrade process of pod network is continuous, and the other containers in the pod, except those being upgraded, keep running normally.
  • The volume remains unchanged, and the mounted device of the original container is completely reused.
  • It ensures the certainty of the cluster and guarantees the cluster topology after the full link pressure test.

Of course, in addition to this, we have enhanced many other advanced capabilities to meet many business demands for large-scale scenarios. This article will not introduce them one by one. However, the following figure shows the functional comparison between openkruse and kubernetes native application loads for stateless and stateful applications

Openkruise: cloud native deployment base of Alibaba double 11 full link application

Openkruise has officially entered CNCF sandbox

On November 11, 2020, at this special time point, Alibaba technologists ushered in another big event: after all members of the CNCF Technical Supervision Committee voted, they unanimously agreed to officially upgrade the open source openkruise of Alibaba cloud to CNCF hosting project.

As mentioned in the beginning, openkruse has completed the community open source, and the internal and external versions are almost identical. In addition, we also provide openkruse to the application directory of Alibaba cloud container services. Any customer on the public cloud can install and use openkruse with one click, so as to truly realize the application of openkruse in the internal business, cloud products and open source community of Alibaba group“three-in-one”。 At present, the customers who use openkruise on ack mainly include Betta TV, Shentong, youzan, etc., while companies such as Ctrip and LYFT in the open source community are also users and contributors of openkruse.

Openkruise will open up the cloud native application load capacity tempered based on Alibaba’s super large-scale scenarios. It not only complements the important part of expanding application load in the cloud native community, but also provides cloud customers with years of management experience of Alibaba’s application deployment and the best practice results of cloud’s original biochemical process. Since the official open source date, the openkruse project has established several key milestones:

  • Five members of maintainer are from Alibaba, Tencent and LYFT
  • 44 contributors

    • Domestic: Alibaba cloud, ant group, Ctrip, Tencent, pinduoduo
    • Abroad, lysid, Microsoft
  • 1900+ GitHub Stars
  • 300+ Forks

In the future, openkruse’s focus includes but is not limited to the following goals:

  • Continue to export the automation capability of general cloud native applications deposited in Alibaba, and take a sustainable Trinity development strategy.
  • In depth mining the application load requirements of subdivided domains, for example, we are exploring the pooling capability for FAAS scenarios.
  • It should be more closely combined with open source products in other related fields, such as OAM / kubevela, etc., to create a more complete cloud native application system.