Development and practice of serverless container instance cube


Cube birth background

With the promotion and implementation of cloud native technology, the proportion of container technology in enterprise production environment is increasing. As a de facto standard for container orchestration, kubernetes is widely used in enterprise services. Ucloud container team launched kubernetes product uk8s in 2018. This product is based on ucloud public cloud environment and seamlessly integrates ucloud IAAs layer computing, network and storage services, so that customers can quickly obtain kubernetes clusters available for production and have the ability to flexibly control clusters.

However, in the process of uk8s product promotion and customer access, the container team has also received some user feedback:

  • Maintaining the kubernetes cluster adds an additional burden; In addition to managing applications, users also need back-end resources, and failed to realize application-centered business management.
  • The kubernetes system is relatively complex and the learning curve is steep, which requires the customer team to have a certain technical reserve; The same is true for customers who have used containers but have not tried kubernetes. On the one hand, they need to understand the technical system of kubernetes, and on the other hand, they need to modify the application architecture to adapt to kubernetes.
  • We hope to have a ready to use container product, which can directly pull up the application through the container without waiting for the virtual machine to be ready before deploying the application, so as to shorten the waiting time for application readiness.

In order to solve the above user problems, the container team has developed a new serverless container productCube, it is currently in the public beta stage. In addition to lowering the threshold for users to use containers, the product also has the following features:

1.Operation and maintenance free: there is no burden of maintenance resources, and there is no need to care about the running location. It takes the application as the center and the container image as the application packaging standard.

2.Pay on demand: pay according to the resources actually used by the application.

3.Automatic expansion and contraction capacity: Based on massive resources, API is provided, which can pull up and close applications on demand and automatically schedule resources.

4.High availability: This product is available and provides application self-healing ability.

Technology selection

In the process of realizing cube product features, the container team needs to solve several technical problems:1. Selection of vessel during operationPublic cloud products must consider the issue of multi tenant isolation. Unlike the isolation of uk8s products based on virtual machine, cube products directly run containers on the host physical machine. The container implemented by standard docker can not realize the strong isolation between different containers of different users on the same host. Therefore, cube products need a container runtime scheme with the characteristics of strong isolation of virtual machines and rapid container startup.

The container team noticed that AWS opened source the lightweight virtual machine firecracker, which has many advantages such as less resource consumption, fast startup speed and easy maintenance. It has been used in the actual production environment and is very in line with the cube business scenario. Therefore, it finally adopted the container runtime scheme based on firecracker lightweight virtual machine. As can be seen from the following two figures, through the special simplification and optimization of cloud computing scenarios, firetracker has obvious advantages in startup speed and memory consumption compared with the current mainstream virtualization component QEMU.

Development and practice of serverless container instance cube

VMM startup time and memory usage comparison, picture referencesource

2. Container management services

Container management services supporting virtual machine container runtime also have a variety of open source solutions, such as containerd / cri-o, Kata container and firetracker containerd. After comparison, the container team chose the combination of cri-o + firetracker containerd. The two can meet the requirements of stand-alone container management in terms of function, and compared with other types, the code architecture is clearer, and the call link is simple and clear, which is convenient for subsequent customization and transformation according to product requirements.

3. Container scheduling service

Kubernetes has become the de facto standard of container scheduling, with rich functions and good scalability. Therefore, the container team adopts kubernetes as the basic scheduling framework and makes relevant modifications according to the product requirements. The final basic service architecture is as follows:

Development and practice of serverless container instance cube

Optimization and improvement

Although the open source solution can speed up the development progress, some problems still need to be solved to meet the product requirements, mainly including the following aspects:

1. Container imageIn the standard container image implementation, the image is stored on the host through a hierarchical structure. When creating a container, the container runtime will create a writable layer above the mirror layer and mount it on the host for use by the container instance. However, the cube container does not run directly on the host, and there is no need to mount the container root directory on the host. Therefore, the container team modified the relevant implementation of the image layer in cri-o, and directly mounted the container writable layer in the form of block device to the lightweight virtual machine rather than the host, reducing the host’s interference with the cube container.

In addition, in order to solve the problem of slow startup of container instances caused by slow pulling of new images, the container team proposed a solution of remote mounting of images. Store the container image in the cache cluster in the form of a block device. When you need to generate a container instance on this image, first mount the container image to the host in the form of remote mount, and then create a writable layer on the host to generate a container instance when the container runs. At the same time, the background will synchronize the remote image to the host to further accelerate the reading and reduce the cluster risk. The above method can shorten the time for the host to obtain the image for the first time to less than 3S, and there is room for further optimization. At present, this function is provided to users in the form of image cache products, and is gradually integrated into the ordinary image pull process.

Development and practice of serverless container instance cube

2. Use public cloud resources

In terms of network, the network model of cube container is basically the same as that of virtual machine. After the relevant network functions are implemented in the form of CNI plug-ins, the cube container can be well connected to the public cloud VPC network.

In terms of storage, cube container currently supports two types of storage: network file system nfs that can read and write at multiple points and cloud disk Udisk that can only read and write at a single point. In terms of file storage function, cube product realizes the function of automatically mounting NFS in lightweight virtual machine. Users can directly use the network file system in the container by specifying the mounting point and mounting parameters in the configuration file, and can support user built NFS and ucloud public cloud product UFS in VPC network at the same time. In terms of block device function, the container team extended the implementation of firetracker block device. By adding support for Vhost user protocol, cube lightweight virtual machine can be directly connected to spdk service, so as to mount and use high-performance RSSD cloud disk.

3. Container operating environment

In order to reduce the consumption of additional resources, the container team has done a lot of optimization work on container management services and container runtime.

The container team modified the architecture of cri-o management container group and adopted the model of single pod corresponding to single shim. Managing all containers in a pod with one shim can significantly reduce shim resource consumption and simplify container management. For lightweight virtual machines, the container team has also fully simplified and optimized the kernel / rootfs / init process, retaining only the most basic functions to speed up startup, reduce security attack surface and reduce resource consumption. In addition, the container team also built the implementation of infra container into the lightweight virtual machine. When cube runs as a pod, there is no need to mount additional infra container.

Development and practice of serverless container instance cube

4. K8s transformation

As a general container scheduling framework, kubernetes can meet the needs of most container management. However, for the specific usage scenario of cube, the container team still needs to make some modifications to k8s components. On the control side, the container team adopts a customized scheduler, which can better meet the needs of task priority, scheduling speed and resource management in multi tenant scenarios. On the host node, in view of the characteristics of cube container runtime, the container team streamlined some functions that do not need to be implemented by kubelet, such as mounting the configmap / volume directory on the host, running the CNI plug-in, collecting specific directory logs, etc., which enhanced the isolation security between the container and the host.

Cube future outlook

After completing the above development and transformation, cube products were successfully launched and achieved good results. Subsequent cube products will continue to be updated iteratively along the idea of helping users improve efficiency, reduce expenses, simplify maintenance and save costs. In terms of container performance, the container team will continue to optimize the IO path of lightweight virtual machines, reduce the performance loss of virtualization and management components, and ensure the stable and efficient operation of user container instances. In terms of service management, cube products will launch a variety of container management controllers, and realize the ability of cube instances to directly connect to kubernetes cluster, so as to provide users with multi-level resource scheduling mode, which is convenient for users to manage and maintain according to their actual needs.

If you are interested in ucloud cube products, welcome to join the cube test exchange group by scanning the code! code automatic recognition)