The release of mosn 1.0 opens the evolution of new architecture


Wen | Zhu Dejiang (flower name: Rende), a technical expert of ant group, is responsible for the development of mosn core and pays attention to the evolution of cloud native traffic gateway.

The following contents are compiled from the sharing of sofastack’s fourth anniversary

Mosn 1.0 release

Mosn is a cloud native network proxy platform mainly developed in go language. It is open source by ant group and has been verified by the production level of hundreds of thousands of containers promoted by double 11.

After four years of vigorous development, with the joint efforts of 11 committers, more than 100 contributors and the whole community, after 27 small versions of iterations, mosn version 1.0 was officially released.

A mature and stable mosn 1.0 with open source users, commercial landing, community and embracing the original ecology of cloud is coming.

In addition to the full implementation of ant group, mosn is also widely used in the industry, such as the commercialization of industrial and Commercial Bank of China, as well as the production practices of Alibaba cloud, qunar, speed cloud and other enterprises.

At the same time, with the release of 1.0, mosn, which has entered its adolescence, will also start the evolution of a new generation of MOE architecture and run to the sea of stars.

Development history

The release of mosn 1.0 opens the evolution of new architecture
Mosn originated from service mesh. Originally, the calls between microservices were completed through relatively heavy SDK. When the SDK is upgraded, the application needs to be upgraded together with strong interference.

In order to solve this pain point, mosn is evolving towards stripping out the SDK. If there is a sidecar running mosn in the pod where the application is located, the application itself only needs to communicate with mosn to complete the whole service invocation process. Stripping out the SDK is equivalent to mosn evolving as an independent component, and its evolution process does not disturb the application itself. In fact, the internal benefits of ants are very obvious.

In the whole process of evolution, there are two deep experiences: one is that there is an independent sidecar, which can be decoupled from the business logic; In addition, in the era of cloud nativity, the control surface and data surface are divided into two independent components. As a component of the data surface, mosn has to connect with many control surface services in the evolution process. During this period, standardization is a very important issue. In the whole process of standardization, it is not as intuitive as business decoupling, but the longer it takes, the deeper it is.

Now mosn has been fully deployed within ants, with hundreds of thousands of pods and tens of millions of peak QPS.

Evolution of mosn

The release of mosn 1.0 opens the evolution of new architecture
2018: start to create;

2019: the coverage of core links was completed in Shuang 11, and mosn began to operate independently at the end of 19;

2020: officially recommended by istio in July. At the same time, mosn opened the exploration of commercialization and completed the landing of Jiangxi rural credit at the end of the year;

2021: connect with envoy, dapr and wasm ecology and cooperate with mainstream communities. In December of the same year, the commercialization was completed in ICBC, setting a new benchmark in the industry.

Mosn is not only fully launched within ants and commercialized landing practice, but also a gradually improved community. Mosn community currently has 11 committers, of which more than 70% are non ant committers and more than 100 contributors, which have undergone 28 small version iterations. Mosn also has many open source users. They put mosn in their own company and have made a lot of contributions to mosn.

The release of mosn 1.0 opens the evolution of new architecture

In addition to the contribution of the community to the mosn project, there are also contributions to other projects / communities, including Holmes, babassl, proxy wasm and other projects, as well as the docking with other ecological projects.

Generally speaking, mosn is mature and stable enough, with commercial landing, community, surrounding and ecology. Therefore, we choose this time point to release mosn version 1.0.

1.0’s core competence and expanding ecology

Mosn version 1.0’s landmark achievement is the integration of istio version 1.10.

As the software of network agent, mosn has supported TCP, UDP and transparent hijacking modes in core forwarding. In the East-West gateway scenario, mosn has many internal and private non-standard protocols. In addition to supporting HTTP standard protocol, mosn also has a very important xprotocol framework, which can be very simple and convenient to support private non-standard protocols. The built-in bolt and Dubbo protocols are also realized through xprotocol framework. We also support the automatic identification of multiple protocols, which is also the core and special capability support in the east-west traffic gateway.

Back end management and load balancing are basic conventional capabilities in the case of network agents. Mosn also supports connection pooling, active health check and various load balancing strategies.

In terms of core routing, mosn supports domain based virtualhost, introduces a very powerful variable support, makes complex routing rules through variables, and also supports metadata packet routing. There are also routing level timeout, retry configuration, and request header and response header processing.

In short, as a network agent platform, the general core competence mosn has been fully possessed.

At the same time, in the scenario of network agent, many extensions are usually required. To what extent has the extension ecology of mosn achieved?

The release of mosn 1.0 opens the evolution of new architecture

RPC Protocol: supported by Dubbo and sofabolt, and supported by state secrets based on babassl;

Control surface: mosn has been supported by istio;

Registration Center: sofa registry;

Observability: skywalking and Holmes automatically analyze and diagnose resource usage exceptions during go runtime.

In the gateway scenario, a lot of logic needs to be customized. In addition to the conventional use of go to write some filter extensions, it also supports the lightweight mode of go plugin, and also supports the wasm extension of proxy wasm standard to run in mosn. Sentinel is also connected in service governance.

Istio 1.10

The release of mosn 1.0 opens the evolution of new architecture
Mosn will communicate with istio through standard XDS protocol, which is a very standard way of use. At the same time, we are also actively participating in the construction of standards.

In the process of standard formulation, we actively participate in the discussion of proposals, such as current limiting and routing proto. It is because we have a lot of cooperation with istio that we can obtain the official recommendation of istio.

Mosn originated from the scenario of east-west traffic in servicemesh. After four years of efforts, we chose to release mosn version 1.0 at this time point. It is presented as a mature and stable version with commercialization, community and ecology. We welcome more people to use mosn, and we also welcome you to build and grow together.

MOE new architecture

What is the original intention of this vision?

What are the advantages of doing so?

What are the new developments in the exploration of MOE’s new architecture?

The release of mosn 1.0 opens the evolution of new architecture

First of all, enovy and mosn are two data planes on the market. They have their own characteristics. Enovy is written in C + +, and the processing performance will be relatively high. Mosn has high R & D and efficiency, and has a good ecology.

MOE is mosn plus enovy. We hope to integrate the advantages of the two, learn from each other, combine high performance and high R & D efficiency, and support us to become bigger and stronger and go further.

MOE architecture from the perspective of enovy, mosn, as a plug-in extension of enovy, makes a horizontal comparison among all the extension modes of enovy.

The release of mosn 1.0 opens the evolution of new architecture

1. First, Lua

Embedded script language has strong advantages, and its operation is simple. However, as a relatively small language, the disadvantage is also obvious – the ecology is not good. Our goal is to improve R & D efficiency. Lua can’t let us achieve our goal.

2. Wasm is an attractive scheme

In the early stage of wasm development, many things just stay on the vision. Many language support is unfriendly, and the runtime performance is not good enough, which is a real pain point.

3. External cross process communication

The performance of external cross process communication is poor, which is nearly an order of magnitude lower than that of CGO. Secondly, the management of many other external processes is complex. If there are different languages, different processes are required, and the management cost is relatively large.

Compared with MOE, MOE has two advantages:

  • Mosn has many service governance capabilities, which can be reused to the greatest extent;
  • The ecology of go language needs to write more extensions on the way of future evolution, and the efficient R & D efficiency of go can be used.

The release of mosn 1.0 opens the evolution of new architecture

Looking back at the whole architecture, from the perspective of mosn, envoy plays a role as the network runtime of mosn. The request will first pass through the network runtime (envoy), and then pass the CGO bridge to give the request information to the mosn. After the mosn completes the request logic, it will return the response to the network layer.

The release of mosn 1.0 opens the evolution of new architecture

At present, the architecture of MOE has been implemented in ant, and we have obtained the expected benefits.

As an important bridge between mosn and envoy, CGO’s performance largely determines the overall performance of MOE. The specific implementation of CGO includes two call directions from C to go and go to C. There are some implementation differences between the two call directions. Specifically, the MOE architecture is mainly from envoy to mosn, that is, from C to go.

The release of mosn 1.0 opens the evolution of new architecture

At present, an order of magnitude optimization improvement has been completed – from 1600 nanoseconds to 140 nanoseconds. (through the simplest local test, it basically covers the cost of CGO itself, without considering the complex logic in go.)

What is the concept of 140 nanoseconds?

It’s almost the same order of magnitude as go adjusting C, which is the current official implementation. (our current optimization has also been submitted to the official of go. We have passed several rounds of review and are still waiting for the review of other official members.)

Because go is cross platform, the current implementation only supports x86 / 64 system, and corresponding implementations need to be added to different architectures.

In terms of CGO, a lot of optimization space is also analyzed. For example, build an extra P mechanism, which corresponds to the extra m mechanism, to solve the contention for P resources in high load scenarios.

The other is register parameter transfer. Now C and go transfer parameters. Put the parameters into a structure. If you can use register parameter transfer instead, you should get better performance.

At present, the mosn filter has been supported to run in envoy. This part can be found in the open source warehouse. You are welcome to try it.…

MOE open source program

Provide the abstract API to enovy official, and then implement the extension of go based on the standard API (it will be completed in August). Complete the overall open source of MOE in the second half of the year, and welcome your continued attention.

Roadmap 2022

The release of mosn 1.0 opens the evolution of new architecture

This year, in addition to the continuous evolution and iteration in the east-west direction, it will also be a north-south gateway.

We will provide an open source product in combination with istio, which is a long-term plan and the direction in which we think the cloud native gateway may evolve in the future.

In roadmap in 2022, in addition to this core capability, for example, we will make a modular structure and exit gracefully (these have been implemented in version 1.0). In addition, the ecosystem of various micro services will also connect with more registration centers, such as configuration centers, cloud native and integrated istio1 10。 In particular, stability construction has been added. With more and more users of mosn, people have higher and higher voices for stability capability.

We have integrated the Holmes landed inside the ant into the open source mosn. For resource exceptions at runtime, we can capture the exception scene for analysis. About Holmes, we had a share before. You can read it if you are interested.…

North South gateway access

The release of mosn 1.0 opens the evolution of new architecture

In addition to the development plans of various capabilities in roadmap, a very important evolution direction is north-south gateway access. After version 1.0, the mosn will be upgraded to the MOE architecture. The North-South access gateway is a scenario with greater traffic, and will also be supported by the mosn of the new architecture.

Due to historical reasons, mosn has many gateway forms, such as internal spanner and mobilegw. And each gateway form and gateway data plane are implemented differently.

Our evolution direction is that the data plane uniformly uses mosn as the underlying support, and the control plane adopts the standard XDS protocol to connect with istio. In this way, both east-west and North-South directions can be connected in a standard way in the original way of cloud.

Based on the traditional north-south gateway architecture, it may be a difficult way for us to do cloud native evolution. We prefer to use mosn and MOE new architecture, which is a more cloud native architecture.

Why explore MOE architecture?

Mosn is not limited to east-west traffic, but focuses on unified network forwarding. And in the era of cloud origin, cloudy is a very realistic demand, which urges all networks to connect in a standard way. This is an important reason for the North-South access gateway. We hope to unify all network forwarding to support more application scenarios.

Of course, the North-South gateway will also face many challenges. As a centralized gateway, it has many configuration rules and higher requirements for stability and performance. Including istio, which we choose, will also face a scale challenge and how to face the cost of migrating from the old data side.

In the face of new challenges, CGO has been optimized to ensure performance, and the TLS protocol capability has been enhanced. At present, envoy’s capability in TLS protocol is more suitable for East-West gateway. In order to adapt to the North-South gateway, we will make some enhancements, such as dynamic certificate issuance and the support of single domain name and multiple certificates. In terms of stability, based on enovy’s multithreading model, process crash will have a greater impact than the multi process scheme. First, we will improve the recovery speed after crash.

The release of mosn 1.0 opens the evolution of new architecture

Our long-term goal is to build with the community, provide complete open source products, and cooperate based on open source to make the products bigger and stronger.

At present, as the presentation of data, mosn products can contain a complete set of solutions. Our first goal is to achieve the effect of out of the box.

The other is dual-mode support. First, mosn will support standard XDS, which is a potential evolution direction. Secondly, in the process of landing, mosn will not only keep XDS. Mosn will still support all registration centers and configuration centers. In this way, both sides can run at the same time in the process of business landing. Based on the original high-performance R & D efficiency, maintain convenient customization development ability.

Finally, it is hoped that mosn will become a unified network forwarding platform to support east-west, north-south traffic and support in cloudy scenarios.

When the data plane network can be unified, mosn will continue to explore in the direction of open source and commercialization, concentrate on doing more long-term things, and hope more friends can participate and build together.

Welcome to use mosn and grow together

Mosn official website:

Mosn GitHub address: