Analysis of cloud native network agent mosn extension mechanism


< SOFA:Channel/ >Interesting and practical distributed architecture channel.
Review video and PPT view address at the end of the article. Welcome to join the live interactive nail group: 21992058, do not miss every live broadcast.

Analysis of cloud native network agent mosn extension mechanism

In this paper, according to sofachannel ා 14 live sharing, theme: cloud native network agent mosn extension mechanism analysis.

Hello everyone, I am Yongpeng, a lecturer from ant financial. I am mainly responsible for the development of mosn, and I am also the committer of mosn. Today, I share with you the extension mechanism of cloud native network agent mosn. I hope that after this sharing, you can understand the programmable expansion ability of mosn. You can carry out secondary development according to your actual business requirements based on the expansion ability of mosn.


Today, we will introduce the extension mechanism of mosn from the following aspects:

  • The expansion capability and mechanism of mosn are introduced in detail;
  • The filter extension mechanism and plug-in extension mechanism of mosn are introduced in detail with examples;
  • The planning and Prospect of mosn’s subsequent expansion capability;

Welcome to join hands to build mosn. The examples involved in this speech are in the examples / codes / MOS n extensions directory of GitHub. If you are interested, you can download them and run them. We have also done some small activities about these examples, and hope you can participate actively.


Introduction to mosn

As a cloud native network agent, mosn aims to provide multi protocol, modular, intelligent and secure agent capabilities for services. In the actual production and use, different manufacturers will have different use scenarios, the general network agent ability will appear some deficiencies in the face of specific business scenarios, usually need to carry out secondary development to meet the business needs. In the core framework, mosn provides a series of extension mechanisms and extension points, which is to meet the needs of secondary development based on business. At the same time, part of the general logic provided by mosn is also based on the implementation of extension mechanism and extension point.

Analysis of cloud native network agent mosn extension mechanism

For example, the ability of transparent hijacking through the “built-in implementation” of mosn is realized through the mosn filter mechanism. To implement a message broker, a similar extension can be used. In the case of general agent, the authentication and authentication of business can be realized through the filter mechanism, and the customized load balancing logic can be realized. Besides the extension of the forwarding process, mosn can also extend the implementation of log, which can be used to benchmark the existing log system and extend XDS Realize customized configuration update; according to different business scenarios, there will be many specific extensions, so we will not start here. Those who are interested can pay attention to the source code analysis series articles and documents being built by the mosn community.

Analysis of cloud native network agent mosn extension mechanism

As a network agent, mosn provides scalability in the network layer, protocol layer and forwarding layer of the forwarding link, and the configuration, log and admin API of the non forwarding link. For the part of protocol extension, those who are interested can take a look at the analysis of the multi protocol mechanism of mosn in the last live broadcast. Today, we will focus on the stream filter of the forwarding layer Extension mechanism and plug-in mechanism of mosn.

Stream filter mechanism

In the actual business scenario, before forwarding the request or writing back the response, it may be necessary to do some processing for the request / response, such as judging whether the authentication / authentication of forwarding is required, whether the current restriction is required, or some records with business semantics are needed for the request / response, and the protocol needs to be converted. These scenarios are highly coupled with specific business, which is a typical case of secondary development. The stream filter mechanism of mosn is designed to meet such extension scenarios, and it has become the most frequently used extension point in mosn extension.

Analysis of cloud native network agent mosn extension mechanism

In the current implementation of built-in mosn, the stream filter mechanism is temporarily bound to the built-in network filter: proxy. Later, we will also consider abstracting this part of capability so that other network filters can reuse this part of capability.

About stream filter, we will explain two parts today:

  1. What are the parts of a stream filter and how it works in mosn;
  2. Through a demo demonstration to deepen the implementation and application of stream filter;

A complete stream filter

A complete streamfilter contains three parts:

  • A streamfilter object exists in every request / response. It works when the mosn receives a request. We call it receiverfilter. It works when the mosn receives a response. We call it senderfilter. A streamfilter can be either of them or both;
  • A streamfilterfactory object for mosn to generate a streamfilter object each time it receives a request. When the listener configuration is parsed, a streamfilterfactory will be generated for a streamfilterfactory. The same streamfilterfactory may correspond to different streamfilterfactories under different listeners, but in some special cases, the streamfilterfactory may need to be implemented as a singleton;
  • A method of createstreamfilterfactory, which is called to generate streamfilterfactory during configuration resolution;

Analysis of cloud native network agent mosn extension mechanism

How stream filter works in mosn

Next, let’s take a look at how stream filter works in mosn.

Analysis of cloud native network agent mosn extension mechanism

When the mosn receives a complete request after protocol parsing, a stream will be created. At this time, every time a streamfilterfactory exists in the listener receiving the request, a streamfilter object will be generated, and then it will enter the proxy process.

After entering the proxy process, if there is a receiverfilter, the corresponding logic will be executed. The receiverfilter includes two stages: “before routing” and “after routing”. After each filter is processed, it will return a status. If it is stop, it will abort the subsequent receiverfilter that has not been executed. Generally, it will return the filter in stop state Will write back a response. If it is continue, the next receiverfilter will be executed until the receiverfilter of this stage is completed or terminated; after the receiverfilter of the pre routing phase is completed, the post routing phase will be executed, and its logic is consistent with that before routing. If it is normal forwarding, then mosn will receive a response or directly write back a response if it finds other exceptions. At this time, it will enter the senderfilter process to complete the senderfilter processing. After senderfilter processing is completed, mosn will write a response to the client and complete the final closeout. The closeout includes some data recycling, log recording, and “destroy” of streamfilter (calling ondestroy).

Stream Filter Demo

After having a basic understanding of stream filter, let’s take a look at an actual demo code to see how to implement a streamfilter and make it work in mosn.

Analysis of cloud native network agent mosn extension mechanism

According to our introduction, a streamfilter consists of three parts: filter, factory and createfactory.

  • First of all, we implement a filter whose logic is to simulate an authentication filter: only when the header of the request contains the configured key value, the mosn will continue to forward the request, otherwise it will directly return 403 error;
  • Then we implement a factory, which is responsible for generating the filter we implement, and explains the stage in which the filter should play a role (before the request phase and route matching);
  • Finally, we define a function called “createdemofactory” to generate demofactory and “register” it through init. After registration, the streamfilter can be identified by mosn configuration analysis;

Analysis of cloud native network agent mosn extension mechanism

After the implementation, we can achieve the corresponding functions through specific configuration. In the configuration of the example, the streamfilter is configured as the filter we just implemented, and only the forwarding header contains user:admin Request for. In the example configuration, the listening port is 2046 and the forwarding back-end server port is 8080. Before the demonstration, I have completed the startup of 8080 server, which will return 200 for any request received. Let’s take a look at mosn forwarding. Demo operation can be viewed in the live video review at the end of the article.

Stream Filter Demo:
Demo Readme:

Mosn plugin mechanism

Let’s take a look at the plugin mechanism of mosn.

Just now we have an understanding of stream filter. Other extensions in mosn are implemented in a similar way. The idea is to encode the interfaces required for mosn extension points, and then run the extension implementation using the framework of mosn.

Analysis of cloud native network agent mosn extension mechanism

However, a problem can be found here, that is, sometimes the extensibility we need has already been implemented. Can we make a simple transformation so that mosn can obtain the corresponding capabilities, even if the currently available implementation is not go Language implementation, such as the implementation of ready-made current limiting ability and injection capability; or for some specific capabilities, it needs more strict control and higher standards, such as security related capabilities.

In similar scenarios, we introduce the plug in mechanism of mosn, which enables us to independently develop the capabilities required by mosn, or we can introduce them into mosn after making appropriate modifications to existing programs.

The plug mechanism of mosn includes two parts. One is the self defined plug framework of mosn, which supports the realization of the extension ability of mosn by realizing the interaction between agent and an independent process in mosn. The other is based on the plug framework of golang, and the extension of mosn is realized by loading dynamic library (so). There are some limitations in the way of dynamic library loading, which is still in the beta stage.

Let’s take a look at the multiprocess plugin framework.

Multiprocess plugin framework

The plug framework of mosn is a way of mosn encapsulation, which allows mosn to interact with independent processes through grpc

  1. Independent processes are managed by the mosn plugin framework as sub processes of mosn, and the plug framework of mosn can manage them, such as startup and shutdown;
  2. Through the agent implemented in mosn, grpc can interact with subprocesses in the way of grpc. Grpc can be based on TCP or domain socket;

Analysis of cloud native network agent mosn extension mechanism

Based on this framework, we only need to develop or make some modifications to make the program meet the specification of mosn framework, and then we can be a part of the multi process plug-in of mosn.

First of all, we need to provide a grpc service and meet the proto definition under the mosn framework. After the grpc server is started, it outputs a agreed string to stdout as the handshake protocol between mosn and subprocesses. The corresponding agent in mosn will establish the connection with the subprocess through handshake protocol. The string of handshake protocol contains five fields, and each field is separated by “|”, in which the value marked with $is the value that needs to be filled in according to the actual process situation, and the rest are the current fixed fields. Network supports TCP / Unix, which represents whether to communicate through TCP or UNIX domain socket. Addr indicates the address monitored by grpc server.

Mosn provides a sub process server package of go language. In the go language scenario, the program as a subprocess only needs to implement a mosn framework plugin.Service Interface, and through the plugin.Serve Method can be started.

Through the plugin framework, mosn can support isolation, heterogeneous language extension, modularization and process management.

For the extension of mosn through multiple processes, we have prepared two examples to share with you today. One is the extension of TLS based on mosn, which simulates the ability to obtain TLS configuration certificate, private key and other sensitive information through a certificate management program with high security level; the second is to modify the stream filter demonstrated before to “sub process” to simulate “how to introduce ready-made capabilities” into mosn.

Extension example of TLS based on mosn

First, let’s look at the extension of TLS. The example includes two parts:

  • The independent sub process is implemented in go language plugin.Service Interface, and through the plugin.Serve Method start;
  • Mosn extension point implements interactive agent. The details of TLS extension point are not detailed here, but the interaction process: send grpc request through call method, get response and complete relevant logic;

Analysis of cloud native network agent mosn extension mechanism

load cert demo:
Demo Readme:

Let’s take a look at the effect. First, the port listening to 2046 is still configured. After configuring the extended TLS configuration, you need HTTPS to access the mosn.

Stream filter as an example of agent

Let’s take a look at the example between stream filter as an agent and multiple processes to simulate “how to introduce ready-made capabilities” into mosn. In the example, we consider the previous “authentication” as a “ready-made” capability.

Analysis of cloud native network agent mosn extension mechanism

Independent process implements the same “authentication” capability as before, and its configuration comes from the start parameters of the process. Stream filter is implemented as an agent, in which the “check” logic is modified to interact with the subprocess, and the startup and configuration settings of the subprocess are completed when the factory is generated.

After running this example, the effect of stream filter is the same as before.

Stream Filter Plugin demo:
Demo Readme:

Dynamic library (so) extension mechanism

In the current multiprocess framework, although the extensibility can be realized by an independent subroutine, it still needs to implement an agent for interaction in mosn, and still need to write a part of code in mosn. We hope to introduce the mechanism of dynamic library (so) loading, so that we can load different kinds of codes without recompiling mosn So, to achieve different expansion capabilities.

Analysis of cloud native network agent mosn extension mechanism

Compared with the subroutine mode, so is also an independent binary, but when it is finally started, there will be no additional subprocesses. Its life cycle can be completely consistent with that of mosn. Moreover, the dynamic library mechanism has another advantage: it can completely decouple the extended code and mosn.

However, there are still some limitations in the way of dynamic library loading. Therefore, mosn is still in the beta stage for this ability, and it has not been put into actual use and needs to be improved. Related reasons include:

  • Some mosn extensions need some definitions in mosn, so it can’t be decoupled completely in the implementation of dynamic library.

In order to solve this problem, mosn separates some basic libraries (such as logs, buffers, etc.) and some API definitions from the core repository of mosn. In this way, both the extension implementation and the mosn core refer to these “independent” libraries, reducing the dependence of extensions on the core code of mosn.

If an extension point wants to support fully decoupled dynamic library extension, the corresponding extension point needs to be modified to support dynamic library loading, including configuration model and implementation.

  • Mosn dynamic library loading method is actually based on the go language plugin package, it can load the dynamic library compiled with go language. However, there are some restrictions on the compilation environment of dynamic library, which must be consistent with the gopath of mosn when compiling; at the same time, the referenced code paths need to be consistent. If there is a vendor directory, it means that the project path when compiling the dynamic library must also be consistent with the mosn core.

To solve this problem, we consider using docker to compile, unify gopath at compile time, force to modify code directory structure, and shield vendor directory differences. This method is still under verification.

Therefore, theoretically, all extension points of mosn can be implemented by loading so using the native mechanism of go language. At present, the most suitable extension point of mosn is stream filter.

Since we only need to implement a specific streamfilter interface in the streamso, we only need to implement a specific interface So the so can be decoupled from the core framework of mosn.

The key point is the design and implementation of this universal filter. Let’s take a look at it through demo.

Design and implementation of general filter

This general filter is different from ordinary streamfilter in that it contains only one element: createfactory. The idea is to load and execute the createfactory in the so through the general createfactory, so that the factory in the so can play a role.

Analysis of cloud native network agent mosn extension mechanism

Generic createfactory includes:

  • There are two parts in the configuration analysis: one is the so path to be loaded, the other is the configuration of the corresponding filter in the so;
  • The so path represents the “registration” of the filter in the so, and this filter will be selected this time;
  • Load so and get the real function of createfactory based on the agreed function name;
  • Call the real createfactory function to load streamfilter in so;

From this, we can see that the streamfilter in so is different from the ordinary filter

  • The function generating streamfilterchainfactory must be a fixed name;
  • Init is no longer required to “register” the function;

Stream Filter SO Demo:
Demo Readme:

Let’s take a look at the effect of this demo. The filter implementation in this demo is still the previous “authentication” example. After verification, we found that this idea is feasible, but we need to improve more details from the production practice.

Code extension activities

After these demonstrations, I believe that you also have some understanding of the extension ability of mosn. Here we will do a code extension activity. I hope you can participate actively. After completing the activity task and submitting the relevant code PR to the mosn warehouse, we will conduct code review and verification. The code that passes the first verification will be merged into the example of mosn, and a reward will be given to the submitted students; for the top three submitted, the same result is correct and original, although we can’t merge the corresponding code, we will also send the reward.

There are five tasks:

  • After the independent process of certificate loading in multi process demo is implemented in Python or Java, the demo runs successfully. Any language is a task.
  • examples/codes/mosn-extensions/plugin/cert_loader/python/
  • examples/codes/mosn-extensions/plugin/cert_loader/java/
  • After the independent process of stream filter in multi process demo is implemented in Python or Java, the demo runs successfully. Any language is a task.
  • examples/codes/mosn-extensions/plugin/filter/python/
  • examples/codes/mosn-extensions/plugin/filter/java/
  • In the so dynamic loading demo, the stream filter implemented in so combines with the multi process framework (go language), and the demo runs successfully.
  • examples/codes/mosn-extensions/plugin/so/subprocess/

Cross language related implementations can refer to the following examples:

Planning and Outlook

Finally, we will introduce the planning of mosn’s follow-up expansion capability. We hope you can give us feedback if you have any demand, and participate in the construction of mosn with interest. First of all, it is necessary to improve so dynamic library loading mechanism, so that mosn can support so loading extension; secondly, script extension for Lua and the ability to support wasm extension; finally, more extension points will be added to mosn to meet more complex scenarios. We are very welcome to participate in the construction of mosn community.

Official website of mosn:

The above is the whole content shared in this issue. If you have any questions and suggestions on mosn, please communicate with us in the group.

Review of live video and PPT viewing address…

Recommended Today

ASP.NET Example of core MVC getting the parameters of the request

preface An HTTP request is a standard IO operation. The request is I, which is the input; the responsive o is the output. Any web development framework is actually doing these two things Accept the request and parse to get the parameters Render according to the parameters and output the response content So let’s learn […]