From 0 to 1 of Ma cellular im mobile architecture

Time:2019-11-5

(horse comb technology original content, public number ID:mfwtech)

Mobile Internet technology has changed the world of tourism, and the heavy cost of information distribution in this field has been greatly reduced in the past. The communication paths between users and service providers, and between users and users are gradually opened, and the communication scenarios are also expanding. This urges all mobile application developers to better meet the needs of users from the perspective of users.

In the forum era, the form of communication between users is relatively simple, mainly for simple replies, etc. In order to meet the needs of users quickly at a small cost, the non real time message scheme was adopted to realize the message transmission between users.

With the development of the industry and the company, hornet nest has established a unique business model of “content + trading”. Under the background of the continuous growth of user scale and the change of business form, providing stable and reliable pre-sale and after-sale technical support for users and businesses has become the top priority of e-commerce mobile business line.

I. design idea and overall structure

We designed and implemented instant messaging services such as private message, user consultation and user feedback in the mobile terminal of Ma cellular tourism based on different business scenarios such as B2C, C2B and C2C. At the same time, in order to better empower the cooperative businesses, we added functions such as session related consultation user management, customer service management and operation resource statistics in the mobile terminal of Ma cellular businesses.

At present, Im involves the following businesses:

From 0 to 1 of Ma cellular im mobile architecture

In order to realize the integration and reuse of Ma cellular’s tourism app and business im business logic, public resources and personalized UI customization, the problem is divided into the following parts to solve:

  1. IM data channel and abnormal reconnection mechanism to solve the real-time message distribution and stability guarantee of different services;
  2. Im real-time message subscription and distribution mechanism can solve the problem of message directed sending and service subscription consumption, and avoid unnecessary request resource waste;
  3. The general solution of IM session list UI drawing solves the complex problems of rapid iterative development and management of different message types;

The overall implementation structure is divided into four parts for encapsulation, including data management, message registration and distribution management, general UI encapsulation and business management in the figure below.

From 0 to 1 of Ma cellular im mobile architecture

II. Technical principle and implementation process

2.1 general data channel

For the acquisition of normal business presentation data, the client needs to initiate the request actively, the process of request and response is one-way, and the requirement of real-time is not high. However, for IM messages, it needs to support both receiving and sending operations, and requires high real-time performance. In order to support this requirement, a data channel with stable connection should be created between the client and the server to provide two-way data communication between the client and the server.

2.1.1 basic interaction principle of data channel

In order to improve the expansibility of data channel to business support, we encapsulate all communication data into data packets with the same outer structure, so that multi service data can be distributed and processed by using common data channel, so as to reduce the number of channels created and the maintenance cost of data channel.

From 0 to 1 of Ma cellular im mobile architecture

The common data interaction between the client and the server depends on the HTTP request response process. Only when the client initiates the request actively can the response result be obtained. Combined with the specific business scenario of Ma cellular, we hope to establish a reliable message channel to ensure that the server actively informs the client to realize the transmission of business data. At present, it is implemented in the form of HTTP long link polling. Each business data message type only needs to follow the agreed general data structure, which can be sent to the client through the data channel. Data channels do not need to care about the specific content of data, only need to pay attention to receiving and sending.

2.1.2 implementation principle of client data channel

The core of client data channel management is to maintain a business scenario request stack, and stack different business scenario parameter data in different business scenario switching process. Each HTTP long link request uses stack top request data, which can simulate different processing in a specific business scenario (such as with different users’ private letters). The data related processing is encapsulated in the data channel management. The business layer only needs to register the corresponding receiving processing in the data channel management to get the required business message data.

From 0 to 1 of Ma cellular im mobile architecture

2.2 message subscription and distribution

In software system, subscription distribution is essentially a message mode. The party who delivers the message indirectly is called the “publisher” and the party who receives the message processing is called the “Subscriber”. Publishers classify different messages and distribute them to subscribers of corresponding types to complete message delivery. For the convenience of unified management, different interceptors can be added to handle message parsing, message filtering, exception handling and data collection.

2.2.1 message subscription

The business layer only focuses on message processing and does not care about the process of message receiving and distribution. The significance of subscription is to better decouple the business processing and data channel processing. The business layer only needs to subscribe to the message types concerned and passively wait for receiving messages.

From 0 to 1 of Ma cellular im mobile architecture

The business message types that the business layer subscription needs to process will automatically monitor the life cycle of the current page after registration, and delete the corresponding message subscription after the page is destroyed, so as to avoid manually writing the paired subscription and unsubscribe, reduce the coupling of the business layer, and simplify the call logic. Subscription distribution management maintains the distribution operation of subscriber queue for message reception according to various business types.

2.2.2 message distribution

The core of data channel is to maintain the corresponding subscriber set of multiple message types and distribute the parsed messages to the business layer.

From 0 to 1 of Ma cellular im mobile architecture

The data channel is shared by multiple business messages. After each request receives a new message list, it is divided into multiple message lists according to its business type, distributed to the subscription processor corresponding to each business type, and finally delivered to the business layer for processing and display on the corresponding page.

2.3 drawing session message list

Based on different scenarios, such as social oriented private letters, user service oriented consultation feedback, etc., the presentation form of conversation list is required; however, each scenario is not the same, so it is necessary to analyze the commonality and encapsulation of the current conversation list, so as to better support the expansion of subsequent business.

2.3.1 composition structure of message displayed in the list

The IM message list is characterized by multiple message types and diversified UI display. Therefore, it is necessary to establish the corresponding relationship between each type of message and layout. After receiving the message, match it to the corresponding layout and add it to the corresponding message list according to the message type.

From 0 to 1 of Ma cellular im mobile architecture

2.3.2 message type and display layout management principle

For different message types and presentations, the core of the problem is to establishMessage type, message data structure, message display layout managementMapping relationship of. In the process of implementation, the above three parts are maintained by establishing mapping management table, respectively establishing list storage message type / message body encapsulation structure / message display layout management, and setting corresponding relationship Association three lists to complete the search.  

From 0 to 1 of Ma cellular im mobile architecture

2.3.3 UI drawing process of receiving and sending messages once

Each type of message is different in content presentation, but the overall session message presentation style can be divided into three types, namelyReceive, send, and middle of page message styles,The only difference is the internal message style. So the drawing of message UI can be divided into two steps. First, create a general display container, and then fill in the specific display style of each message.

From 0 to 1 of Ma cellular im mobile architecture

The purpose of splitting is to make all kinds of message UI processing only need to pay attention to specific data. For example, general messages such as header, name, message time, whether to report, read and unread status, sending failure / retry status can be processed uniformly, reducing the cost of modification and maintenance, and making the processing logic of each message UI less and clearer, which is more conducive to the extension management of new types.

After receiving and sending the message, judge whether it is “sending and receiving type” or “center display type” according to the message type, find the layout style of the outer layer, find the unique UI style according to the specific message type, splice it in the outer layer layout, get the complete message card, and then set the corresponding data to render to the list to complete the drawing of the whole message.

III. details Optimization & stepping on the pit experience

In the process of implementing the above IM system, we have encountered many problems and made a lot of detail optimization. In this paper, we summarize some points that need to be considered in the implementation for your reference.

3.1 message de duplication

In the previous architecture, we use msg_id to mark every message in the message list. The msg_id is generated after storage according to the data uploaded by the client.

From 0 to 1 of Ma cellular im mobile architectureFrom 0 to 1 of Ma cellular im mobile architecture

After client a requests the IM server, it generates the MSG ﹣ ID, which is then distributed to client a and client B through request return and polling. When the process is established, client a and client B perform local de duplication through the msg_id distributed by the server. However, the following problems exist in this scheme:

From 0 to 1 of Ma cellular im mobile architecture

When client a is unable to accept the return of the corresponding request to send a message due to network problems, the resend mechanism will be triggered. At this time, although the IM server has accepted the message sending request of client a once, it cannot determine whether the two requests come from the same original message, so it can only accept again, which leads to the generation of duplicate messages. The solution is to introduce the client message ID. Because we have done a lot of work depending on the old MSG? ID, we do not intend to let the message ID of the client replace the function of MSG? ID, so we redefine a random? ID.

From 0 to 1 of Ma cellular im mobile architecture

random_id = random + time_stamp。 Random ID identifies the unique message body, which is generated by a random number and the timestamp of the generated message body. When the retry is triggered, the two requests will have the same random ID. the server can de duplicate the message according to this field.

3.2 localized push

When we are in the environment of conversation page or list page, we can directly observe the new message received and update the unread through the change of interface. However, after exiting from the session page or list page, it is impossible to simply obtain these information from the interface. At this time, other mechanisms are needed to let users know the status of the current message.

System push and third-party push is a feasible choice, but in essence, push is also a service based on long link. In order to compensate for the instability and risk of push, we use the form of data channel + local notification to improve the message notification mechanism. If the message sent through the data channel needs to achieve the push prompt effect, the corresponding push display data will be carried. At the same time, the current page will be judged to avoid repeated reminders of the message content of the current page.

From 0 to 1 of Ma cellular im mobile architecture

Through thisData channel + local notificationThe mechanism shown can improve the message arrival rate, reduce the dependence on remote push, reduce the pressure of push system, and improve the user experience during the running time of the application.

3.3 abnormal data channel reconnection mechanism

The current data channel is implemented through HTTP polling. The impact of different business scenarios on polling is shown in the following figure:

From 0 to 1 of Ma cellular im mobile architecture

Due to the different network request status of the user’s mobile phone, sometimes it will encounter the situation of network interruption or abnormal server, so as to terminate the polling request. In order to enable users to continue the session service after network recovery, it is necessary to introduce the reconnect mechanism.

In the 1.0 version of retry mechanism, for the situation that there may be more retry requests, the limit of adding 5 consecutive error reporting and delayed retry in 60 s is adopted. The specific process is as follows:

From 0 to 1 of Ma cellular im mobile architecture

The following problems are found in practice:

  • When the server is suddenly abnormal and lasts for more than 1 minute, the client starts the execution retry mechanism and resends the reconnect request every 1 minute. This is equivalent to a short concentrated “attack” on the server, which may even drag down the server.
  • It is not reasonable to retry immediately after the client is disconnected from the network, because it will take a certain time for the user to recover the network, and the reconnection request during this period is meaningless.

Based on the above analysis and improvement, we designed the second version of retry mechanism. This time, the delay time of the following five requests is modified to 5 – 20 seconds random retry, and the client retry requests are scattered in multiple time points to avoid the instantaneous pressure on the server caused by simultaneous requests. At the same time, delay retry is also performed when the client is disconnected from the network.

From 0 to 1 of Ma cellular im mobile architecture

After the polling mechanism is modified, the request quantity is divided. Compared with the previous requests, the distribution is more uniform, and there is no problem of centralized requests.

From 0 to 1 of Ma cellular im mobile architecture

3.4 unique session ID

3.4.1 why message line ID is introduced

Message lines are used to represent the chat relationship of sessions. Different message lines represent sessions of different objects. From the DB level, a table is needed to store this relationship uid + object_id + busi_type = message line ID.

From 0 to 1 of Ma cellular im mobile architecture

In the initial implementation of IM, we use session configuration parameters (including business source and session parameters) to identify session ID, which has three functions:

  • Find the merchant ID, obtain the consulting source, and assign the Butler
  • Find existing message lines
  • Judge the status of client page, decide whether to send push or not, and send message reminder

There are two problems with this approach:

  • Business source and session parameters are used to resolve the corresponding business ID. if one of the two parameters is missing, it will lead to the wrong business ID resolution. It is necessary to query various databases to get the business ID, which will affect the efficiency;
  • The current session type is identified by the session type switch interface, and the switch page will frequently trigger network requests. If the request interface is unexpected, it is easy to cause message content error, which depends heavily on the robustness of the client

It is inevitable for us to use the business source and session parameters to help us allocate the manager, but we can use the message line ID to bind the message line instead of the business source and session parameters to find the message line. In addition, the problem of sending push has been solved through the local push notification mechanism described above.

3.4.2 when to create a message line

  • When entering the session page to send a message, check whether there is a corresponding message line in the DB. If there is no message line, use the message ID as the message line ID, which means reuse.
  • When entering a session, check whether the corresponding message line already exists in the DB according to the user ID, business type ID, etc. if it does not exist, create the message line and reuse it if it exists.

3.4.3 purpose of message line introduction

  • Reduce the cost of querying message lines on the server side.
  • Remove the interface request related to the old version state change, which indirectly improves the push touch rate.
  • Reduce the complexity of message matching for mobile users.

IV. outlook and recent optimization

4.1 upgrade data channel implementation mode to websocket

Websocket is a full duplex communication protocol over a single TCP connection. Websocket makes the data exchange between the client and the server easier, allowing the server to actively push data to the client. In the websocket API, the browser and server only need to complete one handshake, and they can directly create a persistent connection and carry out two-way data transmission.

Compared with the current HTTP polling implementation mechanism, websocket has the following advantages:

  • Less control overhead。 After the connection is created, when data is exchanged between the server and the client, the packet header used for protocol control is relatively small. In the case of no extension, for the content from server to client, the header size is only 2 to 10 bytes (related to the packet length); for the content from client to server, the header needs to be masked with an additional 4 bytes. Compared with HTTP requests, each time they need to carry a complete header, the overhead is significantly reduced.
  • More real time。 Because the protocol is full duplex, the server can actively send data to the client at any time. Compared with HTTP, which needs to wait for the client to send a request to the server to respond, the delay is significantly less; even compared with comet and other similar long polling, it can also deliver data more than once in a short time.
  • Keep connected。 Unlike HTTP, websocket needs to create a connection first, which makes it a stateful protocol. Some state information can be omitted when communicating later. HTTP requests may need to carry status information (such as identity authentication) in each request.
  • Better binary support。 Websocket defines binary frames, which can handle binary content more easily than http.
  • Support extensions。 Websocket defines the extension. Users can extend the protocol and implement some custom sub protocols, such as some browsers support compression.
  • Better compression。 Compared with HTTP compression, websocket can use the context of the previous content with appropriate extension support, and can significantly improve the compression rate when passing similar data.

In order to further optimize our data channel design, we explored and verified the feasibility of websocket, and conducted research and Design:

From 0 to 1 of Ma cellular im mobile architecture

In the near future, the HTTP polling implementation scheme will be replaced to further optimize the efficiency of the data channel.

4.2 extension of business functions

It is planned to build the im mobile function module into a universal instant messaging component, which can more easily give each business im capability, enable each business to quickly add chat function on its own product line, and reduce the cost and difficulty of developing im. At present, Im function implementation mainly consists of two components: public data channel and UI component.

With the development of beehive business, there are many directions to build and upgrade the existing IM system. For example, for the support of message type, expand the support for short video, voice message, quick message reply, etc., improve the convenience and interest of social interaction; for the multi-person scene, we hope to increase the support for the group, interest channel, multi-person audio and video communication, etc.

I believe that in the future, through the expansion of more business functions and the exploration of application scenarios, Ma cellular mobile IM will better enhance the user experience and continue to empower businesses.

The author of this paper: Ma cellular e-commerce im mobile R & D team.

(original content of Ma cellular technology)

From 0 to 1 of Ma cellular im mobile architecture