Tantan’s IM long connection technology practice: technology selection, architecture design, performance optimization

Time:2022-11-25

This article was shared by Zhang Kaihong, a senior technical expert on the Tantan server side. The original title was “Go Language Practice of Tantanchang Link Project”. Due to many errors in the original text, it has been revised and changed.

1 Introduction

The instant messaging long-connection service is at the network access layer, and this field is very suitable for using the Go language to take advantage of its multi-coroutine parallel and asynchronous IO features.

Since the launch of the long connection project, Tantan has optimized the service several times: GC has been reduced from 5ms to 100 microseconds (Go versions are all above 1.9), and the main gRPC interface call delay p999 has been reduced from 300ms to 5ms. When most of the industry focuses on the number of single-machine connections, we focus more on the SLA (service availability) of the service.

This article will share the entire technical practice process and experience summary of the IM long-connection module of the stranger social application Tantan from technology selection to architecture design to performance optimization.

Tantan's IM long connection technology practice: technology selection, architecture design, performance optimization

study Exchange:

  • Instant messaging/push technology development exchange group 5: 215477170 [recommended]
  • Introductory article on mobile IM development: “One entry is enough for beginners: developing mobile IM from scratch”
  • Open source IM framework source code:https://github.com/JackJiang2…

(This article has been synchronously published at:http://www.52im.net/thread-37…

2. About the author

Tantan's IM long connection technology practice: technology selection, architecture design, performance optimization

Zhang Kaihong: Served as a senior technical expert on Tantan server.

6 years of experience in Go language development, used Go language to build several large-scale Web projects, involving network libraries, storage services, long-term connection services, etc. Focus on Go language practice, storage service research and development, and deep optimization of Go language in big data scenarios.

3. The origin of the project

Our project started in the second half of 2018, according to today, it will take about one and a half years.

At that time, Tantan encountered some technical pain points, and the most serious one was the heavy reliance on third-party Push. For example, if the third party had some faults, it would have a relatively large impact on the KPS of real-time IM chat.

At that time, push messages were pushed, and the push delay in the app was relatively high, with an average delay of five or six hundred milliseconds, which we could not accept.

And there is no Ping Plan mechanism (heartbeat check mechanism?), It is impossible to know whether the user is online.

At that time, product and technology students felt that it was an opportunity to make a long connection.

4. An episode

The project lasted for about a quarter. First of all, the IM business was implemented. We think that the long-term connection is more closely bound to IM.

After the implementation of IM and the launch of the follow-up long connection, various businesses are more dependent on the long connection service.

There was a small episode in the middle, mainly because of the naming.

At the beginning of the project, I named the project Socket. Seeing that socket is more friendly, I think it is a long connection. This feeling is inexplicable, and I don’t know why. But the operation and maintenance raised objections, thinking that UDP is also a Socket, and I think UDP can actually be used for long connections.

The operation and maintenance proposal is called Keepcom, which is implemented by Keep Alive. This proposal is quite good, and we finally used this name.

The suggestion given by the client is Longlink, the other is Longconn, one is taken by a technical colleague on the IOS side, and the other is taken by a technical colleague on the Android side.

In the end, we all lost, and the operation and maintenance students won. The operation and maintenance students felt that if the name could not be settled, they should not go online. In the end, we compromised.
Tantan's IM long connection technology practice: technology selection, architecture design, performance optimization

5. Why do you need a long connection?

Why do long connections?

Tantan's IM long connection technology practice: technology selection, architecture design, performance optimization

As shown in the figure above: It is quite obvious to look at the comparison. The left side is a long connection, and the right side is a short and long connection.

For long connections, there is no need to re-enter the connection, or release the connection, and an X packet only needs one RTT to finish. On the right side, for a short connection, a three-way handshake is required to send a push packet, and finally wave.

Conclusion: If a data packet of N messages is sent, the long connection is 2+N times RTT, and the short connection is 3N times RTT, and finally Keep Alive is enabled, and N is the number of connections.

6. Advantages of long connection technology

We decided that long connections have the following four advantages:

1) Real-time: long connection is a two-way channel, and the push of messages is also relatively real-time;
2) Stateful: the long connection itself maintains the user’s state, and determines whether the user is online through the KeepAlive method;
3) Saving process: long connection saves traffic, can do some user-defined data compression, and can save a lot of attribution packets and connection packets, so it saves traffic;
4) More power saving: After reducing network traffic, it can further reduce the power consumption of mobile clients.

7. Can TCP be competent on the mobile side?

Before the project started, we did a lot of consideration.

First of all, let’s see if the TCP protocol can work for long connections on the mobile side?

For the traditional long connection, the long connection TCP on the web side is competent, but is TCP competent on the mobile side? This depends on several properties of TCP.

First of all, TCP has the characteristics of slow start and sliding window. TCP controls PU packets in this way to avoid network congestion.

After the TCP connection, go through a slow start process. This process expands the initial window size by 2 N powers, and finally reaches a certain domain value. For example, the domain value is 16 packets, and gradually increases from 16 packets to 24. packets, so as to reach the maximum value of the window.

Once you encounter packet loss, of course there are two situations. One is fast retransmission, and the window is simple, which is equivalent to a window of 12 packets. If starting an RTO similar to a stateful connection, window one drops to the initial window size.

If RTO retransmission is enabled, the blocking of subsequent packets is very serious, and one packet blocks the sending of other packets.

Tantan's IM long connection technology practice: technology selection, architecture design, performance optimization
(▲ The above picture is quoted from “Towards the High Level: The Network Basics that Excellent Android Programmers Must Know”)

For basic knowledge of the TCP protocol, you can read the following information:

“TCP/IP Detailed Explanation – Chapter 17 TCP: Transmission Control Protocol”
“TCP/IP Detailed Explanation – Chapter 18 Establishment and Termination of TCP Connections”
“TCP/IP Detailed Explanation – Chapter 21 Timeout and Retransmission of TCP”
“Easy to understand – in-depth understanding of the TCP protocol (on): Theoretical basis”
“Easy to understand – in-depth understanding of TCP protocol (Part 2): RTT, sliding window, congestion handling”
“Introduction to Lazy Network Programming (1): Quickly Understand Network Communication Protocols (Part 1)”
“Introduction to Lazy Network Programming (2): Quickly Understand Network Communication Protocols (Part 2)”
“Introduction to Lazy Network Programming (3): A quick understanding of the TCP protocol is enough”
“Introduction to brain-dead network programming (1): follow the animation to learn TCP three-way handshake and four-way wave”
“Getting started with network programming has never been so easy (2): If you were to design the TCP protocol, what would you do? “

8. TCP or UDP?

Tantan's IM long connection technology practice: technology selection, architecture design, performance optimization

(▲ The picture above is quoted from “Mobile IM/Push System Protocol Selection: UDP or TCP?”)

Four problems of TCP implementing long connection:

1) The amount of messages on the mobile terminal is still relatively sparse. After the user gets the mobile phone every time, the total number of messages sent is relatively small, and the interval between each message is relatively long. In this case, the advantages of TCP inter-connection and long-term connection are more obvious;
2) The packet loss rate is relatively high under weak network conditions, and the block’s subsequent data transmission is easy to block after packet loss;
3) The TCP connection timeout time is too long, and the default is 1 second. This is due to the fact that TCP was born earlier, and the network status at that time was not as good as it is now. At that time, the timeout time must be 1s, and now it can be set shorter;
4) In the case of no fast retransmission, the RTO retransmission wait time is longer, the default is 15 minutes, and each time is decremented by the Nth power.

Why did you choose TCP in the end? Because we think UDP is a little more serious.

First of all, UDP has no sliding window, no flow control, and no slow start process. It is easy to cause packet loss, and it is also easy to cause packet loss and timeout in the middle of the network state.

Once UDP packets are lost, there is no retransmission mechanism, so we need to implement a retransmission mechanism at the application layer. The amount of development is not that large, but I think it is relatively low-level and prone to failure, so I finally chose TCP.

TCP or UDP? This has been a rather controversial topic:

“Introduction to Lazy Network Programming (4): Quickly Understand the Differences Between TCP and UDP”
“Introduction to Lazy Network Programming (5): Quickly understand why UDP is sometimes more advantageous than TCP”
“The 5G era has arrived, TCP/IP is old, can it still be used?” “
“Android programmers must know the network communication transport layer protocol – UDP and TCP”
“Unknown Network Programming (6): In-depth understanding of UDP protocol and making good use of it”
“Unknown Network Programming (7): How to make unreliable UDP reliable? “

If you don’t know the UDP protocol yet, you can read this article: “TCP/IP Detailed Explanation – Chapter 11 UDP: User Datagram Protocol”.

9. More reasons to choose TCP

Let’s list it, there are three main points:

1) At present, in terms of mobile terminals, Android, and IOS, the initial window size is relatively large and the default is 10, considering the disadvantages of TCP slow start;
2) In the case of ordinary text transmission, it is not very sensitive to the seriousness of packet loss (not to say that multimedia data streams are transmitted, but some text data, which is not particularly serious for side effects of packet loss in TCP);
3) We feel that TCP is used more in the application layer.

Regarding point “3)”, here are the following three considerations.

First consideration:

Basically, the current application program uses the HTP protocol or the push method is basically TCP, and we think that TCP generally does not cause major problems.

Once TCP is abandoned and UDP or QUIC protocol is used, there will be a relatively big problem if the guarantee is not uniform, which cannot be solved in a short time, so TCP is finally used.

The second consideration point:

Which way our service uses to do LB on the base layer, there were two options at that time, one was the traditional LVS, and the other was HttpDNS (for more information on HttpDNS, please refer to “Comprehensive Understanding of Mobile DNS Domain Name Hijacking and Other Miscellaneous Diseases: Principles , root, HttpDNS solution, etc.”).

Finally, we chose HttpDNS. First of all, we still need LB support across computer rooms, which HttpDNS completely wins. Secondly, if cross-network terminals are required, LVS cannot do it, and other deployment methods are required. Furthermore, in terms of capacity expansion, LVS is slightly better. Finally, for the general LB algorithm, LVS support is not good. It needs the LB algorithm based on the user ID, the LB algorithm based on the consistent hash, and the positioning information based on the geographical location. HttpDNS can perfectly win in these aspects , but LVS can’t do it.

The third consideration point:

What kind of method do we use when doing the saturation mechanism of TCP? How to determine the method of Ping packets, how to determine the interval time, and how to determine the time details of Ping packets?

At that time, I was more entangled whether the client actively sent the ping or the server actively sent the ping?

It is better to support the client keep-alive mechanism, because the client may be woken up, but the client may not be able to send packets after entering the background.

Secondly: the front and back of the APP keep alive at different intervals of ping packets, because the background itself is in a weak online state, and there is no need to send ping packets frequently to determine the online status.

So: the time interval of Ping packets in the background can be longer, and the front-end can be shorter.

Furthermore: Ping exponential growth interval support is required, which is more life-saving in the event of a failure.

For example: Once the server fails, if the client pings desperately, the server may be completely paralyzed. If there is an exponentially increasing ping packet interval, the basic server can still slow down, which is more important in the event of a failure.

Finally: Whether Backoff is required for Ping packet retry, Ping packet resends Ping, if the Bang packet is not received, it needs to wait until Backoff sends Ping.

10. Dynamic Ping packet time interval algorithm

PS: In IM, there is actually a more professional name for this – “Smart Heartbeat Algorithm”.

We also designed a dynamic Ping packet time interval algorithm.

Because domestic network operators have a keep-alive mechanism for NIT equipment, which is currently more than 5 minutes. If you do not send a package within 5 minutes, your cache will be deleted. Basically, all operators are more than 5 minutes, but mobile 4G is hindered. Basically, a Ping packet can be sent within 4 to 10 minutes, and the cache in the network operator’s equipment can be maintained, so that there is no problem, and the long connection is kept alive.

Increasing the interval between Ping packets can reduce network traffic and further reduce the power consumption of the client. This benefit is relatively large.

In the case of low-end Android devices, there are some issues with DHCP leases. This problem focuses on the low version of the Android side, and Android will not renew the expired IP.

Solving the problem is also relatively simple. When the DHCP lease is halfway through, just renew the lease with the DHCP server in time.

Due to space limitations, I will not expand here. If you are interested, you can read these materials:

“Why does the mobile IM based on the TCP protocol still need a heartbeat keep-alive mechanism?” “
“One article to understand the network heartbeat packet mechanism in instant messaging applications: function, principle, implementation ideas, etc.”
“WeChat Team Original Sharing: Android Version of WeChat Backstage Keep Alive Sharing (Network Keep Alive)”
“Mobile IM Practice: Realizing the Smart Heartbeat Mechanism of Android WeChat”
“Mobile IM Practice: Heartbeat Strategy Analysis of WhatsApp, Line, WeChat”
“Discussion on the Design and Implementation of an Android IM Smart Heartbeat Algorithm (Including Sample Code)”
“Teach you to use Netty to implement the heartbeat mechanism and disconnection reconnection mechanism of network communication programs”

11. Service Architecture

11.1 Basic introduction
The service architecture is relatively simple, about four modules:

1) First is HttpDNS;
2) The other is the Connector access layer, which provides IP,
3) Then there is the Router, which is similar to the agent forwarding the message, selects the server of the access layer according to the IP, and finally pushes it to the user;
4) Finally, there is the authentication module Account. We are currently only exploring the APP, which is implemented in the user center.

11.2 Deployment
Deployment is equivalent to three modules:

1) One is Dispatcher;
2) One is Redis;
3) One is Cluser.

As shown in the figure below: When the client is connecting:

1) Need to get an agreement;
2) The second step is to get the ConnectorIP through HttpDNS;
3) Connect through the IP company commander, and send the Auth message for authentication in the next step;
4) The connection is successful, and the Ping packet will be sent later to keep alive;
5) Disconnect afterwards.

11.3 Message forwarding process
The process of message forwarding is divided into two parts.

The first is message uplink: the server initiates a message packet, accesses the service through the Connector, the client sends the message through the Connector, and then sends the message to the microservice through the Connector. If the microservice is not needed, it can be directly forwarded to the Vetor. In this case Connector is more like a Gateway.

For the downlink: all business parties need to request the Router, find a specific Connector, and deploy messages according to the Connector.

Tantan's IM long connection technology practice: technology selection, architecture design, performance optimization

Every company has a microservice architecture, and the interaction between long connections and microservices is basically two. One is when the message goes up, it is more like a Gateway, the downlink is accessed through the Router, and the message is sent through the Connector.

11.4 Some implementation details
The following are some details. We use the GO language 1.13.4. The internal message transmission is gRPC, and the transmission protocol is Http2. We use ETCD as a LB internally to provide service registration and discovery services.

As shown in the figure below: Connector is the status, which is a status information from the user ID to the connection.

Let’s look at the right side of the figure below: it actually has a relatively large MAP. In order to prevent the lock competition of the MAP from being too serious, the MAP is divided into 2 to 56 sub-MAPs. In this way, a high-reading and writing MAP is realized. For the mapping relationship of each MAP from an ID to a connection state, each connection is a Go Ping, and the implementation details are 4KB for reading and writing, which has not been changed.

Let’s take a look at Router: it is a stateless CommonGRPC service, and it is relatively easy to expand. Now the state information is stored in Redis, and Redis is about one layer per group. The current peak value is 3000.

We have two states: one is Connector and one is Router.

First of all, the state of the Connector is the main one, and the Router is the guarantee of the state consistency.

There are two cases: if the connection is on the same Connector, the Connector needs to ensure that the order of copying to the Router is correct. If the order is inconsistent, the state of the Router and Connector will be inconsistent. Achieve message consistency by unifying the Connector window. If across Connectors, implement the Compare And Update method in the Redis Lua script to ensure that only the status written by your own Connector can be updated by yourself. If it is another Connector, others cannot be updated. Confidence. We ensure that both across Connectors and the same Connector can update the connection status in the Router in a consistent manner in sequence.

Dispatche is relatively simple: it is a pure Common Http API service, it provides Http API, the current delay is relatively low about 20 microseconds, and 4 CPUs can support 100,000 concurrency.

At present, a high availability is achieved through a structure without a single point: first, Http DNS and Router, these two are barrier-free services, which only need to be guaranteed by LB. For Connectors, Ordfrev at the connection layer is implemented through Http DNS client active drift. This way, once a Connector fails, the client can immediately drift to the next Connector to realize automatic work transfer. Currently, it is There is no single order.

12. Performance optimization

12.1 Basic information
Subsequent optimization mainly includes the following aspects:

1) Network optimization: This part is done together with the client. First, the client sends three sniffing packets when the client needs to retransmit the packet. In this way, a fast retransmission mechanism is made, and the fast retransmission is improved through this mechanism. proportion;
2) Heartbeat optimization: reduce the number of Ping packets through dynamic Ping packet interval time, this is still under development;
3) Prevent hijacking: it is an operation to avoid domain name hijacking through the client using IP direct connection;
4) DNS optimization: HttpDNS is used to request the client’s HttpDNS by returning multiple IPs each time.
12.2 Network Optimization
For the access layer, in fact, the number of connections of the Connector is relatively large, and the load of the Connector is also relatively high.

We have made a relatively large optimization of the Connector. First, we can see that the earliest GC time of the Connector has reached 4 or 5 milliseconds, which is terrible.

Let’s take a look at the picture below (on the picture) is the optimized result, about 100 microseconds on average, which is relatively good. The second picture (below the picture) is the result of the second optimization, which is about 29 microseconds, and the third picture is about 20 microseconds.

12.3 Message Delay
Looking at the message delay, Tantan has relatively high requirements for the delay of IM messages, and pays special attention to the user experience.

This part is about 200ms at the beginning. For an operation, 200ms is quite serious.

After the first optimization (picture below – top), the state is about 1:00 milliseconds. After the second optimization (picture below – bottom), it has now dropped to the lowest point of about 100 microseconds, which is relatively close to the general Net operation time dimension .

12.4 Connector optimization process
The optimization process is as follows:

1) First of all, the Info log on the critical path is needed, and the Access Log is realized through sampling. The Info log is a relatively heavy operation at the access layer;
2) Second, cache objects through Sync.Poll;
3) The third is to distribute the Escape Analysis object online as much as possible.
Later, the non-destructive release of Connector was also realized: this one is more valuable. There are many releases of the long connection just after it is launched, and each release has a sense for users.

The Graceful Shutdown method of the Connector is implemented, and the connection is optimized in this way.

First: log off the machine from HttpDNS, and then slowly disconnect the user until the number of connections is less than a certain threshold. The latter is to restart the service and release the binary version.

Finally: HttpDNS is launched on the machine. It takes a long time to implement user release in this way. At that time, it took a long time to measure how many connections are disconnected per second, and what is the final threshold.

The following is some data: the GC is also part of it just now, and the current number of connections is relatively critical data. First of all, look at the number of connections. The number of single-machine connections is relatively small. Don’t dare to let go too much. The maximum number of single-machine connections is 150,000, about 100 microseconds.

The number of Goroutines is the same as the number of connections, about 150,000:

Look at the memory usage status. The figure below (above) shows the total memory of GO, which is about 2:3. The remaining one-fifth is unoccupied, and the total memory is 7.3 G.

The figure below shows the GC status. The GC is relatively healthy. The red line is the number of GC active memory each time. The red line is much higher than the green line.

Seeing that the current status of GC is about 20 microseconds, I feel that it is more consistent with the official time of GO. We feel that GC has been optimized.

12.5 Subsequent optimizations to be done
The last step is to plan for follow-up optimization.

First of all: For the system, it is still necessary to optimize the Connector layer more, reduce memory allocation more, and try to allocate memory to the heap instead of the station. In this way, the GC pressure is reduced. We see that GO is not a Generational Collection GE. The more memory in the heap, the more memory will be scanned, so it is not a linear growth.

Second: Use Sync Pool more internally for short-term memory allocation, such as Context or temporary Dbyle.

The protocol also needs to be optimized: the WebSocket protocol is currently used, and some function flags will be added later to transmit some important information to the server. For example, some retransmission flags, if the client adds the retransmission flag, we can first check whether the packet is a retransmission packet. If it is a retransmission packet, we will judge whether the packet is a duplicate or has been sent before. If it has been sent, there is no need to unpack it, which can save a lot of server-side operations.

In addition: You can remove the current Mask mechanism of Websocket, because the Mask mechanism prevents the packet modification operation on the Web side, but it is basically the client’s packet transmission, so the Mask mechanism is not needed.

In terms of business: At present, more things need to be done after planning. We think that because the persistent connection is an access layer, it is a very good place to count the distribution of some clients. For example, the distribution of Android and IOS clients.

Further: Statistics of user portraits can be made, such as male and female, age, and geographical location. Probably these, thank you!

Tantan's IM long connection technology practice: technology selection, architecture design, performance optimization

13. Reply to popular questions

  • Question: I just said that the connection layer dialogue is restarted, and those disconnected users will float to other users in the indirect process. Is this done?

Zhang Kaihong: The current situation is that the client does automatic drifting.

  • Question: Now there are 10 million DAU, if the server pushes 1 million to the client, how to do this scenario?

Zhang Kaihong: At present, we do not have such a large amount of news pushes. Sometimes we will send some business-related pushes. Currently, we have implemented a current limit, which is realized through the client-side current limit, about three to four thousand.

  • Question: If the backend is implemented, it means there will be security risks, and attackers will continue to establish connections, making it difficult to defend. Is there such a problem? Because of malicious attacks, it is enough to establish a connection if there is an attack, and no authentication mechanism is required.

Zhang Kaihong: I understand what you mean. This is not only a long connection, but also a short connection. The client has been falsifying the access results, and the traffic is still relatively large, which is realized by the firewall and the IP layer firewall.

  • Question: The persistent connection server is hung on the outermost side. Is there a layer in the middle?

Zhang Kaihong: At present, the following layer is directly exposed to the external network layer, and a layer of IP anti-DNSFre firewall is passed in front of it. Other than that there are no other network devices.

  • Question: Based on what considerations, there is no layer added in the middle, because there is another layer added in front.

Zhang Kaihong: There is no such plan at present, and an LS layer will be added in front of the Websofte access layer to facilitate expansion. This benefit is not particularly large, so there is no plan now.

  • Question: What is the meaning of the three sniffs of disconnection and retransmission just mentioned?

Zhang Kaihong: We want to trigger fast retransmission more, so that the retransmission interval for TCP is shorter. The server judges whether fast retransmission is based on three cyclic packets. We will send three cyclic packets to avoid an RTO retransmission open.

  • Question: Did Tantan use a third-party Android server at the beginning?

Zhang Kaihong: Yes, it was Jiguang Push at first.

  • Question: From third-party Android servers to self-developed.

Zhang Kaihong: If Jiguang has some malfunctions, it will still have a great impact on us. In the past, Jiguang had a high frequency of failures, so we wondered if we could improve the service ourselves. The second point is that Jiguang itself can provide a judgment on whether the user is online, but its judgment needs to go through the channel, and the delay is relatively high. Its own judgment is that the connection reduces the delay.

  • Question: For example, when a new user connects online, and some users send him messages, how does he get the first-line messages?

Zhang Kaihong: We guarantee through the business side that an ID number will be saved for unsent messages. When the user reconnects, the business side will pull it again.

14. Reference materials

[1] Protocol selection of mobile IM/push system: UDP or TCP?
[2] The 5G era has arrived, and TCP/IP is old, can it still be used?
[3] Why does the mobile IM based on the TCP protocol still need a heartbeat keep-alive mechanism?
[4] An article to understand the network heartbeat packet mechanism in instant messaging applications: function, principle, implementation ideas, etc.
[5] The original sharing of the WeChat team: Android version of WeChat background keep-alive combat sharing (network keep alive)
[6] Mobile IM Practice: Realize the Smart Heartbeat Mechanism of Android WeChat
[7] Towards a high level: the network foundation that excellent Android programmers must know
[8] A comprehensive understanding of miscellaneous diseases such as mobile DNS domain name hijacking: principle, root cause, HttpDNS solution, etc.
[9] Technical literacy: a new generation of UDP-based low-latency network transport layer protocol – QUIC detailed explanation
[10] One entry is enough for beginners: developing mobile IM from scratch
[11] Special Topic on Long Connection Gateway Technology (2): Knowing the Practice of High-Performance Long Connection Gateway Technology with Tens of Millions of Concurrency
[12] Special Topic on Long Connection Gateway Technology (3): The Road to Technology Evolution of Mobile Taobao’s Billion-Level Mobile Terminal Access Layer Gateway
[13] Long connection gateway technology topic (5): Himalaya self-developed billion-level API gateway technology practice
[14] A set of dry goods of IM architecture technology for billion-level users (Part 1): overall architecture, service splitting, etc.
[15] A set of dry goods of IM architecture technology for billion-level users (Part 2): reliability, orderliness, weak network optimization, etc.
[16] From Novice to Expert: How to Design a Distributed IM System with Hundreds of Millions of Messages

This article has been simultaneously published on the “Instant Messaging Technology Circle” public account.
The synchronous release link is:http://www.52im.net/thread-37…