TCP packet sticking and its solution — Based on NewLife.Net Pipeline frame length gluing method for Network Library

Time:2020-9-27

catalog
  • 1. Sticking
  • 2. Sticking reasons
    • 2.1. The same client sends continuously
    • 2.2. Packet sticking caused by network congestion
    • 2.3. The server is stuck
  • 3. The harm of sticking package
    • 3.1. The packet cannot be parsed correctly
    • 3.2. Error packets are parsed by errors
    • 3.3. Enter the dead loop
  • 4. Logical processing method of gluing package
    • 4.1. Distinguish according to the characteristic parameters of package tail
    • 4.2. Distinguish according to the characteristic parameters of Baotou and Baowei
    • 4.3. Packet sticking is distinguished according to message length
  • 5. Distinguish the code landing of sticking packets according to the message length — Based on NewLife.Net Pipeline treatment
    • 5.1. NewLife.Net Pipeline architecture processing method
    • 5.2. Pipeline analogy with http
    • 5.3. Split the sticky packet decoder (decode according to the length)
      • 5.3.1. Length offset address offset attribute
      • 5.3.2. Length byte size attribute
      • 5.3.3. Encoding method
      • 5.3.4. Decode
        • 5.3.4.1. Decoding step 1: instantiating the length decoder object
        • 5.3.4.2. Decoding step 2 – print the message before decoding
        • 5.3.4.3. Decoding step 3 – decode the message
        • 5.3.4.4. Print the results of sticking package
      • 5.3.5. Empty the gluing encoder
      • 5.3.6. Complete splitting of sticky packet decoder code
  • 6. Length calculation delegate getlength
  • 7. Final gluing and splitting effect drawing

1. Sticking

Each TCP long connection has its own socket cache buffer. The default size is 8K, which can be set manually. Packet sticking is the most common phenomenon in long TCP connections, as shown in the following figure

There are 5 frames (or 5 packets) of heartbeat data in the socket cache. The packet header is f0aa 5500f (hexadecimal). By counting the header data, we confirm that there are 5 frames of heartbeat packets in the cache, but the five frames are connected with each other head and tail, which is a common phenomenon of TCP caching, which we call sticky packets.

2. Sticking reasons

2.1. The same client sends continuously

The same client sends heartbeat data continuously. When the TCP server has no time to parse (if the parsing is finished, the cache will be cleared). Results in the same cache packet glue.

2.2. Packet sticking caused by network congestion

When network congestion occurs at a certain time, after a while, suddenly the network is unblocked. The TCP server receives multiple heartbeat packets from the same client, and the multiple packets will be glued in the cache of the TCP server.

2.3. The server is stuck

When the server is too slow to process the data in the TCP socket cache because of too much computation or other reasons, multiple heartbeat packets (or other messages) will also be connected end to end in the socket cache, blocking packets.

In a word, it is the phenomenon that multiple packets are connected end to end in the same TCP socket cache, that is, packet sticking.

3. The harm of sticking package

Because of the objectivity of packet sticking phenomenon, we must distinguish them artificially in the program logic. If we do not distinguish them and let the packets stick together, we will do the following harm:

3.1. The packet cannot be parsed correctly

The server will continue to identify as invalid packets and tell the client that the client will report again, which will increase the running pressure on the server side of the client. If it has a large amount of computation, some abnormal run-off will occur.

3.2. Error packets are parsed by errors

It is no coincidence that a book is formed. If the wrong package is successfully parsed by the server, it will be handled by the wrong handler. This kind of error handling will do more than 3.1 harm.

3.3. Enter the dead loop

If the frequency is too fast, this phenomenon will occur. The server constantly identifies the sticky packets as invalid packets, and the client constantly reports them to consume the CPU utilization.

To sum up, we must carry out TCP packet sticking, which is the basis of software system robustness and exception handling mechanism.

4. Logical processing method of gluing package

4.1. Distinguish according to the characteristic parameters of package tail

A few bytes are defined as the packet tail characteristics (such as 4 bytes) of each frame of TCP packet. The whole socket cache bytes are retrieved. When the packet tail characteristic bytes are detected, the packets are divided to correctly segment the sticky packets.
Features: need to detect each byte, low efficiency, suitable for short message, if the message is very long, not suitable.

4.2. Distinguish according to the characteristic parameters of Baotou and Baowei

Similar to 4.1, there is more Baotou detection part.
Features: only need to detect each byte of the first frame, the second frame only need to detect the packet header, suitable for long messages

4.3. Packet sticking is distinguished according to message length

According to the message length offset value, read the first frame of message, divide the first frame of correct message from the sticky packet (socket cache), find the message length of the second frame, divide the second frame, and divide it to the end.
For example: the following length offset is 5 (calculated from 0), that is, the 6th and 7th bytes are message length bytes.

Features: only need to detect the length of the message, suitable for the partition of long and short packets.

5. Distinguish the code landing of sticking packets according to the message length — Based on NewLife.Net Pipeline treatment

5.1. NewLife.Net Pipeline architecture processing method

Newlife.Net The design of pipeline architecture refers to Java’s netty open source framework, so most of netty’s codecs can be used here.
The specific performance in the code is

_pemsServer.Add(new StickPackageSplit { Size = 2 });

The codec of lengthcodec is added to the pipeline, and all messages will pass through lengthcodec. Here, the decoding function is mainly used. If the decoding is successful (the sticky packet is divided into several valid packets according to the length), it is pushed to the onReceive method. Size = 2 indicates that the message length is 2 bytes.

5.2. Pipeline analogy with http

Do you find it familiar with the pipeline addition of webapi project of net core?

app.UseAuthentication();
  app.UseRequestLog();
  app.UseCors(_defaultCorsPolicyName);
  app.UseMvc();

The order in which a pipeline is added is the order in which data flows through the pipeline. It just doesn’t pursue whether there is a socket pipeline processing mechanism or an HTTP context pipeline processing mechanism. But the truth is the same.

5.3. Split the sticky packet decoder (decode according to the length)

5.3.1. Length offset address offset attribute

The offset address where the length is located. The default value is 5. See 4.3 for details.

//
        //Abstract:
        //Location of length
        public int Offset
        {
            get;
            set;
        } = 5;

5.3.2. Length byte size attribute

This paper discusses the length byte number is 2, see 4.3 for details

//
        //Abstract:
        //Length takes up the number of bytes, 1 / 2 / 4 bytes, 0 is the compression encoding integer, the default is 2
        public int Size
        {
            get;
            set;
        } = 2;

5.3.3. Encoding method

//
        //Abstract:
        //Encoding, this application does not need encoding, only needs decoding,
        //The sticky packets are divided into multiple packets according to their length
        //
        //Parameters:
        //   context:
        //
        //   msg:
        protected override object Encode(IHandlerContext context, Packet msg)
       { 
           return msg;
       }

There is no need to encode, so MSG is returned directly.

5.3.4. Decode

//
        //Abstract:
        //Decoding
        //
        //Parameters:
        //   context:
        //
        //   pk:
        protected override IList Decode(IHandlerContext context, Packet pk)
        {
            IExtend extend = context.Owner as IExtend;

            LengthCodec packetCodec = extend["Codec"] as LengthCodec;
           
            if (packetCodec == null)
            {
                IExtend extend2 = extend;
                LengthCodec obj = new LengthCodec
                {
                    Expire = Expire,
                    GetLength = ((Packet p) => MessageCodec.GetLength(p, Offset, Size))
                };
                packetCodec = obj;
                extend2["Codec"] = obj;
            }
            
            Console.WriteLine (before message decoding: {0} ", BitConverter.ToString ( pk.ToArray ()));
            IList list = packetCodec.Parse(pk);
            Console.WriteLine ("message decoding");
            foreach (var item in list)
            {
                Console.WriteLine ("sticking result: {0}", BitConverter.ToString ( item.ToArray ()));
            }

            return list;
        }

5.3.4.1. Decoding step 1: instantiating the length decoder object

After the instantiation of the length decoder is completed, it is added to the dictionary.

IExtend extend2 = extend;
    LengthCodec obj = new LengthCodec
    {
        Expire = Expire,
        GetLength = ((Packet p) => MessageCodec.GetLength(p, Offset, Size))
    };
    packetCodec = obj;
    extend2["Codec"] = obj;

5.3.4.2. Decoding step 2 – print the message before decoding

This step is not necessary, in order to let the reader see the effect increase finally.

Console.WriteLine (before message decoding: {0} ", bitconvertetostring( pk.ToArray ()));

5.3.4.3. Decoding step 3 – decode the message

IList list = packetCodec.Parse(pk);

The decoding code is as follows:

//
        //Abstract:
        //Analyze the data stream and get a frame of data
        //
        //Parameters:
        //   pk:
        //Packets to be analyzed
        public virtual IList Parse(Packet pk)
        {
            MemoryStream stream = Stream;
            bool num = stream == null || stream.Position < 0 || stream.Position >= stream.Length;
            List list = new List();


            if (num)
            {

                if (pk == null)
                {
                    return list.ToArray();
                }
                int i;
                int num2;

                for (i = 0; i < pk.Total; i += num2)
                {
                    Packet packet = pk.Slice(i);

                    num2 = GetLength(packet);

                    Console.WriteLine(" pk. GetLength(packet):{0}", num2);

                    if (num2 <= 0 || num2 > packet.Total)
                    {
                        break;
                    }
                    packet.Set(packet.Data, packet.Offset, num2);
                    list.Add(packet);
                }


                if (i == pk.Total)
                {
                  
                    return list.ToArray();
                }
                pk = pk.Slice(i);
            }

            lock (this)
            {
                CheckCache();
                stream = Stream;
                if (pk != null && pk.Total > 0)
                {
                    long position = stream.Position;
                    stream.Position = stream.Length;
                    pk.CopyTo(stream);
                    stream.Position = position;
                }
                while (stream.Position < stream.Length)
                {
                    Packet packet2 = new Packet(stream);
                    int num3 = GetLength(packet2);
                    if (num3 <= 0 || num3 > packet2.Total)
                    {
                        break;
                    }
                    packet2.Set(packet2.Data, packet2.Offset, num3);
                    list.Add(packet2);
                    stream.Seek(num3, SeekOrigin.Current);
                }
                if (stream.Position >= stream.Length)
                {
                    stream.SetLength(0L);
                    stream.Position = 0L;
                }


                return list;
            }
        }

The decoding core code is as follows:
That is to get the length of each frame message, and then cycle all the sticky packets through the delegation method getlength (packet). According to the length of each frame, the packet is partitioned and saved in the list. Finally, the list is returned. Each element of the list triggers a message receive event.

Please pay attention to the next article for the use of delegation. See 6

for (i = 0; i < pk.Total; i += num2)
    {
        Packet packet = pk.Slice(i);

        num2 = GetLength(packet);

        Console.WriteLine(" pk. GetLength(packet):{0}", num2);

        if (num2 <= 0 || num2 > packet.Total)
        {
            break;
        }
        packet.Set(packet.Data, packet.Offset, num2);
        list.Add(packet);
    }

5.3.4.4. Print the results of sticking package

foreach (var item in list)
    {
        Console.WriteLine ("sticking result: {0}" BitConverter.ToString ( item.ToArray ()));
    }

5.3.5. Empty the gluing encoder

The method is developed by NewLife.Net Network library calls, we do not need to care.

//
    //Abstract:
    //Empty the sticky encoder when the connection is closed
    //
    //Parameters:
    //   context:
    //
    //   reason:
    public override bool Close(IHandlerContext contextstring reason)
    {
        IExtend extend = context.Owner as IExtend;
        if (extend != null)
        {
            extend["Codec"] = null;
        }
        return base.Close(context, reason);
    }

5.3.6. Complete splitting of sticky packet decoder code

//Abstract:
    //Length field as header
    // 
    public class StickPackageSplit : MessageCodec
    {
        //
        //Abstract:
        //Location of length
        public int Offset
        {
            get;
            set;
        } = 5;

        //
        //Abstract:
        //Length takes up the number of bytes, 1 / 2 / 4 bytes, 0 is the compression encoding integer, the default is 2
        public int Size
        {
            get;
            set;
        } = 2;


        //
        //Abstract:
        //Expiration time. After the expiration time, it will be processed as scrap data. The default is 500ms
        public int Expire
        {
            get;
            set;
        } = 500;


        //
        //Abstract:
        //Encoding, this application does not need encoding, only needs decoding,
        //The sticky packets are divided into multiple packets according to their length
        //
        //Parameters:
        //   context:
        //
        //   msg:
        protected override object Encode(IHandlerContext context, Packet msg)
       { 
           return msg;
       }

        //
        //Abstract:
        //Decoding
        //
        //Parameters:
        //   context:
        //
        //   pk:
        protected override IList Decode(IHandlerContext context, Packet pk)
        {
            IExtend extend = context.Owner as IExtend;

            LengthCodec packetCodec = extend["Codec"] as LengthCodec;
           

            if (packetCodec == null)
            {
                IExtend extend2 = extend;
                LengthCodec obj = new LengthCodec
                {
                    Expire = Expire,
                    GetLength = ((Packet p) => MessageCodec.GetLength(p, Offset, Size))
                };
                packetCodec = obj;
                extend2["Codec"] = obj;
            }
            
            Console.WriteLine (before message decoding: {0} ", BitConverter.ToString ( pk.ToArray ()));
            IList list = packetCodec.Parse(pk);
            Console.WriteLine ("message decoding");
            foreach (var item in list)
            {
                Console.WriteLine ("sticking result: {0}", BitConverter.ToString ( item.ToArray ()));
            }

            return list;
        }

        //
        //Abstract:
        //Empty the sticky encoder when the connection is closed
        //
        //Parameters:
        //   context:
        //
        //   reason:
        public override bool Close(IHandlerContext context, string reason)
        {
            IExtend extend = context.Owner as IExtend;
            if (extend != null)
            {
                extend["Codec"] = null;
            }
            return base.Close(context, reason);
        }
    }

6. Length calculation delegate getlength

5.3.6 calls the length calculation delegate for each package as follows. The usage of delegation will be explained in the next article, which will not be expanded here.

//
//Abstract:
//Get the whole frame data length from data stream
//
//Parameters:
//   pk:
//
//   offset:
//
//   size:
//
//Return result:
//Data frame length (including head length bits)
protected static int GetLength(Packet pk, int offsetint size)
{
    if (offset < 0)
    {
        return pk.Total - pk.Offset;
    }
    int offset2 = pk.Offset;
    if (offset >= pk.Total)
    {
        return 0;
    }
    int num = 0;
    switch (size)
    {
        case 0:
            {
                MemoryStream stream = pk.GetStream();
                if (offset > 0)
                {
                    stream.Seek(offset, SeekOrigiCurrent);
                }
                num = stream.ReadEncodedInt();
                num += (int)(stream.Position - offset);
                break;
            }
        case 1:
            num = pk[offset];
            break;
        case 2:
            num = pk.ReadBytes(offset, 2).ToUInt16();
            break;
        case 4:
            num = (int)pk.ReadBytes(offset, 4).ToUInt32;
            break;
        case -2:
            num = pk.ReadBytes(offset, 2).ToUInt16(0isLittleEndian: false);
            break;
        case -4:
            num = (int)pk.ReadBytes(offset, 4).ToUInt(0, isLittleEndian: false);
            break;
        default:
            throw new NotSupportedException();
    }
    if (num > pk.Total)
    {
        return 0;
    }          
    return num;
}

7. Final gluing and splitting effect drawing


Copyright notice: This article is the original article of the blogger. It is in accordance with the CC 4.0 by-sa copyright agreement. Please attach the link of the original source and this notice to reprint.

Link to this article: https://www.cnblogs.com/JerryMouseLi/p/12659903.html