Go’s network programming sharing

Go’s network programming sharing

Review the network protocol 5-layer model we shared last time

• physical layer
• data link layer
• network layer
• Transport layer
• application layer

Each layer has independent functions of each layer. Most networks adopt a layered architecture. Each layer is built on its lower layer to provide certain services to its upper layerThe details of how to implement this service mask the upper layer。

What are the protocols behind each layer and why they appear? You can have a look at those who are interestedWhat do you know about Internet protocols

After understanding the layering of network protocol, how to packet, how to unpack, and how to get the source data, I feel a little bit nervous when I look down

What does go network programming mean?

Go network programming, here refers toSocket programming

I believe I didc/c++Friends of network programming are no strangers here. Let’s review it again

Network programming is divided into client-side development and server-side development, which will involve the corresponding key processes

Processes involved in the server

• Socket establish socket
• Bind binding address and port
• Listen sets the maximum number of listeners
• Accept starts blocking connections waiting for clients
• Read read data
• Write write back data
• Close close

Client involved processes

• Socket establish socket
• Connect connect to the server
• Write write data
• Read read data

Let’s see what socket programming is?

SOCKETSocket is the process communication mechanism of BSD UNIX. It is ahandle, used to describeIP addressandportof

of courseSOCKETIt can also be understood asTCP / IP networkofAPI (application program interface)，SOCKETMany functions are defined, and we can use them to develop tCP / IP networkApplications on.

Applications running on a computer usually run throughSOCKETMake a request to or respond to a network request.

Ha, it suddenly occurred to meInterface oriented programming

As the name suggests, in an object-oriented system, various functions of the system are completed by many different objects.

under these circumstances,How each object implements itself is not so important to system designers；

The cooperative relationship between various objects has become the key of system design, the idea of interface oriented programming is that no matter the size of the module, the interaction between the corresponding modules must be considered in the system design.

Ha, we blindly rely on the interface provided by others. We don’t know whether there are pits inside the interface and why it fails

Start socket programming

Let’s take a look at the picture first

SocketApplication layer and tCP / IP protocol familyMiddle of communicationSoftware abstraction layer

In the design pattern, socket is actually a facade pattern, which combines complexTCP / IP protocol familyHidden behind the socket

For users, you only need to call the relevant functions specified in socket, so thatSocketDo the rest

Socket, applications typically useSocketSend to networkrequest / answerNetwork request

Common socket types are2 kinds：

• Streaming socket (stream)

Streaming is aConnection orientedofSocketFor connection oriented TCP service applications

• Datagram socket

Datagram formatSocketIt’s a kind ofConnectionless socketFor connectionless UDP service applications

A simple comparison:

• TCP: relatively reliable, connection oriented, safe and reliable transmission mode, but relatively slow

• UDP: not very reliable and unreliable. Packet loss will not be retransmitted, but it is relatively fast

Take one of the most common examples in life today:

Case 1

Others buy you a small gift and pay on delivery. At this time, when the courier delivers the goods to your home, you must see your person and you have to pay. This is the completion of a process. This is TCP

Case 2

It’s also an example of express delivery. For example, you grab some unimportant gadgets and gadgets on the Internet. When the courier delivers goods, he directly throws them to a delivery point without looking back. This is UDP

Network programming is nothing more than simply looking at itTCP programmingandUDP programming

Let’s take a look at how golang implements TCP based communication and UDP based communication

Go programming based on TCP

Let’s see what TCP protocol is first?

TCP/IP(Transmission Control Protocol/Internet Protocol)

Transmission control protocol / inter network protocol is aConnection oriented(connection guide)reliable、Based on byte streamofTransport layer communication protocol

Because it is a connection oriented protocol, data is transmitted like water flow, which will lead to the problem of sticky packet.

The above mentioned server-side process and client-side process of general socket programming. In fact, the underlying implementation of go is also inseparable from these steps, but let’s look at it from the perspective of applicationGo TCP programmingWhat are the processes on the server

TCP server

TCP server can connect many clients at the same timeThere is no doubt that if a server can only accept a client connection, then you are finished and you can pack your things and go home

Take a chestnut

For all kinds of crazy shopping activities to be started recently, their servers and clients all over the world will connect. How do the TCP servers handle itC/C++In, we will process based on the epoll model. For a client connection / request event, we will open a thread to process it

So how is it handled in golang?

In golang, every time a connection is established, a collaboration process will be openedgoroutineTo handle this request

The processing flow of the server is roughly divided into the following steps

• Listening port
• Receive client request to establish link
• establishgoroutineProcessing links
• close

The simplicity and friendliness of the processing method can be greatly enhanced thanks to theNet package

Specific implementation of TCP server:

func process(conn net.Conn) {
    //Close connection
    defer conn.Close() 
    for {
        reader := bufio.NewReader(conn)
        var buf [256]byte
        //Read data
        n, err := reader.Read(buf[:]) 
        if err != nil {
            fmt.Println("reader.Read  error : ", err)
            break
        }
        recvData := string(buf[:n])
        fmt.Println("receive data ：", recvData)
        //Send the data to the client again
        conn.Write([]byte(recvData)) 
    }
}

func main() {
    //Monitor TCP
    listen, err := net.Listen("tcp", "127.0.0.1:8888")
    if err != nil {
        fmt.Println("net.Listen  error : ", err)
        return
    }
    for {
        //Establish a connection and see the friends here. Do you think the practice here is the same as that of C / C + +
        conn, err := listen.Accept() 
        if err != nil {
            fmt.Println("listen.Accept error : ", err)
            continue
        }
        //Open a goroutine to handle the connection
        go process(conn) 
    }
}

Is it easy to write the TCP server

Let’s look at the TCP client

TCP Client

The client process is as follows:

• Establish connection with the server
• Read and write data
• close

func main() {
    conn, err := net.Dial("tcp", "127.0.0.1:8888")
    if err != nil {
        fmt.Println("net.Dial error : ", err)
        return
    }
    //Close connection
    defer conn.Close() 
    //Type data
    inputReader := bufio.NewReader(os.Stdin)
    for {
        //Read user input
        input, _ := inputReader.ReadString('\n') 
        //Truncation
        inputInfo := strings.Trim(input, "\r\n")
        //Read the user input Q or Q and exit
        if strings.ToUpper(inputInfo) == "Q" { 
            return
        }
        //Send the input data to the server
        _, err = conn.Write([]byte(inputInfo)) 
        if err != nil {
            return
        }
        buf := [512]byte{}
        n, err := conn.Read(buf[:])
        if err != nil {
            fmt.Println("conn.Read error : ", err)
            return
        }
        fmt.Println(string(buf[:n]))
    }
}

matters needing attention：

• For joint debugging between the server and the client, you need to start the server first and wait for the client to connect,

• If the order is reversed, the client will report an error because the server cannot be found

It was mentioned above that TCP is a streaming protocol and there will be packet sticking. Let’s simulate it and see the actual effect

How to solve TCP sticky packets?

To simulate writing a server

server.go

package main

import (
   "bufio"
   "fmt"
   "io"
   "net"
)

//Dedicated to handling client connections
func process(conn net.Conn) {
   defer conn.Close()
   reader := bufio.NewReader(conn)
   var buf [2048]byte
   for {
      n, err := reader.Read(buf[:])
      //If the client is closed, exit this process
      if err == io.EOF {
         break
      }
      if err != nil {
         fmt.Println("reader.Read error :", err)
         break
      }
      recvStr := string(buf[:n])
      //Print the received data. Later, we will mainly see whether the data output here is what we expect
      fmt.Printf("received data：%s\n\n", recvStr)
   }
}

func main() {

   listen, err := net.Listen("tcp", "127.0.0.1:8888")
   if err != nil {
      fmt.Println("net.Listen error : ", err)
      return
   }
   defer listen.Close()
   fmt.Println("server start ...  ")

   for {
      conn, err := listen.Accept()
      if err != nil {
         fmt.Println("listen.Accept error :", err)
         continue
      }
      go process(conn)
   }
}

Write a client to cooperate

client.go

package main

import (
   "fmt"
   "net"
)

func main() {
   conn, err := net.Dial("tcp", "127.0.0.1:8888")
   if err != nil {
      fmt.Println("net.Dial error : ", err)
      return
   }
   defer conn.Close()
   fmt.Println("client start ... ")

   for i := 0; i < 30; i++ {

      msg := `Hello world, hello xiaomotong!`

      conn.Write([]byte(msg))
   }

   fmt.Println("send data over... ")

}

Actual effect

server start ...
received data：Hello world, hello xiaomotong!Hello world, hello xiaomotong!

received data：Hello world, hello xiaomotong!Hello world, hello xiaomotong!Hello world, hello xiaomotong!Hello world, hello xiaomotong!Hello world, hello xiaomotong!Hello worl
d, hello xiaomotong!Hello world, hello xiaomotong!Hello world, hello xiaomotong!Hello world, hello xiaomotong!Hello world, hello xiaomotong!Hello world, hello xiaomotong!Helloworld, hello xiaomotong!

received data：Hello world, hello xiaomotong!Hello world, hello xiaomotong!Hello world, hello xiaomotong!Hello world, hello xiaomotong!Hello world, hello xiaomotong!Hello world, hello xiaomotong!Hello world, hello xiaomotong!Hello world, hello xiaomotong!Hello world, hello xiaomotong!

received data：Hello world, hello xiaomotong!Hello world, hello xiaomotong!Hello world, hello xiaomotong!Hello world, hello xiaomotong!Hello world, hello xiaomotong!Hello worl
d, hello xiaomotong!Hello world, hello xiaomotong!

From the above effects, we can see that the client sends data to the server 30 times, but the server only outputs it 4 times, but multiple pieces of data are output together. This phenomenon is sticky packets. How do we deal with it?

How to handle TCP sticky packets

Sticking reason:

• tcpThe data transfer mode is streaming, which can receive and send multiple times while maintaining a long connection

The actual situation is as follows:

• Sticky packets at the sender caused by Nagle algorithm

Nagle algorithmIt is an algorithm to improve network transmission efficiency

When we submit a piece of data to TCP for sending, TCP will not send this piece of data immediately

But wait for a short time to see. During this waiting time,Is there any data to be sent? If yes, these two pieces of data will be sent at one time

• Packet sticking at the receiving end caused by untimely receiving at the receiving end

TCP will store the received data in its own memoryIn bufferNotify the application layer to fetch data

When the application layer can’t take out the TCP data in time for some reasons, several segments of data will be stored in the TCP buffer.

After knowing the reason, let’s see how to solve it

Start solving TCP sticky packet problem

When we know the reason for the sticky packet, we’ll start with the reason. Analyze why TCP waits for a period of time. Is it because TCP doesn’t know how big the packet we want to send him, so he wants to eat as much as possible?

So, our solution isPacket and unpack data packets.

• Packet:

A packet is to add a header to a piece of data, so that a packet is divided intoBaotou and inclusionThere are two parts. Sometimes, in order to filter illegal packages, we will add package tail.

The length of the package head is fixed. He will clearly indicate the size of the package, so that we can correctly dismantle a complete package

• according toFixed Baotou length
• according toThe header contains the variable of inclusion length

We can define a protocol ourselves. For example, the first two bytes of a packet are the packet header, which stores the length of the transmitted data.

This is a custom protocol that both the client and server should know, otherwise they won’t have to play

Start solving the problem

server2.go

package main

import (
   "bufio"
   "bytes"
   "encoding/binary"
   "fmt"
   "io"
   "net"
)

//Decode decode message
func Decode(reader *bufio.Reader) (string, error) {
   //Read the length of the message
   lengthByte, _ :=  reader. Peek (2) // read the first two bytes and look at the packet header
   lengthBuff := bytes.NewBuffer(lengthByte)
   var length int16
   //Read actual envelope length
   err := binary.Read(lengthBuff, binary.LittleEndian, &length)
   if err != nil {
      return "", err
   }
   //Buffered returns the number of bytes that are currently readable in the buffer.
   if int16(reader.Buffered()) < length+2 {
      return "", err
   }

   //Read real message data
   realData := make([]byte, int(2+length))
   _, err = reader.Read(realData)
   if err != nil {
      return "", err
   }
   return string(realData[2:]), nil
}

func process(conn net.Conn) {
   defer conn.Close()
   reader := bufio.NewReader(conn)

   for {
      msg, err := Decode(reader)
      if err == io.EOF {
         return
      }
      if err != nil {
         fmt.Println("Decode error : ", err)
         return
      }
      fmt.Println("received data ：", msg)
   }
}

func main() {

   listen, err := net.Listen("tcp", "127.0.0.1:8888")
   if err != nil {
      fmt.Println("net.Listen error :", err)
      return
   }
   defer listen.Close()
   for {
      conn, err := listen.Accept()
      if err != nil {
         fmt.Println("listen.Accept error :", err)
         continue
      }
      go process(conn)
   }
}

client2.go

package main

import (
   "bytes"
   "encoding/binary"
   "fmt"
   "net"
)

//Encode message
func Encode(message string) ([]byte, error) {
   //Read the length of the message and convert it to int16 type (accounting for 2 bytes). The agreed packet header is 2 bytes
   var length = int16(len(message))
   var nb = new(bytes.Buffer)

   //Write header
   err := binary.Write(nb, binary.LittleEndian, length)
   if err != nil {
      return nil, err
   }

   //Write message body
   err = binary.Write(nb, binary.LittleEndian, []byte(message))
   if err != nil {
      return nil, err
   }
   return nb.Bytes(), nil
}

func main() {
   conn, err := net.Dial("tcp", "127.0.0.1:8888")
   if err != nil {
      fmt.Println("net.Dial error : ", err)
      return
   }
   defer conn.Close()
   for i := 0; i < 30; i++ {
      msg := `Hello world,hello xiaomotong!`

      data, err := Encode(msg)
      if err != nil {
         fmt.Println("Encode msg error : ", err)
         return
      }
      conn.Write(data)
   }
}

For the convenience and simplicity of demonstration, we put the packet into the client code, unpack it and put it into the server code

Effect demonstration

Now, there will be no problem of sticking, becausetcpHe knows how many packets he reads each time. If the buffer data is not as long as expected, he will read them together when the data is enough, and then print them out

Friends here are still a little interested in golang’s TCP programming, so we can take a look at UDP programming. Compared with TCP, it is much simpler and there will be no problem of sticky packets

Go programming based on UDP

Similarly, let’s talk about UDP protocol first

UDP protocol (User Datagram Protocol)

Is a user datagram protocol, a connectionless transport layer protocol

Data can be sent and received directly without establishing a connection

It belongs to unreliable and non sequential communication. It is precisely because of this feature thatUDP protocolIt has good real-time performance and is usually used in the field related to live video broadcasting, because for video transmission, some frames are lost in the transmission process, which has little impact on the whole

UDP server

Let’s create a UDP client and server

server3.go

func main() {
    listen, err := net.ListenUDP("udp", &net.UDPAddr{
        IP:   net.IPv4(0, 0, 0, 0),
        Port: 8888,
    })
    if err != nil {
        fmt.Println("net.ListenUDP error : ", err)
        return
    }
    defer listen.Close()
    for {
        var data [1024]byte
        //Receive data message
        n, addr, err := listen.ReadFromUDP(data[:]) 
        if err != nil {
            fmt.Println("listen.ReadFromUDP error : ", err)
            continue
        }
        fmt.Printf("data == %v  , addr == %v , count == %v\n", string(data[:n]), addr, n)
        //Send the data to the client again
        _, err = listen.WriteToUDP(data[:n], addr) 
        if err != nil {
            fmt.Println("listen.WriteToUDP error:", err)
            continue
        }
    }
}

UDP client

client3.go

func main() {
   socket, err := net.DialUDP("udp", nil, &net.UDPAddr{
      IP:   net.IPv4(0, 0, 0, 0),
      Port: 8888,
   })
   if err != nil {
      fmt.Println("net.DialUDP error : ", err)
      return
   }
   defer socket.Close()
   sendData := []byte("hello xiaomotong!!")
   //Send data
   _, err = socket.Write(sendData)
   if err != nil {
      fmt.Println("socket.Write error : ", err)
      return
   }
   data := make([]byte, 2048)
   //Receive data
   n, remoteAddr, err := socket.ReadFromUDP(data)
   if err != nil {
      fmt.Println("socket.ReadFromUDP error : ", err)
      return
   }
   fmt.Printf("data == %v  , addr == %v , count == %v\n", string(data[:n]), remoteAddr, n)
}

Effect display

Server printing:
data == hello xiaomotong!!  , addr == 127.0.0.1:50487 , count == 18

Client printing:
data == hello xiaomotong!!  , addr == 127.0.0.1:8888 , count == 18

summary

• Review the 5-layer model of the network, the server and client processes of socket programming
• How to program go based on TCP and how to solve the problem of TCP packet sticking
• How to program go based on UDP

Welcome to like, follow and collect

My friends, your support and encouragement are the driving force for me to insist on sharing and improve quality

Well, that’s all for this time,How to set HTTPS in the next share go，

Technology is open, and our mentality should be open. Embrace change, live in the sun and strive to move forward.

I amLittle Devil boy Nezha, welcome to like and pay attention to the collection. See you next time~

This work adoptsCC agreement, reprint must indicate the author and the link to this article