Detailed explanation of producers and consumers of golang concurrent programming

Time:2021-8-10

The most attractive thing about golang may be concurrency. Golang has absolute advantages in both code writing and performance

Learn the concurrency characteristics of a language. I like to implement a producer consumer model. This model is very classic and suitable for many concurrency scenarios. Next, I will briefly introduce the concurrency programming of golang through this model

Go concurrency syntax

Synergetic go

Coprocessing is the smallest unit of golang concurrency, which is similar to threads in other languages. However, threads are implemented with the help of the operating system. Each thread scheduling is a system call and needs to switch from user state to kernel state. This is a very time-consuming operation. Because there are too many threads in general programs, a lot of performance will be consumed in thread switching. This kind of scheduling is implemented inside golang. The switching of coroutines under this scheduling is very lightweight. It is normal for hundreds of coroutines to run in a golang program

Golang is born for concurrency. The syntax for starting a collaborative process is very simple. You can use the go keyword


go func () {
    // do something
}

Sync.waitgroup

Multiple processes can be synchronized through sync. Waitgroup, which is similar to semaphores in Linux

Var WG sync.waitgroup // declare a semaphore
WG. Add (1) // semaphore plus one
WG. Done() // semaphore minus one
WG. Wait() // block when the semaphore is positive, and wake up when the semaphore is 0

Channel Chan

A channel can be understood as a message queue. The producer puts it in the queue and the consumer takes it from the queue. Channels can be closed using close

IC: = make (Chan, int, 10) // declare a channel
IC < - 10 // put it in the channel
I: = < - IC // fetch from the channel
Close (IC) // close the channel

Producer consumer realization

Define product classes

This product class is defined according to specific business requirements


type Product struct {
    name  int
    value int
}

producer

If the stop flag is not false, continuously put product into the channel, and the semaphore is completed after completion


func producer(wg *sync.WaitGroup, products chan<- Product, name int, stop *bool) {
    for !*stop {
        product := Product{name: name, value: rand.Int()}
        products <- product
        fmt.Printf("producer %v produce a product: %#v\n", name, product)
        time.Sleep(time.Duration(200+rand.Intn(1000)) * time.Millisecond)
    }
    wg.Done()
}

consumer

The for loop will not terminate until the channel is closed and the products is empty, which is exactly what we expect


func consumer(wg *sync.WaitGroup, products <-chan Product, name int) {
    for product := range products {
        fmt.Printf("consumer %v consume a product: %#v\n", name, product)
        time.Sleep(time.Duration(200+rand.Intn(1000)) * time.Millisecond)
    }
    wg.Done()
}

Main thread

var wgp sync.WaitGroup
var wgc sync.WaitGroup
stop := false
products := make(chan Product, 10)
//Create 5 producers and 5 consumers
for i := 0; i < 5; i++ {
    go producer(&wgp, products, i, &stop)
    go consumer(&wgc, products, i)
    wgp.Add(1)
    wgc.Add(1)
}
time.Sleep(time.Duration(1) * time.Second)
Stop = true // set the producer termination signal
WGP. Wait() // wait for the producer to exit
Close (products) // close the channel
WGC. Wait() // wait for the consumer to exit

Supplement: go concurrent programming — implementing producer consumer model through channel

summary

Producer consumer model is a classic model of multithreading design, which is widely used in multithreading / process model design of various systems.

This paper introduces the characteristics of channel in go language, and implements two producer consumer models through go language.

Some features of channel

In go, channel is a very important means of process communication. Channel is a two-way channel. Data transmission between processes can be realized through channel, and synchronization between processes can also be realized through channel (described later).

The producer consumer model introduced in this paper mainly uses the following characteristics of channel: only one collaboration can access an item in the channel at any time.

Single producer single consumer model

Putting producers and consumers into a wireless loop is very similar to our server-side task processing. Producers constantly put data into the channel, while consumers constantly take data out of the channel and process (print) the data.

Since the producer’s collaboration will not exit, the channel write will exist forever. In this way, when there is no data in the channel, the consumer will block and wait for the producer to put data.

The code is implemented as follows:

package main
import (
    "fmt"
    "time"
)
var ch1 chan int = make(chan int)
var bufChan chan int = make(chan int, 1000)
var msgChan chan int = make(chan int)
func sum(a int, b int) {
    ch1 <- a + b
}
// write data to channel
func writer(max int) {
    for {
        for i := 0;  i < max;  I + + {// simply put an integer into the channel
            bufChan <- i
            Time. Sleep (1 * time. Millisecond) // control the frequency of insertion
        }
    }
}
// read data fro m channel
func reader(max int) {
    for {
        r := <-bufChan
        fmt.Printf("read value: %d\n", r)
    }
    //Notify the main thread that the work is finished, and this step can be omitted
    msgChan <- 1
}
func testWriterAndReader(max int) {
    go writer(max)
    go reader(max)
    //When the writer and reader tasks are finished, the main thread will be notified 
    res := <-msgChan
    fmt.Printf("task is done: value=%d\n", res)
}
func main() {
    testWriterAndReader(100)
}

Multi producer consumer model

We can implement the producer consumer model by using the feature that only one process in the channel can access a certain data at a certain point in time. Because the channel has such characteristics, we can put data and consumption data without locking.

package main
import (
    "time"
    "fmt"
    "os"
)
var ch1 chan int = make(chan int)
var bufChan chan int = make(chan int, 1000)
var msgChan chan string = make(chan string)
func sum(a int, b int) {
    ch1 <- a + b
}
// write data to channel
func writer(max int) {
    for {
        for i := 0; i < max; i++ {
            bufChan <- i
            fmt.Fprintf(os.Stderr, "%v write: %d\n", os.Getpid(), i)
            time.Sleep(10 * time.Millisecond)
        }
    }
}
// read data fro m channel
func reader(name string) {
    for {
        r := <-bufChan
        fmt.Printf("%s read value: %d\n", name, r)
    }
    msgChan <- name
}
func testWriterAndReader(max int) {
    //Open the goroutine of multiple writers and constantly write data to the channel
    go writer(max)
    go writer(max)
    //Open the goroutine of multiple readers, constantly read data from the channel and process data
    go reader("read1")
    go reader("read2")
    go reader("read3")
    //Get the task completion status of the three readers
    name1 := <-msgChan
    name2 := <-msgChan
    name3 := <-msgChan
    fmt.Println("%s,%s,%s: All is done!!", name1, name2, name3)
}
func main() {
    testWriterAndReader(100)
}

The output is as follows:

read3 read value: 0

80731 write: 0

80731 write: 0

read1 read value: 0

80731 write: 1

read2 read value: 1

80731 write: 1

read3 read value: 1

80731 write: 2

read2 read value: 2

80731 write: 2

… …

summary

This paper implements the classic producer and consumer model through channel, and makes use of the characteristics of channel. However, it should be noted that when the speed of the consumer is lower than that of the producer, the channel may be congested, resulting in increased memory consumption. Therefore, in the actual scenario, the size of the channel buffer needs to be considered.

When the size of the channel is set, when the production data is greater than the capacity of the channel, the producer will block. These problems need to be considered in the actual scenario.

One solution is to use a fixed array or slice as a ring buffer instead of a channel, synchronize through the sync package mechanism, and implement the producer consumer model, so as to avoid the blocking of the consumer end due to the full channel.

However, for ring buffer, old data may be overwritten, and specific usage scenarios need to be considered. The principle and implementation of ring buffer will be further analyzed when analyzing the use of sync package.

The above is my personal experience. I hope I can give you a reference, and I hope you can support developpaer. If you have any mistakes or don’t consider completely, please don’t hesitate to comment.