On the analysis of think queue

preface

Before analysis, please understand the implementation of message queue

If you don’t know, please read the following:

Design of liking message queue

Design of qunar network message queue

TP5 message queue is based on database redis and top think, which is officially implemented by TP
This chapter focuses on the analysis of redis

Storage key:

key	type	describe
queues:queueName	list	Tasks to perform
think:queue:restart	string	Restart queue timestamp
queues:queueName:delayed	zSet	Delay task
queues:queueName:reserved	zSet	Execution failed, waiting for re execution

Execute the order

The difference between work and listen is explained below

command	describe
php think queue:work	listen queue
php think queue:listen	listen queue
php think queue:restart	Restart queue
php think queue:subscribe	No, it may be reserved. What other ideas the government has but it hasn’t been realized

Behavior label

label	describe
worker_daemon_start	Daemons on
worker_memory_exceeded	Out of memory
worker_queue_restart	Restart the daemons
worker_before_process	Before the task starts
worker_before_sleep	Task delay
queue_failed	Task execution failed

Command parameters

parameter	Default value	Modes that can be used	describe
queue	null	work,listen	The name of the task to perform
daemon	null	work	Perform tasks as Daemons
delay	0	work,listen	Time to re execute after failure
force	null	work	Time to re execute after failure
memory	128M	work,listen	Limit maximum memory
sleep	3	work,listen	Waiting time when there is no task
tries	0	work,listen	Maximum attempts after task failure

Pattern differences

1: The execution principle is different
Work: single process processing mode;
The work process directly ends the current process after processing the next message. When there is no new message, it will sleep for a period of time and then exit;
With the daemon parameter, the work process processes the messages in the queue in a loop until the memory exceeds the parameter configuration. When there is no new message, it will sleep for a period of time in each loop;

Listen: the processing mode of parent process + child process;
A work child process with single execution mode will be created in the parent process, and the next message in the queue will be processed through the work child process. After the work child process exits, the next message in the queue will be processed;
The parent process will listen to the exit signal of the child process and re create a new single execution work sub process;

2: Different exit timing
Work: look at it
Listen: the parent process will normally run, unless the following two situations are encountered
01: the execution time of a work child process created exceeds the — timeout parameter configuration in the listen command line. At this time, the work child process will be forced to end, and the parent process where listen is located will also throw a processtimeoutexception and exit;

Developers can choose to catch the exception and let the parent process continue to execute;
02: the parent process has memory leakage for some reason. When the memory occupied by the parent process exceeds the — memory parameter configuration in the command line, both the parent and child processes will exit. Under normal circumstances, the memory occupied by the listen process itself is stable.

3: Different performance
Work: it makes a loop inside the script. The framework script has been loaded in the early stage of command execution;

Listen: a new work process is opened after processing a task, and the framework script will be reloaded at this time;

As a result, listen mode has a higher performance than work mode.
Note: when the code is updated, PHP think needs to be executed manually in work mode queue:restart The command restarts the queue to make the changes take effect; the listen mode takes effect automatically, and no other action is required.

4: Time out control capability
Work: in essence, it can neither control the running time of the process itself, nor limit the execution time of the executing task;
Listen: you can limit the timeout time of the work subprocess it creates;

The maximum allowed running time of work subprocess can be limited by timeout parameter, and the child process that has not finished beyond the time limit will be forced to end;
The difference between expiration and time

Expiration is set in the configuration file, which refers to the expiration time of the task. This time is global and affects all work processes
Timeout is set in the command line parameter, which refers to the timeout time of the work subprocess. This time is only valid for the currently executed listen command. Timeout is targeted at the work subprocess;

5: Different scenarios

The applicable scenarios of work are as follows:
01: more tasks
02: high performance requirements
03: the execution time of the task is short
It is not easy to cause a dead loop in die() class

The application scenarios of listen are as follows:

01: fewer tasks
02: the task takes a long time to execute
03: task execution time needs to be strictly limited

Public operation

Since we do the analysis according to redis, we only need to analyze Src / queue / connector/ redis.php
01: call firstsrc/Queue.phpMagic method in__callStatic
02: Yes__ Called in the callStatic method.buildConnector
03: first load the configuration file in buildconnector. If there is no configuration file, it will be executed synchronously
04: create a connection according to the configuration file and pass in the configuration

stay redis.php Operation in constructor of class:
01: check whether the redis extension is installed
02: merge configuration
03: check whether it is redis extension or predis
04: create connection object

Release process

Publish parameters

Parameter name	Default value	describe	Methods that can be used
$job	nothing	The class to perform the task	push,later
$data	empty	Task data	push,later
$queue	default	Task name	push,later
$delay	null	delay time	later

Immediate execution

    push($job, $data, $queue)
    Queue::push(Test::class, ['id' => 1], 'test');

After a single operation, an array is returned and the serialized rpush is transferred to redis. The key is queue:queueName
Array structure:

[
    'job' = > $job, // the class to perform the task
    'data '= > $data, // task data
    'ID' = >'xxxxx '// task ID
]

Write redis and return the queue ID
As for the middle of that Sao operation is too long to write

Delayed release

    later($delay, $job, $data, $queue)
    Queue::later(100, Test::class, ['id' => 1], 'test');

It’s similar to the one above
After a single operation, it returns an array and serializes zadd to redis. The key isqueue:queueName:delayedScore is the current timestamp + $delay

Execution process

There are two kinds of differences in the execution process: work mode and listen mode. As mentioned above, the code logic is too much to be decomposed;
Finally, let’s talk about the use of labels

//Daemons on
    'worker_daemon_start' => [
        \app\index\behavior\WorkerDaemonStart::class
    ],
    //Out of memory
    'worker_memory_exceeded' => [
        \app\index\behavior\WorkerMemoryExceeded::class
    ],
    //Restart the daemons
    'worker_queue_restart' => [
        \app\index\behavior\WorkerQueueRestart::class
    ],
    //Before the task starts
    'worker_before_process' => [
        \app\index\behavior\WorkerBeforeProcess::class
    ],
    //Task delay
    'worker_before_sleep' => [
        \app\index\behavior\WorkerBeforeSleep::class
    ],
    //Task execution failed
    'queue_failed' => [
        \app\index\behavior\QueueFailed::class
    ]

public function run(Output $output)
    {
        $output - > write ('< info > task execution failed < / Info >', true);
    }

Console executionphp think queue:work --queue test --daemon
It will be output once in the console

Daemons on
Task delay

If a task fails or the execution times reach the maximum value
Will triggerqueue_failed

stayapp\index\[email protected]Method to write failure logic, such as email notification to log, etc

Finally, let’s talk about how to push message queues to TP projects in other frameworks or projects. For example, two projects are separated and the other uses a framework that is not TP5

Push tasks in other projects

PHP version

<?php

class Index
{
    private $redis = null;

    public function __construct()
    {
        $this->redis = new Redis();
        $this->redis->connect('127.0.0.1', 6379);
        $this->redis->select(10);
    }

    public function push($job, $data, $queue)
    {
        $payload = $this->createPayload($job, $data);
        $this->redis->rPush('queues:' . $queue, $payload);
    }

    public function later($delay, $job, $data, $queue)
    {
        $payload = $this->createPayload($job, $data);
        $this->redis->zAdd('queues:' . $queue . ':delayed', time() + $delay, $payload);
    }

    private function createPayload($job, $data)
    {
        $payload = $this->setMeta(json_encode(['job' => $job, 'data' => $data]), 'id', $this->random(32));
        return $this->setMeta($payload, 'attempts', 1);
    }

    private function setMeta($payload, $key, $value)
    {
        $payload = json_decode($payload, true);
        $payload[$key] = $value;
        $payload = json_encode($payload);

        if (JSON_ERROR_NONE !== json_last_error()) {
            throw new InvalidArgumentException('Unable to create payload: ' . json_last_error_msg());
        }

        return $payload;
    }

    private function random(int $length = 16): string
    {
        $str = '0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ';
        $randomString = '';
        for ($i = 0; $i < $length; $i++) {
            $randomString .= $str[rand(0, strlen($str) - 1)];
        }
        return $randomString;
    }
}

(new Index())->later(10, 'app\index\jobs\Test', ['id' => 1], 'test');

Go version

package main

import (
    "encoding/json"
    "github.com/garyburd/redigo/redis"
    "math/rand"
    "time"
)

type Payload struct {
    Id       string      `json:"id"`
    Job      string      `json:"job"`
    Data     interface{} `json:"data"`
    Attempts int         `json:"attempts"`
}

var RedisClient *redis.Pool

func init() {
    RedisClient = &redis.Pool{
        MaxIdle:     20,
        MaxActive:   500,
        IdleTimeout: time.Second * 100,
        Dial: func() (conn redis.Conn, e error) {
            c, err := redis.Dial("tcp", "127.0.0.1:6379")

            if err != nil {
                return nil, err
            }

            _, _ = c.Do("SELECT", 10)

            return c, nil
        },
    }

}

func main() {

    var data = make(map[string]interface{})
    data["id"] = "1"

    later(10, "app\index\jobs\Test", data, "test")
}

func push(job string, data interface{}, queue string) {
    payload := createPayload(job, data)
    queueName := "queues:" + queue

    _, _ = RedisClient.Get().Do("rPush", queueName, payload)
}

func later(delay int, job string, data interface{}, queue string) {

    m, _ := time.ParseDuration("+1s")
    currentTime := time.Now()
    op := currentTime.Add(time.Duration(time.Duration(delay) * m)).Unix()
    createPayload(job, data)
    payload := createPayload(job, data)
    queueName := "queues:" + queue + ":delayed"

    _, _ = RedisClient.Get().Do("zAdd", queueName, op, payload)
}

//Creates data in the specified format
func createPayload(job string, data interface{}) (payload string) {
    payload1 := &Payload{Job: job, Data: data, Id: random(32), Attempts: 1}

    jsonStr, _ := json.Marshal(payload1)

    return string(jsonStr)
}

//Create random string
func random(n int) string {

    var str = []rune("0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ")

    b := make([]rune, n)
    for i := range b {
        b[i] = str[rand.Intn(len(str))]
    }
    return string(b)
}