Implementation details of packaging and process management of rabbitmq based on yii2 (4)

Time:2021-2-24

preface

Web process management is to manage consumer sub processes in the form of writing files. In the process management file, a row of records corresponding to a child process is similar to:

89104_89102 = 07_14_delay_queue,07_14

< child process >_ < parent process > = < queue name >, < file flag >

The reason why files are selected for management is that it is easy to implement, and if there is a real error, the manual operation of files is also very simple and easy. In line with the initial design idea, the implementation of the code is started.

Flow chart of implementation

Process management is divided into several parts, and then these parts are spliced together to realize the management of consumer process.

1. Socekt part: receive requests from web pages and distribute them

2. Dispatcher part: it becomes the guardian process and starts the AMQP consumer

3. Process file reading and writing part: read and write the sub process information in the file

The flow chart of each part is as follows:

Implementation details of packaging and process management of rabbitmq based on yii2 (4)

However, there are several problems to be solved in the process of implementation

1. How to ensure the sequence of process file reading and writing in the case of multi process?

2. Subprocess management and communication

In fact, it’s easy to solve the problem of ensuring the sequence of file reading and writing. What we write is about AMQP message queue, so it’s easy to think of using message queue. But what kind of message queue? Three methods are implemented in the class library: AMQP, beanstalk and redis. Process communication is also a message queue, in addition to the above three can be reused, but also added a pipe (file). The process of implementation is not smooth, thanks to the release of “Q & a” here, it can solve the problem that “multi generation subprocess can’t receive SIGTERM signal”. This is a community with temperature, so I decided to publish this series of articles in the community.

Details can be seen as follows:Q & A: multi process signal processing related: the first generation of the parent process can receive SIGTERM signal

Consumer profile

The consumer file is the key to start the consumer process

[consumer]
program =07_ 14; document mark
queueName = 07_14_delay_queue
qos =1; the number of consumer pretreatment
duplicate =1; the number of queue copies, 0 or 1, is the value of queuename directly
numprocs =2; the number of subprocesses
script =/The execution script of path / to / Yii; yii2
request =Command controller of AMQP / consumer; yii2

; {PHP} is the command placeholder for executing PHP, which has been configured in the [common] module
; {script} PHP execute script
; {request} the requested controller
; {queuename} is the value of the queuename property
; {QoS} is the value of QoS attribute, which is 1 by default
; command = {php} {script} {request} {queueName} {qos}
Finally, the executed command is as follows: '/ usr / bin / PHP / path / to / Yii AMQP / consumer {queuename} 1'. The queuename will be changed due to the configuration of duplicate queue.

amqp.ini configuration file

amqp.ini Configuration file is the key to start the whole process management

; common configuration properties
[common]
; log also supports recording% Y -% m -% d by date
;% Y: 2020, or% Y: 20
;% m month: 08
;% d day: 07
; ../log/%Y-%m/access_%Y-%m-%d.log
access_log = ../log/access.log   ; ../log/%Y-%m/access_%Y-%m-%d.log
error_log = ../log/error.log
Options: debug, info, notice, warning, error, critical, alert, emergency
; the selected types are separated by commas, and only the selected type logs are recorded
level = INFO,WARNING,ERROR,NOTICE
; save the PID of AMQP consumer daemons
pidfile = /var/run/amqp_master.pid
; execution ExecDispatcher.php Script commands
command = /usr/bin/php
; - pzr / AMQP / cli / server local file address to start UNIX connection
unix = /var/run/amqp_consumer_serve.sock
; process file management path
process_file = ./process_manager.ini

; AMQP consumer read connection configuration
[amqp]
host = 127.0.0.1
port = 5672
user = guest
password = guest

Process file processing: enable Beanstalk
[beanstalk]
host = 127.0.0.1
port = 11300

Process file processing: enable redis
[redis]
host = 127.0.0.1
port = 6379
user = 
password = 

; to inform the communication mode of the parent process
[pipe]
The file address of the communication between the child process and the parent process
pipe_file = /tmp/amqp_pipe

[communication]
; options: redis, AMQP, Beanstalk, pipe (default)
class = pipe

[handler]
Optional: Beanstalk, redis, AMQP
; there is no default value, so you must configure one
class = beanstalk

[include]
files = ./consumer/*.ini

The relative paths in the configuration file are in the amqp.ini The directory of is the root path.

follow-up

Furthermore, the previous three articles mainly show the simple operation of rabbitmq based on yii2, and do not elaborate on the implementation details. However, in fact, the implementation details of the basic operation encapsulation of rabbitmq seem to be easier to understand. Here we will talk about the bit by bit recorded in the process of implementation.

Why write rabbitmq encapsulation based on yii2?

The reason is simple: want to know some details about the implementation of yii2. Because yii2 has not been used in previous projects, so I’m ready to take a look and feel for it. But seeing is always seeing, so it may be more helpful to understand to decide what to do. So I plan to explore yii2 with the implementation of rabbitmq.

But in the end, it slowly deviated from the original intention. Yii2 felt satisfied with my implementation of rabbitmq after watching it for about a week or two, so I started to encapsulate rabbitmq, but it took about a month (of course, in my spare time, I went home late, but it was very substantial). Before implementation, I have a look at ityii2-queueIt is found that it can only implement the simplest queue, but many of its implementation methods are worth learning, such as the encapsulation idea of message body, the serialization implementation of message body, the idea of event subscription and so on. The serialization implementation of message body is intact. The encapsulation idea and event subscription idea of message body follow the same idea, but the implementation is different.

What are the basic usages?

1. Queue implementation

(1) The implementation of various queues includes ordinary queue, delay queue, priority queue and RPC queue.

(2) There are four routing types: direct, topic, fan out and header.

(3) By default, the queue started will start backup routing, which can prevent the loss caused by message routing failure.

(4) By default, the client message confirmation mechanism is turned on, which can be turned off through event subscription. By default, the consumer confirmation mechanism is turned on, or it can be turned off through event subscription.

(5) The concept of queue replica is added, which can be implemented by configuration.

(6) Message publishing can be single or batch.

(7) Serialization support of message body: JSON, igbinary, serialize.

2. Consumer realization

Consumers include general queue consumers and RPC queue consumers. Because the implementation of RPC is different from that of general queue, the implementation and consumer implementation are separate.

3. Provides the API function of AMQP

Through some information provided by API, we can supervise AMQP service, realize image strategy, protect existing queue, monitor and manage consumer connection, etc.

unknown delivery_ Tag anecdotes

In the process of implementation, we encountered an interesting problem, and proposed an issue to the author of amqplib. But they don’t think it’s a bug, they think it’s a wrong way to use it.

Problem Description: Publishing batch messages in batches for many times and reporting errorsPHP Fatal error: Uncaught PhpAmqpLib\Exception\AMQPRuntimeException: Server ack'ed unknown delivery_tag "19"

Cause of the problem: after the client message confirmation mechanism is turned on, the message publishing will wait for the server to return the result, so as to confirm whether the message is sent successfully. But every time you send a message, rabbitmq willnext_delivery_tagSet to 1, and the server returns the confirmation messagenext_delivery_tagBut it will not be set to 1 every time, which shows that it is increasing. This leads to the inconsistency between the last sent tag and the returned tag. For details, please refer to issue address:github.com/php-amqplib/php-amqplib…

In order to meet similar problemsServer ack'ed unknown delivery_tag "19"The solution to this problem: this error may not be due to the consumer’s repeated ACK, it is likely to be due to multiple batch messages. (when searching for answers on the Internet, they all say that it’s ack repetition, and it’s copy and paste, which is particularly annoying!)

This work adoptsCC agreementReprint must indicate the author and the link of this article

Recommended Today

Third party calls wechat payment interface

Step one: preparation 1. Wechat payment interface can only be called if the developer qualification has been authenticated on wechat open platform, so the first thing is to authenticate. It’s very simple, but wechat will charge 300 yuan for audit 2. Set payment directory Login wechat payment merchant platform( pay.weixin.qq . com) — > Product […]