PHP message queue implementation and application details [queue processing order system and distribution system]

Time:2019-10-26

This paper introduces the implementation and application of PHP message queue. To share with you for your reference, as follows:

In the Internet project developers often encounter “mass texting users”, “order system has a large number of logs to record” or the server can’t bear the pressure of instant concurrency when seckilling business.

In this case, how can we ensure the normal and effective operation of the system?

At this time, we can introduce a concept called “message queue” to solve the above requirements.

Concept, principle and scenario of message queuing

In the case of high concurrency, the program is often unable to handle in time. We introduce an intermediate system for shunt and depressurization.

So in essence, message queue is a middleware of queue structure. That is to say, after you put the message and content into this container, you can directly return it without waiting for its post-processing results. There will also be a program that reads the data and processes it in order.

1. Middleware of queue structure
2. After the message is put in, it does not need to be processed immediately.
3. Handled by subscribers / consumers in order

That is to say: when you encounter a large or time-consuming link, and your business does not need to know the results of this link immediately, using message queuing is a good choice.

The core structure is as follows:

Message queuing scenarios

I. when data needs redundancy
For example, in the order system, subsequent data conversion and recording are required. The message queue can store these data in the queue persistently, and then the order post-processing program will process them. After processing, the record will be deleted from the queue.

Decoupling of the system
Message queuing solves the problem of deep coupling between two systems.
When message queuing is used, there is no direct relationship between the queued system and the queued system.
Entry system and exit system, one of which will not affect the normal operation of the other after a crash.

III. peak shaving
That is, when the second kill and rush to buy, there will be a sharp increase in traffic, which has a great pressure on the server.
In the actual project development, it is a good scheme to use message queue with cache.

IV. asynchronous communication
Message queuing itself implements asynchronous operation of programs, so message queuing can be used as long as it is suitable for asynchronous scenarios.

V. expansibility
For example, the order system, after the order enters the team, may have the financial system to process later, but if a distribution system is added.
Just let the distribution system subscribe to the message queue.

Vi. sequencing guarantee
In some scenarios, the sequence of data processing is very important. The queue itself can be made into a single thread single in single out system.
So as to effectively ensure that the data is processed in order.

Advantages and disadvantages of common queue implementation

Queue media:

MySQL: high reliability, easy implementation and slow speed
Redis: fast, low efficiency for a single large message package
Message system: professional, reliable, high learning cost (e.g. rabbitimq)

Triggering mechanism of message processing:

Dead cycle reading: easy to realize, unable to recover in time in case of failure;
Timing task: pressure sharing, with upper limit of processing capacity. (the biggest defect: the time interval of positioning task and the data processed need to be accurately grasped. If the previous task has not been processed, the next one will be started.)
Daemons: similar to php-fpm and php-cgi, requiring shell knowledge

Decoupling case: queue processing order system and distribution system

We have seen the usage scenario of message queuing before.

Here, we are going to deal with one of the scenarios: decoupling of the system.

In the e-commerce project, when a customer submits an order, the customer can see that the order is in the delivery in the personal center.
At this time, we need to participate in a system called “distribution system”. If we design the order system and distribution system together when we build the architecture, there will be some problems: the pressure of the order system is relatively large, but the distribution system does not need to respond to these pressures in a timely manner; we do not need to cause the distribution system to fail after the order system fails.

So we need to separate these two systems and realize the communication of these two systems through a middle team list.

As shown in the following figure:

 

Specific to our program code, the general logic is as follows:

 

General process: order.php receives the user’s order, generates the order number and processes the order (order system); in the order system, the data required by the distribution system will be put into the team list; our distribution system, goods.php, will have a timing script that executes every minute to process the data in the team list.

Simple design team list

CREATE TABLE `order_queue` (
 `id` int(11) unsigned NOT NULL AUTO_INCREMENT,
 `Order [ID ` int (11) unsigned not null comment 'order ID (from order system)',
 `User_info ` varchar (255) not null default '' comment 'can be the user's mobile number / user ID, etc. (here is just a demonstration)',
 `Created "at ` datetime not null comment 'order creation time',
 `Updated "at ` datetime not null comment 'the last processing completion time of this record',
 `Status ` tinyint (2) not null comment '0 not processed, 1 processed, 2 processing',
 PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;

MySQL order queue

We have analyzed the logic before, and the rest is the code implementation.

Note: I’m just demonstrating the code here, just to show the implementation process.

1. Receive order and process order.php

<?php
//This file is used to receive the user's order information and write it to the queue.
if(!empty($_GET['user_info'])){
  //Verify filter received data
  // todo...
  //Here is the processing flow of the order center.
  //Because the order system is a separate system, this system will not be written here.
  // todo...
  $order_id = rand (10000099999); // the normal order number comes from the order system. Here we just demonstrate
  //Store the order data required by the distribution system in the team list
  $insert_data = array(
    'order_id'=>$order_id,
    'user_info'=>$_GET['user_info'],
    'created_at'=>date('Y-m-d H:i:s',time()),
    'status'=>0
  );
  //Insert the above data into the order queue table
  // insert into order_queue
}

2. Distribution system goods.php

<?php
//This file is mainly used by the distribution system to process and mark the orders in the team list.
// analysis:
//Step 1: first, update the records to be processed to "wait for processing"
//Step 2: select the record just marked as "waiting for processing", and then process the distribution system.
//Step 3: mark the program handled above as "completed"
/////////////////////It's important here. You have to understand.//////////////////////////////////////////////
//Question: why don't you update the final processing to "completed" and mark it as "waiting for processing" first?
//This is because the distribution system is likely not completed in time, and there will be a period of processing time in the middle. If there are other programs in the process to read and operate, there will be conflicts.
//This design is actually a lock mechanism.
//1、
$waiting = array('status'=>0);
$lock = array('status'=>2);
//Mark the record with the status of 0 as 2, and update 3 records each time (depending on the situation)
$sql = "update order_queue set status=2 where status=0 limit 3";
//2、
If (update above succeeded){
  //Select the order content to be processed
  // select * from order_queue where status = 2;
  //It is then processed by the distribution system
  // todo...
  //3. After processing, update the order status to completed.
  $success = array(
    'status'=>1,
    'updated_at'=>date('Y-m-d H:i:s',time())
  );
}else{
  echo 'All Finished';
}

3. Linux server timing task

Write a shell script: goods.sh


#!/bin/bash
date "+%G-%m-%d %H:%M:%S"
cd /var/www/
php goods.php

This script is to execute the orders.php program.

To deploy a scheduled task on a Linux server:


crontab -e

*/1 * * * * /var/www/goods.sh >> /var/www/goods_shell.log 2>$1

Execute the goods.sh file once a minute, and record the log to the goods_shell.log file (create the file in the corresponding directory)

For more information about PHP, readers interested in PHP can refer to our website’s topics: PHP data structure and algorithm tutorial, PHP programming algorithm summary, PHP string usage summary, PHP array operation skills, PHP common traversal algorithm and skills summary and PHP mathematical operation skills summary.

I hope that this article will be helpful for PHP programming.