(High concurrency detection) 3. Use of message queues for common problems in distributed scenarios

Time:2019-9-30

Scenario requirements

For existing MySQL master-slave projects, when the amount of data is large, we often adopt the method of sub-database and sub-table. In order to shorten the page response, we adopt the structure of one master-slave master-write + Slave read-write separation.

Using redis level: first, using redis as the buffer layer of PHP to store common and relatively fixed public data; second, adding redis long-term cache to compose MySQL write + redis read architecture; third, even directly using redis read + write architecture. The migration of redis by MySQL requires the close cooperation of background programs. “Read-write separation” is prone to data inconsistency.

Some optimization correlations

Traditional front-end and back-end optimization approaches:
 1. front end 
    Reduce the number of requests: CSS wizard (small map merged into large map), data-image (data-icon: SRC = data: image / jpg; base64; xx, small map merged into JS file);
 2. gateway
    Refer monitoring of web resource anti-theft chain, current limiting of nginx, load balancing of nginx, static resource caching of nginx, gzip, etc., http 2.0;
 3. rear end
    MySQL is optimized by using redis and Memcache caching.

Advance:
 1. gateway
    MySQL migrates redis and deploys distributed clusters.
 2. rear end
    Adding message queues for high concurrency, multi-threading, co-programming, using connection pool;

Common problems with distributed deployment:
    Login session sharing problem; read-write separation synchronization data problem.
In addition, there are also a variety of images such as database, object storage, and so on.

In order to deal with the problem of inventory oversold and order payment on multiple servers at the same time, frequent process checks are needed, so serialization of parallel tasks, using one machine and single thread are the most reliable ways to deal with this consistency problem. Message queue should be used to balance the pressure of server in case of large data volume.

1. swoole message queue correlation

Using swoole_table+coroutines, queues can be used to persist these in-memory data in MySQL and redis.

A. swoole_table shared memory table

Server side:

// Refer to official https://wiki.swoole.com/wiki/page/292.html
The $table = new Swoole Table (1024); the //$size parameter specifies the maximum number of rows in the table, which must be an index of 2, such as 1024, 8192, 65536, etc.
$table->column('fd', swoole_table::TYPE_INT);
$table->column('from_id', swoole_table::TYPE_INT);
$table->column('data', swoole_table::TYPE_STRING, 1024);
$table->create();

$serv = new Swoole\Server('127.0.0.1', 9501, SWOOLE_BASE, SWOOLE_SOCK_TCP);
// Save the table on the serv object
$serv->table = $table;

$serv->on('receive', function ($serv, $fd, $from_id, $data) {
    $pre = substr($data, 0, 1);
    $data = substr($data, 1);
    if($pre == \Swoole\Table::TYPE_STRING){
        /*
        foreach ($serv->table as $row) var_dump($row);
        print_r($serv->table);
        */
        $table = [];
        foreach ($serv->table as $k => $row) {
            $row['key'] = $k;
            $table[] = $row;
        }
        $count = count($serv->table);
        Print_r ("Number of rows in the current table". $count);
        $serv->send($fd, json_encode($table, JSON_UNESCAPED_UNICODE));
    }elseif($pre == \Swoole\Table::TYPE_FLOAT){
        $key = $data;
        $exist = $serv->table->exist($key);
        if($exist){
            $row = $serv->table->get($key);
            $exist = $serv->table->del($key);
            $data = json_decode($row['data'], true);
            $serv - > send ($fd, "consumption:". ($exist=== true?'true':'false').'. $data ['order']. PHP_EOL);
        }
    }elseif($pre == \Swoole\Table::TYPE_INT){
        $ret = $serv->table->set($fd, array('from_id' => $from_id, 'data' => $data, 'fd' => $fd));
        $data = json_decode($data, true);
        $serv - > send ($fd, "server:". ($ret=== true?'true':'false'),'from'. $fd.'order:'. $data ['order']);
    }else{
        print_r($data);
        $serv->send($fd, 'others');
    }
});

$serv->start();

Client:

for($i=1; $i<=50; $i++){
    go(function () use($i) {
        $client = new \Swoole\Client(SWOOLE_SOCK_TCP);
        usleep(mt_rand(10,1000));
        $res = $client->connect('127.0.0.1', 9501, 0.5);
        if (!$res) {
            exit("connect failed. Error: {$client->errCode}\n");
        }

        Echo "Client:" $i. PHP_EOL;
        $data = ['order'=>$i, 'data'=>md5(mt_rand(10,1000).time())];
        echo \Swoole\Table::TYPE_INT. json_encode($data, JSON_UNESCAPED_UNICODE). PHP_EOL;
        $client->send(\Swoole\Table::TYPE_INT. json_encode($data, JSON_UNESCAPED_UNICODE));
        //echo $client->recv(). PHP_EOL;
        $client->close();
    });
}

$client = new \Swoole\Client(SWOOLE_SOCK_TCP);
$res = $client->connect('127.0.0.1', 9501, 0.5);
if (!$res) {
    exit("connect failed. Error: {$client->errCode}\n");
}
$client->send(\Swoole\Table::TYPE_STRING. "GET_TABLE_INFO");
$table = $client->recv();
$client->close();

$table = json_decode($table, true);
while($row = array_pop($table)){
    go(function () use($row) {
        $client = new \Swoole\Client(SWOOLE_SOCK_TCP);
        usleep(mt_rand(10,1000));
        $res = $client->connect('127.0.0.1', 9501, 0.5);
        if (!$res) {
            exit("connect failed. Error: {$client->errCode}\n");
        }
        
        Echo "Processing {row ['fd']} from {row ['data']}". PHP_EOL;
        $client->send(\Swoole\Table::TYPE_FLOAT. $row['key']);
        $client->close();
    });
}

Swoole’s protocol has no obvious effect on ordinary processing, but asynchronous and non-blocking efficiency is reflected when tasks include remote connections such as mysql, redis, curl, etc. Asynchronous programming should always consider the issue of execution order. In the client side of the example above, sleep is really blocked rather than asynchronous. The second half reads the swoole_table table table callback report to complete and can be placed separately in the file request.

B. swoole + redis / MySQL message processing

Similarly, the advantage is more persistence.

2. rabbitmq message queue

Characteristics: passive acceptance of tasks, not refused. Select the persistent configuration, if not completed, all exit / PHP interrupt exit and restart will continue to execute.

Install RabbitMQ message queue

Rabbitmq official docker notes, refer to “Rabbitmq Cluster High Availability Deployment Details” here to deploy the common mode. Mybridge is established between network joining and fixed IP is used.

[]:~/tmp/dk/rabbitmq# docker pull rabbitmq
#docker run --name rabbit --network mybridge -e RABBITMQ_ERLANG_COOKIE='123456' -d rabbitmq 
# docker CP rabbit:/etc/rabbitmq/rabbitmq.conf. / Copy the configuration file
[]:~/tmp/dk/rabbitmq# docker run --name rbt1 -p 15672:15672 \
    --hostname rbt1 \
    --network mybridge --ip=172.1.12.13 \
    -v /root/tmp/dk/rabbitmq/rabbitmq.conf \
    -v /root/tmp/dk/rabbitmq/data13:/var/lib/rabbitmq/mnesia \
    -e RABBITMQ_ERLANG_COOKIE='123456' \
    -e RABBITMQ_DEFAULT_USER=root -e RABBITMQ_DEFAULT_PASS=123456 \
    -d rabbitmq
docker run --name rbt2 --hostname rbt2 \
    --network mybridge --ip=172.1.12.14 \
    -v /root/tmp/dk/rabbitmq/rabbitmq.conf \
    -v /root/tmp/dk/rabbitmq/data14:/var/lib/rabbitmq/mnesia \
    -e RABBITMQ_ERLANG_COOKIE='123456' \
    -e RABBITMQ_DEFAULT_USER=root -e RABBITMQ_DEFAULT_PASS=123456 \
    -d rabbitmq

Common Cluster Model

[]:~/tmp/dk/rabbitmq# docker exec -it rbt1 bash
[email protected]:/# rabbitmqctl stop_app
Stopping rabbit application on node [email protected] ...
[email protected]:/# rabbitmqctl join_cluster --ram [email protected]
Clustering node [email protected] with [email protected]
[email protected]:/# rabbitmqctl start_app
Starting node [email protected] ...
 completed with 0 plugins.
[email protected]:/# rabbitmqctl cluster_status
Cluster status of node [email protected] ...
[{nodes,[{disc,[[email protected]]},{ram,[[email protected]]}]},
 {running_nodes,[[email protected],[email protected]]},
 {cluster_name,<<"[email protected]">>},
 {partitions,[]},
 {alarms,[{[email protected],[]},{[email protected],[]}]}]
[email protected]:/# rabbitmq-plugins enable rabbitmq_management

You can then log in and access the http://remote_host:15672/administration page.
PHP adds AMQP extensions and Dockerfile adds (cffycls/php7:1.8):

apk add rabbitmq-c-dev
pecl install amqp

B. Message addition and consumption testing

Usage Reference 1, Reference 2: Enter new AMQPConnection () in phpstrom, press Ctrl and mouse confirmation key to enter amqp.php method usage annotation (php.net official temporarily can not find).
Producer (publish.php):

date_default_timezone_set("Asia/Shanghai");
// Configuration information
$conn_args = array(
    'host' => '172.1.12.13',
    'port' => '5672',
    'login' => 'root',
    'password' => '123456',
    'vhost'=>'/'
);
$k_route = k_route_1'; // routing key, used to bind switches and queues
$e_name ='e_switches'; // switch name

// Create connections and channels
$conn = new AMQPConnection($conn_args);
if (!$conn->connect()) {
    die("Cannot connect to the broker!PHP_EOL");
}

$channel = new AMQPChannel($conn);
Echo "<font color='red'> producer </font> PHP_EOL has been connected successfully! Prepare to publish information...". PHP_EOL;

// Creating Switch Objects
$ex = new AMQPExchange($channel);
$ex->setName($e_name);
$ex - > setType (AMQP_EX_TYPE_DIRECT); // Set switch type
$ex - > setFlags (AMQP_DURABLE); // Set whether the switch persists messages

// Send a message
$channel - > startTransaction (); // Start a transaction
for($i=1; $i<=50000; ++$i){
    Usleep (100); // Hibernation for 1 second
    $message = message data. $i.''. date ("Y-m-d H: i: s A");
    Echo: "Message Send Back:". $ex - > publish ($message, $k_route). PHP_EOL;
}
$channel - > commitTransaction ();//commit transaction

$conn->disconnect();

Consumer.php:

date_default_timezone_set("Asia/Shanghai");
// Configuration information
$conn_args = array(
    'host' => '172.1.12.13',
    'port' => '5672',
    'login' => 'root',
    'password' => '123456',
    'vhost'=>'/'
);
$k_route = k_route_1'; // routing key, used to bind switches and queues
$e_name ='e_switches'; // switch name
$q_name ='q_queue'; // queue name

// Create connections and channels
$conn = new AMQPConnection($conn_args);
if (!$conn->connect()) {
    die("Cannot connect to the broker!".PHP_EOL);
}
$channel = new AMQPChannel($conn);
Echo "<font color='red'> consumer </font>:". PHP_EOL. "Connected successfully! Prepare to receive information...". PHP_EOL;

// Create a switch
$ex = new AMQPExchange($channel);
$ex->setName($e_name);
// Direct type: [AMQP_EX_TYPE_DIRECT, AMQP_EX_TYPE_FANOUT, AMQP_EX_TYPE_HEADERS or AMQP_EX_TYPE_TOPIC]
$ex->setType(AMQP_EX_TYPE_DIRECT);
$ex - > setFlags (AMQP_DURABLE); //persistence

// Create queues
$q = new AMQPQueue($channel);
$q->setName($q_name);
$q - > setFlags (AMQP_DURABLE); // Persistence
$q->declareQueue();

// Bind switches to queues and specify routing keys
$q->bind($e_name, $k_route);

// Blocking mode receives messages
Echo "Blocking mode to receive messages:". PHP_EOL;
while(True){
    $q->consume(function ($envelope, $queue) {
        $msg = $envelope->getBody();
        Echo'Received:'. $msg. PHP_EOL; // Processing messages
        Sleep (1); //Receive Analog Processing Sleep for 1 Second
    } AMQP_AUTOACK; //ACK response
}
$conn->disconnect();

In the page can be executed, only in cli interactive mode to display the results received, (publisher web access, consumer processing in php-cli). Open the transaction above: All messages are received before processing begins.
As long as there are more than one consumer at work, the task will continue, which is to add new consumers (two at the same time):

[]:~/tmp/dk/html# php rabbitmq/consumer.php 
<font color='red'>consumer</font>:
Connected successfully! Prepare to receive information.
Blocking mode receives messages:
Received: Message data 7201 2019-07-08 16:08:05 PM
Received: Message data 7202 2019-07-08 16:08:05 PM
... ...

Like redis and mysql, using swoole coroutines can speed up significantly.

3. Second Kill/Red Pack Scene

The redis message queue can be used to deal with the second kill problem, the core is to solve the inventory oversold problem, testing.

A. Sale. PHP on the server side: add promotional information and initialize message queues

require "../vendor/autoload.php";

// Read from the cluster
$servers = ['172.1.50.11:6379', '172.1.50.12:6379', '172.1.50.13:6379', '172.1.50.21:6379'];
define('PROMOTION_KEY_PREFIX', 'promotionList_');
define('TIME_LIMIT_PREFIX', 'timeLimit_');

// Initialization: Find out the distribution of all nodes, prepare lPush and lRange functions, and cache basic information
$redisServers = [];
$slotNodes = [];
foreach ($servers as $addr){
    // random
    $r = new Redis();
    $server=explode(':',$addr);
    $r->connect($server[0], (int) $server[1]);
    $r->auth('123456');
    $redisServers[$addr] = $r;

    if(empty($slotInfo)){
        // A single node can see all nodes with slots
        $slotInfo = $r->rawCommand('cluster','slots');
        foreach ($slotInfo as $ix => $value){
            $slotNodes[$value[2][0].':'.$value[2][1].' '.($ix+1)]=[$value[0], $value[1]];
        }
    }
}
$crc = new \Predis\Cluster\Hash\CRC16();
// Note that when method = lRange, args parameters are ($key, $start, $end)
$opt = function ($method, $key, ...$args) use (&$redisServers, &$slotNodes, &$crc) {
    $code = $crc->hash($key) % 16384;
    foreach ($slotNodes as $addr => $boundry){
        if( $code>=$boundry[0] && $code<=$boundry[1] ){
            $host =explode(' ', $addr)[0];
            if(empty($args)){
                return $redisServers[$host]->$method($key);
            }elseif(count($args)==1){
                return $redisServers[$host]->$method($key, $args[0]);
            }else{ //...
                return $redisServers[$host]->$method($key, $args);
            }
        }
    }
};

// Adding data: centralize adding commodities, initialize message queue
/**
 *@ Param String $key commodity key
 *@ Param int $stock inventory
 */
$createQueue = function (String $key, int $stock) use (&$opt) {
    for ($i=0; $i<$stock; $i++){
        $opt('lPush', $key, $i);
    }
};
// First task
go(function () use (&$opt, &$limit1, &$createQueue){
    $limit1 = [
        'goods_id'=>'9505900000',
    ];
    $createQueue(PROMOTION_KEY_PREFIX. $limit1['goods_id'], 20);
    $has = $opt('lLen', PROMOTION_KEY_PREFIX. $limit1['goods_id']);
    Print_r ('Create goods1'. $has.'Number'. PROMOTION_KEY_PREFIX. $limit1 ['goods_id']. PHP_EOL);
    $has = 0;
    swoole_timer_tick(10, function ($timer_id) use (&$opt, &$limit1, &$has){
        $now = $opt('lLen', PROMOTION_KEY_PREFIX. $limit1['goods_id']);
        if($has != $now) {
            $has = $now;
            if ($now) {
                Print_r ('goods1 has another'. $has.'surplus'. PHP_EOL);
            } else {
                print_r('OK DONE1!'. PHP_EOL . PROMOTION_KEY_PREFIX . $limit1['goods_id'] . PHP_EOL);
                Swoole\Timer::clear($timer_id);
            }
        }
    });
});

// Second task: Increase time limit
go(function () use (&$opt, &$limit1, &$has, &$createQueue){
    $limit2 = [
        'goods_id'=>'2302500000',
        'start'=>date("Y-m-d H:i:s", strtotime('+ 1 minutes')),
        'end'=>date("Y-m-d H:i:s", strtotime('+ 5 minutes')),
    ];
    $createQueue(PROMOTION_KEY_PREFIX. $limit2['goods_id'], 500);
    // Preservation time condition
    $opt('lPush', TIME_LIMIT_PREFIX. $limit2['goods_id'], json_encode($limit2, JSON_UNESCAPED_UNICODE));

    $has = $opt('lLen', PROMOTION_KEY_PREFIX. $limit2['goods_id']);
    Print_r ('Create goods2'. $has.'Number'. PROMOTION_KEY_PREFIX. $limit2 ['goods_id']. PHP_EOL);
    $has = 0;
    swoole_timer_tick(2000, function ($timer_id2) use (&$opt, &$limit2, &$has){
        $now = $opt('lLen', PROMOTION_KEY_PREFIX. $limit2['goods_id']);
        if($has != $now) {
            $has = $now;
            if ($now) {
                Print_r ('goods2 has another'. $has.'surplus'. PHP_EOL);
            } else {
                print_r('OK DONE2!'. PHP_EOL . PROMOTION_KEY_PREFIX . $limit2['goods_id'] . PHP_EOL);
                Swoole\Timer::clear($timer_id2);
            }
        }
    });
});

B. Client buy. php: Consumption data

// Client work for each purchase request -- swoole resident memory, cache run first step
// Initialization: Find out the distribution of all nodes, prepare lPush and lRange functions, with the server (abbreviated)

// 2. Consumption data: simulating decentralized requests
Goods1 = 9505900000'; // Commodity known
$goods2 = '2302500000';

// Unlimited: 100 robbers - goods1 = 20
go(function () use (&$opt, &$limit1, &$has, &$createQueue, $goods1){
    $cart = [];
    $cart['key'] = PROMOTION_KEY_PREFIX. $goods1;
    for ($i=0; $i<100; $i++) {
        $cart['user'] = 'tom'.$i;
        go(function () use($cart, &$opt){
            Co:: sleep (mt_rand(1,500)*0.001); //Increase redis network connection time
            $state = $opt('lPop', $cart['key']);
            if($state){
                Print_r ($cart ['user'].'Successful purchase! '.PHP_EOL';
                return true;
            }else{
                // print_r ('- has been robbed'. $cart ['user']. PHP_EOL);
                print_r('*');
                return false;
            }
        });
    }
});
// Time limit: 1000 robberies - goods2 = 500
go(function () use (&$opt, &$limit1, &$has, &$createQueue, $goods2){
    $cart = [];
    $cart['key'] = PROMOTION_KEY_PREFIX. $goods2;
    // Obtaining time conditions
    $limit2 = json_decode($opt('lIndex', TIME_LIMIT_PREFIX. $goods2, 0), true);
    for ($i=0; $i<10000; $i++) {
        $cart['user'] = 'jack'.$i;
        go(function () use($cart, &$opt, &$limit2){
            $min = strtotime($limit2['start']);
            $max = strtotime($limit2['end']);
            Co:: sleep (mt_rand (2,6*60); // request dispersion 2s-6min/(1-5min)

            $cur = time();
            if($cur<$min){
                Print_r ('- not yet started'. $cart ['user']. PHP_EOL);
                return true;
            }elseif($min<=$cur && $cur<=$max){
                Co:: sleep (mt_rand(100,500)*0.001); //Increase redis network connection time
                $state = $opt('lPop', $cart['key']);
                if($state){
                    Echo $cart ['user'].'Successful purchase! '.PHP_EOL;
                    return true;
                }else{
                    echo '.';
                    // echo'++ has been robbed of'. $cart ['user']. json_encode ([$limit2, date ('Y-m-d H: i: s', $cur)], JSON_UNESCAPED_UNICODE). PHP_EOL;
                    return false;
                }
            }else{
                Echo'++ has been robbed of'. $cart ['user']. json_encode ([$limit2, date ('Y-m-d H: i: s', $cur)], JSON_UNESCAPED_UNICODE). PHP_EOL;
                return false;
            }
        });
    }
});

C. Operation results

[]:~/tmp/dk/html/php-swoole# php sale.php
Create goods1 20 promotionList_9505900000
Create goods2 500 promotionList_2302500000
Goods1 has another 20 left
Goods2 has another 500 left
Goods1 has another 20 left
Goods2 has another 500 left
Goods1 has another 10 left
OK DONE!
promotionList_9505900000
Goods2 has another 500 left
Goods2 has 496 left
Goods2 has 441 left
Goods2 has 383 left
Goods2 has 314 left
Goods2 has 263 left
Goods2 has 208 left
Goods2 has 144 left
Goods2 has 88 left
Goods2 has 39 left
OK DONE!
promotionList_2302500000

[]:~/tmp/dk/html/php-swoole# php buy.php 
# To achieve the desired results, the length is too long.

Summary

Here swoole + redis asynchronously executes secondkill very quickly, and pays attention to memory usage when a large number of coordinations occur.