How can laravel implement pseudo multiprocessing without using pcntl / pthreads / spool?

Time:2021-4-30

introduction

  • As we all know, multiprocess / multithread can execute multiple tasks in parallel / concurrent, which can improve the efficiency.
  • PHP does not support multiprocessing / multithreading by default. You need to install the pcntl / pthreads extension to support it. If you don’t use the framework such as spool, the implementation is more complex.
  • The above methods are not used, so how to improve the efficiency of the program– more–>

thinking

  1. For time-consuming tasks, they are usually pushed to the task queue, and then the queue consuming process obtains the task execution from the task queue.
  2. A queue can open multiple consuming processes, so the efficiency of task execution is much higher than that of single process sequential execution.
  3. If you don’t need to wait for all tasks to complete to get the results, it’s enough to start multiple queue consumption processes.
  4. If you need to wait for all tasks to complete before returning the results, for example, in the timing task, you need to read 100 records of MySQL and call the third-party API. The third-party API is very low, and it takes 2 seconds to call once. Finally, you need to generate the CSV file of these 100 records. Sequential execution takes at least 200 seconds to complete. If there are four queue consumption processes, it only takes about 50 seconds to complete.
  5. The main problem is how to solve the problem of interprocess communication? Because we need to know whether these subtasks are finished and the result of the task. Can redis be used to communicate? Use redis to count. After each task is executed, add 1 to the counter. The execution result of each task is put in the list / hash of redis. When the total number of counters is equal to the total number of tasks, it can be concluded that the task has been executed. Then take out the result stored in redis and generate a CSV file.

realization

The actual project has been running steadily online for several months. Here, I only post pseudo code to show the implementation idea. If you are interested, you can try it yourself.

  • Timing task command, here is the program entry, set the counter, and wait for the queue task execution to complete, get the result.

    How can laravel implement pseudo multiprocessing without using pcntl / pthreads / spool?

  • The job code of specific task mainly executes time-consuming tasks and adds 1 to the execution completion counter. It should be noted that the counter + 1 should also be guaranteed for exceptions. Otherwise, the total number of tasks is not equal to the total number of execution, and the main process command will be stuck.

    How can laravel implement pseudo multiprocessing without using pcntl / pthreads / spool?

  • To save the task related information to the hash data structure of redis, you need to save the key of the total number of tasks, which is used to count the total number of executed tasks. The count can be done with the time complexity of O (1)hincrbyOrders.

    How can laravel implement pseudo multiprocessing without using pcntl / pthreads / spool?

  • The results can also be saved in the hash. Before getting the results through hvals, delete the two count keys.

How can laravel implement pseudo multiprocessing without using pcntl / pthreads / spool?

  • The code example is only suitable for learning to use, if you need to use it in production, you need to deal with the details yourself.
  • To open multiple task queues, you can manually open several terminals for manual executionphp artisan queue:work redis --queue=calculateTo start multiple queue consumption processes, you can also use thesupervisorTo start, through configurationnumprocs=8Parameter to limit the number of processes.