Examples of using QueryList to crawl and store MySQL database based on ThinkPHP5 framework

Time:2019-10-6

This paper illustrates how to use QueryList to crawl and store MySQL database based on ThinkPHP5 framework. Share for your reference, as follows:

QueryList4 tutorial address:

https://doc.querylist.cc/site/index/doc/45

Install QueryList by executing the composer command in the ThinkPHP5 code root directory:


composer require jaeger/querylist

If the following error occurs

Loading composer repositories with package information
Updating dependencies (including require-dev)
    Authentication required (packagist.phpcomposer.com):
      Username:

This is the case.

Use


composer config -g repo.packagist composer https://packagist.laravel-china.org

The following demonstrates the use of QueryList in the Index controller:

use QL\QueryList;
public function qulist(){
  $data = QueryList::get('http://maoyan.com/board/4')
  // Setting up acquisition rules
  ->rules([
    // Crawl Picture Address
    "src"=>array(".board-wrapper dd img.board-img","data-src"),
    // Crawl for the title of the movie
    "name"=>array(".board-wrapper dd .movie-item-info .name","html"),
    // Crawling for movie starring information
    "star"=>array(".board-wrapper dd .movie-item-info .star","html"),
    // Climb the show time
    "releasetime"=>array(".board-wrapper dd .movie-item-info .releasetime","html"),
  ])
  ->query()->getData();
  $excel_array=$data->all();
  var_dump($excel_array);exit;
}

If there is no mistake

If you want to continue to grab data from the next page, follow the rules.

Here I continue to grab 20 pages, but not as many as 10 pages of data.

public function qulist(){
    for($i=0;$i<20;$i++){
      $page=$i*10;
      $data = QueryList::get('http://maoyan.com/board/4?offset='.$page)
        // Setting up acquisition rules
        ->rules([
          // Crawl Picture Address
          "src"=>array(".board-wrapper dd img.board-img","data-src"),
          // Crawl for the title of the movie
          "name"=>array(".board-wrapper dd .movie-item-info .name","html"),
          // Crawling for movie starring information
          "star"=>array(".board-wrapper dd .movie-item-info .star","html"),
          // Climb the show time
          "releasetime"=>array(".board-wrapper dd .movie-item-info .releasetime","html"),
        ])
        ->query()->getData();
      $excel_array=$data->all();
      var_dump($excel_array);exit;
    }
}

Detailed steps for inserting data into the database

Readers who are interested in thinkPHP-related content can see the topics of this site: Introduction to ThinkPHP, Summary of Operation Skills of ThinkPHP Templates, Summary of Common Methods of ThinkPHP, Introduction to CodeIgniter, Introduction to Zend Framework Work Framework and Summary of PHP Template Technology.

I hope this article will be helpful to you in the design of PHP program based on ThinkPHP framework.