Crawler management platform crawleb v0.4.8 release (environment installation + git advanced application)



This update includes several parts:

  1. Supports interface installation (pre installation) of multiple language environments other than python
  2. Better installation interface, you can see the installation of multiple nodes
  3. Optimize RPC, based on redis delay queue as RPC
  4. More function configuration

Update log

Function / optimization

  • Support more programming language installationNow users can install or pre install more programming languages, including Java,. Net core and PHP
  • Install UI optimizationUsers can better view and manage the installation of node list page
  • More git support. allow users to view git commit records and check out to the corresponding commit
  • Support host name as node registration typeThe user can use the host name as the unique identification number of the node
  • RPC support. join RPC support to better manage node communication
  • Is the switch running at the master nodeThe user can decide whether to run on the master node. If not, all tasks will run on the work node
  • The tutorial is disabled by default.
  • Add related documents sidebar.
  • Loading page optimization.

Bug fix

  • Repeating node. #391
  • Repeat upload crawler. #603
  • The third-party module installation failure of the node causes that the third-party part of the node installation cannot be used. #609
  • Offline nodes also create tasks. #622

Product planning

  • Results display

    • Support for other databases
  • Configurable crawler

    • Configurable crawler support for splash
    • Configurable crawler support crawler
    • Configurable crawler supports regular expression fields
    • Configurable crawler supports conversion to custom crawler
  • task

    • Task retrial mechanism
  • Timed tasks

    • Calendar display
  • overall situation

    • Support version update log display
  • The server

    • Support terminal operation docker image
  • SDK

    • More command support
  • Plug in system

reference resources

  • Github:…
  • Demo:


If you feel that crawlab is helpful to your daily development or company, please add the author’s wechat tikazyq1 and indicate “crawlab”, and the author will pull you into the group. Welcome to star on GitHub, and if you have any problems, please feel free to ask issue on GitHub. In addition, you are welcome to contribute to the development of crawlab.

Crawler management platform crawleb v0.4.8 release (environment installation + git advanced application)