Crawlab Lite officially released, a lighter crawler management platform

Time:2021-6-16

Crawleb is a distributed crawler management platform based on golang. The product has been released for more than a year. After continuous polishing by the development team, it will be iterated to version v0.5. During this period, we added a large number of functions that the community users expected together to make the product more professional. But at the same time, we also notice that some users are not distributed crawler users, they only use crawleb in the case of stand-alone. Secondly, at present, crawleb has gradually added a wealth of functions, which makes the whole product gradually huge. Most users can not make full use of all the functions.

Therefore, crawleb Lite is officially released as a lightweight version of crawleb. Compared with crawleb, this version will focus on solving crawler management in stand-alone environment, and the platform itself will no longer rely on any external database. The existing and future functions are only limited to the necessary functions for high-frequency use, so as to maximize the functional restraint.

Functional features

The first released version isv0.0.1As the first experience version, this version already has the core functions of Lite planning. Compared with crawleb, the main differences are as follows:

  • It only supports crawler management in stand-alone environment and no longer has distributed function
  • The built-in database relies on the key value file, so it is no longer necessary to configure the external database
  • A large number of unnecessary functions have been deleted, making the platform lighter and easier to use
  • A large number of back-end code refactoring, platform stability and speed greatly improved

Operation mode

We still reserve the most convenient way of docker combine:

  1. Create in any directorydocker-compose.ymlThe contents are as follows:
version: '3'
services:
  master:
    image: zkqiang/crawlab-lite:latest
    container_name: master
    ports:
      - "8080:8080"
  1. Run the command in the directory:
docker-compose up -d
  1. visithttp://localhost:8080

Project address

https://github.com/crawlab-team/crawlab-lite

Project screenshot

Crawlab Lite officially released, a lighter crawler management platform

Crawlab Lite officially released, a lighter crawler management platform

Crawlab Lite officially released, a lighter crawler management platform

Crawlab Lite officially released, a lighter crawler management platform