Source code analysis of distributed scheduling engine elastic job (II) job and registration based V3

Time:2022-5-22

1. Background

The scheduling engine is the key infrastructure. It not only executes tasks regularly, but also is a large-scale distributed task engine, distributed parallel processing platform, manages the cluster of computing nodes, and provides scalable data processing capacity with high throughput.

With the growing business of the company, the requirements for high throughput and high concurrency of the scheduling engine are also growing rapidly. It is necessary to build a task scheduling platform with no center, large cluster management, robust fragmentation and fault tolerance to support business development.

At the same time, the distributed scheduling engine is also the core technology of distributed transformation of dataX and sentinel dashboard

2. Reference

Taro source codehttps://blog.csdn.net/X5fnncxzq4/article/details/86770194

https://www.jianshu.com/nb/18843035

3. Terminology

Decentralized / decentralizedThere are central distributed settings. The central node is responsible for cluster coordination and metadata storage. For example, the admin / executor, dolphin scheduler and master worker of XXL job are all central distributed designs; There are few truly centerless designs. Most of them are equal nodes and can be elected as the master node, that is, any node can become the center

Cerebral fissureIn the decentralized design, when there is a problem in the network, the nodes are divided into multiple clusters, and the state consistency between clusters cannot be achieved due to lack of communication. The general solution is that half of the clusters with an odd number of cluster nodes and less than the total number of nodes stop working

Fragmentation / fault tolerance Sharding is a very important feature of the scheduling platform. Scheduling and processing large-scale data requires sharding execution. Sharding execution brings new problems. If sharding fails, the platform reclaims shards and transfers them to other nodes for execution

4. Detailed analysis

Services can be divided into functional services and core services. The core services support the services of functional services. Functional tasks include task registration, task execution, failure transfer, etc., which are the “business” functions of the scheduling platform

Design and execution model

Source code analysis of distributed scheduling engine elastic job (II) job and registration based V3

Elasticjob tag class

Elasticjobexecutor job execution logic, using jobitemexecutor to execute the job, in5.2 job registration and schedulingFurther analysis

Jobitemexecutor includes two types of jobs: Type D / classified. Type D jobs implement remote call executors, such as HTTP and feign. There is no need to implement job classes. The job process is a general execution node and will not rely on job business; Classed implementation executor / job

Simplejobexecution / simplejob simple job implementation. The platform provides other common job implementations, such as script, HTTP and dataflow. These are a pair of supporting implementations

5.2 job registration and scheduling

1) Job registration is divided into one-time / fixed-time

Primary / scheduled scheduling can be divided into whether there is cron configuration or not. If there is no cron configuration, it is regarded as primary scheduling and can be usedTrigger serviceTrigger job execution, and regularly schedule jobs to use local quartz

Source code analysis of distributed scheduling engine elastic job (II) job and registration based V3

2) Job initialization

spring boot starter

Source code analysis of distributed scheduling engine elastic job (II) job and registration based V3

ElasticJobLiteAutoConfigurationSpring boot automatically configures the main portal and imports other automatic configurations

ElasticJobBootstrapConfiguration From the name, it can be seen that the function is to initialize jobbootstrap, obtain elasticjob instance from elasticjobproperties, build jobbootstrap, register with singletonbeanregistry, and obtain spring bean factory

ScheduleJobBootstrapStartupRunner Commandlinerunner is implemented. The spring bean factory obtains the schedulejobbootstrap instance and schedules the job to the local quartz

3) Job registration and scheduling

Source code analysis of distributed scheduling engine elastic job (II) job and registration based V3

JobSchedulerElastic job scheduler, initialization of job execution environment, construction and initialization of quartz scheduler

JobRegistrySingleton, which is equivalent to the job execution context, owns the status of the job, scheduler and registry

JobScheduleControllerEncapsulation of quartz scheduler

ScheduleJobBootstrap/OneOff JobBootstrapScheduling jobs using jobscheduler

*It seems appropriate to build and initialize the quartz scheduler and hand it over to the jobschedulecontroller

Ø rely on core services:

Set up service  Put in the job configuration and initialize other core services

Scheduling service: scheduling jobs as local quartz jobs

Source code analysis of distributed scheduling engine elastic-job3 (2) job model and registration

Source code analysis of distributed scheduling engine elastic-job3 (III) – job execution

Source code analysis of distributed scheduling engine elastic-job3 (IV) core service I

Source code analysis of distributed scheduling engine elastic-job3 (V) – core service II_ Leehom’s blog – CSDN blogvip

Source code analysis of distributed scheduling engine elastic-job3 (VI) – failure transfer_ Leehom’s blog – CSDN blogvip

Source code analysis of distributed scheduling engine elastic-job3 (2) job model and registration