Flink source code 4-Slot allocation and task execution

Time:2021-11-25

Continued from the previous issue: — jobmaster#startjobexecution ()
resetAndStartScheduler();
——》JobMaster#resetAndStartScheduler
schedulerAssignedFuture.thenRun(this::startScheduling);
——》JobMaster#startScheduling()
schedulerNG.startScheduling();

SchedulerBase# startScheduling();

startAllOperatorCoordinators();
*Note: start scheduling
*/
startSchedulingInternal();

DefaultScheduler#startSchedulingInternal
schedulingStrategy.startScheduling();

LazyFromSourcesSchedulingStrategy#startScheduling();

*Note: apply for slot and start deploying executionvertices
*/
allocateSlotsAndDeployExecutionVertices(schedulingTopology.getVertices());
——》LazyFromSourcesSchedulingStrategy#allocateSlotsAndDeployExecutionVertices

*Note: apply for slot and deployment
* schedulerOperations = DefaultScheduler
*/
schedulerOperations.allocateSlotsAndDeploy(Collections.

DefaultScheduler#allocateSlotsAndDeploy()

*******Come to the official entrance: defaultscheduler #allocateslotsanddeploy()***********
technological process:
1. Jobmaster sends request slot
2. The resource manager receives the request and executes slot request processing
3. Taskmanager handles slot requests sent by ResourceManager
4. The Jobmaster receives the slot application processing result sent by the taskmanager

1. Jobmaster sends request application slot 0:31~

—— 5 》DefaultScheduler#allocateSlotsAndDeploy()

*1 note: apply for slot
final List<SlotExecutionVertexAssignment> slotExecutionVertexAssignments = allocateSlots(executionVertexDeploymentOptions);
………….
*2 note: deployment run
*/
waitForAllSlotsAndDeploy(deploymentHandles);

——Defaultscheduler #allocateslots() 0:21
*Note: apply for slot
*/
final List<SlotExecutionVertexAssignment>
slotExecutionVertexAssignments = allocateSlots(executionVertexDeploymentOptions);
——》DefaultScheduler#allocateSlots()

——4 》DefaultExecutionSlotAllocator#allocateSlotsFor

  • Note: normalslotproviderstrategy*/
    slotProviderStrategy.allocateSlot

    SlotProviderStrategy#NormalSlotProviderStrategy#allocateSlot()

    SchedulerImpl#allocateSlot()
    ——》SchedulerImpl#allocateSlotInternal
    ——》SchedulerImpl#internalAllocateSlot
    —— 》SchedulerImpl# allocateSingleSlot

    //Step 1: get from the pool
    Optional<SlotAndLocality> slotAndLocality = tryAllocateFromAvailable
    if(slotAndLocality.isPresent()) {
    Return / / if there is a direct allocation in the slot pool
    CompletableFuture.completedFuture(completeAllocationByAssigningPayload
    Else {/ / otherwise new
    return requestNewAllocatedSlot(slotRequestId, slotProfile
    }
    ——》SchedulerImpl# requestNewAllocatedSlot
    return slotPool.requestNewAllocatedSlot(slotRequestId,

    SlotPoolImpl#requestNewAllocatedSlot()
    ——》 SlotPoolImpl# requestNewAllocatedSlotInternal()
    ——》 SlotPoolImpl# requestSlotFromResourceManager()
    //Actually send the request to the resource manager
    CompletableFuture<Acknowledge> rmResponse = resourceManagerGateway
    .requestSlot

    ResourceManager#requestSlot()
    slotManager.registerSlotRequest(slotRequest);

    SlotManagerImpl#registerSlotRequest()
    ——0 》SlotManagerImpl#internalRequestSlot()

OptionalConsumer.of(findMatchingSlot(resourceProfile))
.ifPresent(
101:) taskManagerSlot -> allocateSlot(taskManagerSlot, pendingSlotRequest))
.ifNotPresent(() -> fulfillPendingSlotRequestWithPendingTaskManagerSlot(pendingSlotRequest));

——1. There is a free slot "slot managerimpl#findmatchingslot"
  ——2 "fullpendingslotrequestwithpendingtaskmanagerslot"

pendingTaskManagerSlotOptional = allocateResource(resourceProfile);
——》 SlotManagerImpl# allocateResource();
if(!resourceActions.allocateResource(defaultWorkerResourceSpec))

ResourceManager#ResourceActionsImpl#allocateResource()
//There are two implementations. The first standalone resource manager directly returns false
//The second startnewworker is to apply for a new yarncontainer
return startNewWorker(workerResourceSpec);

Return to the above 101:) taskmanagerslot – > allocateslot
—— 》 SlotManagerImpl#allocateSlot() 1:18:00

*Note: apply for slot
CompletableFuture<Acknowledge> requestFuture = gateway
.requestSlot(slotId, pendingSlotRequest.getJobId(),

102:) TaskExecutor#requestSlot()

allocateSlot(slotId, jobId, allocationId, resourceProfile);
——》 TaskExecutor#allocateSlot
if(taskSlotTable.allocateSlot(slotId.getSlotNumber(), jobId

TaskSlotTableImpl#allocateSlot()
slots.add(allocationId);
▲▲▲▲▲ go back to the above 102:) line method ▲▲▲▲▲▲▲▲▲▲

       offerSlotsToJobManager(jobId);

——》 TaskExecutor#offerSlotsToJobManager
—— 》 TaskExecutor# internalOfferSlotsToJobManager

Flink source code 4-Slot allocation and task execution

image.png

Flink source code 4-Slot allocation and task execution

image.png

acceptedSlotsFuture .whenCompleteAsync(handleAcceptedSlotOffers
——》 TaskExecutor#handleAcceptedSlotOffers()
▲▲▲▲▲ back up – 4 ▲▲ 1:48
——4 》DefaultExecutionSlotAllocator#allocateSlotsFor()

Flink source code 4-Slot allocation and task execution

image.png

▲▲▲▲▲ all the way back to the top – 5 ▲▲
—— 5 》DefaultScheduler#allocateSlotsAndDeploy()
*Note: deployment run
*1. Apply to the slot
*2. Build the handler
*3. Execute deployment
*/
waitForAllSlotsAndDeploy(deploymentHandles); 1:52
▲▲▲▲▲ so far, the slot has been obtained. Next, you will enter task deployment and submission ▲▲ 1:48

4.3. Task deployment and submission 2:02~

entrance:
—— 5 》DefaultScheduler#waitForAllSlotsAndDeploy()

Flink source code 4-Slot allocation and task execution

image.png

—— 》DefaultScheduler#deployAll()

  • Note: deploy through deployorhandleerror
    *Of course, an error may be reported when deploying a task!
    */ FutureUtils.assertNoException(slotAssigned.handle(deployOrHandleError(deploymentHandle)));

—— 》DefaultScheduler#deployOrHandleError()

  • Note: deploy the task (determine the task according to the executionvertexid at that time)
    */
    deployTaskSafe(executionVertexId);
    —— 》DefaultScheduler#deployTaskSafe()

    Flink source code 4-Slot allocation and task execution

    image.png
    /*************************************************
           Note: get executionvertex according to executionvertex ID
         */
        final ExecutionVertex executionVertex = getExecutionVertex(executionVertexId);
    
        /*************************************************
           Note: a task executes an executionvertex
         *  executionVertexOperations = DefaultExecutionVertexOperations
         */
        executionVertexOperations.deploy(executionVertex);
                                                                          ↓
               DefaultExecutionVertexOperations#deploy();
      —— 》ExecutionVertex#deploy
             *Note: call the deploy () method of execution to deploy
     */
    currentExecution.deploy();
          —— 》Execution#deploy();
                                            ▼
Flink source code 4-Slot allocation and task execution

image.png

CompletableFuture.supplyAsync(() -> taskManagerGateway.submitTask
(deployment, rpcTimeout), executor).
thenCompose(Function.identity())

—— 》RpcTaskManagerGateway#submitTask()

Flink source code 4-Slot allocation and task execution

*Note: taskexecutor
     *Submit to the taskexecutor of the corresponding slot node to execute the executionvertex. In fact, it has become: task
     *This is the end of all the details about how the client submits a job and finally becomes a physical execution diagram of a distributed task.
     *After that, I went to a node taskexecutor in taskmanager 
            *To execute a task 
     */
    return taskExecutorGateway.submitTask(tdd, jobMasterId, timeout);
                                                            ↓

—— 》TaskExecutor#submitTask()

*Note: submit a task for taskmanager to start
*The first parameter: task deployment descriptor contains all the information needed to start the current task
*Note: build task
*An execution thread is initialized internally. A task is thread level execution granularity
*/
Task task = new Task(jobInformation, taskInformation, tdd.getExecutionAttemptId(),

——》Task constructor ()
                                        ▼  
              *Note: initialize resultpartitionerwriter
     */
    // produced intermediate result partitions
    final ResultPartitionWriter[] resultPartitionWriters = shuffleEnvironment
           ..............
           *Note: initialize inputgate
     */ shuffleEnvironment.createInputGates(taskShuffleContext, this, inputGateDeploymentDescriptors)
        .toArray(new IndexedInputGate[0]);
            .............
            *Note: the thread that executes the task but does not start 
     *Go to the run () method of the task
     */
    // finally, create the executing thread, but do not start it
    executingThread = new Thread(TASK_THREADS_GROUP, this, taskNameWithSubtask);
             .............

Flink source code 4-Slot allocation and task execution

image.png

*Note: register a task
*/
taskAdded = taskSlotTable.addTask(task);
…………….
*Note: if the registration is successful, the task is run through a thread
*/
task.startTaskThread();
——》Task # startTaskThread

*Note: this thread will be initialized when the task object is created
*After conversion, the final result is to call the run () method of the current class
*/
executingThread.start();
—— 》Task #run()
——aa ★★》Task #dorun()

Flink source code 4-Slot allocation and task execution

image.png
a:)      setupPartitionsAndGates(consumableNotifyingPartitionWriters, inputGates);
     ——a 》Task # setupPartitionsAndGates()

Flink source code 4-Slot allocation and task execution

image.png

——a:1 》 partition.setup();

ResultPartition# setup();
partitionManager.registerResultPartition(this);

Flink source code 4-Slot allocation and task execution

image.png
——a: 2. Startup
                                        // we are requesting partitions
                   for(InputGate gate : inputGates) {
                             gate.setup();
                                                  ↓ 
                               ——》 SingleInputGate#.setup();
                                             ▼

Flink source code 4-Slot allocation and task execution

BufferPool bufferPool = bufferPoolFactory.get();
setBufferPool(bufferPool);
▲ back to the top – AA “task #dorun ▲
——aa 》Task #dorun ()
…………….
b: ) * Note: build an environment object
*/

Flink source code 4-Slot allocation and task execution

Environment env = new RuntimeEnvironment(jobId, vertexId,
c:) invokable = loadAndInstantiateInvokable(userCodeClassLoader,

Flink source code 4-Slot allocation and task execution

d:) invokable.invoke();

StreamTask#invoke();

… beforeInvoke();
…★runMailboxLoop();
…afterInvoke();

Recommended Today

Apache sqoop

Source: dark horse big data 1.png From the standpoint of Apache, data flow can be divided into data import and export: Import: data import. RDBMS—–>Hadoop Export: data export. Hadoop—->RDBMS 1.2 sqoop installation The prerequisite for installing sqoop is that you already have a Java and Hadoop environment. Latest stable version: 1.4.6 Download the sqoop installation […]