Kafka source code analysis: the running process of the server side

Time:2021-10-13

Summary:The server side of Kafka network module introduces the process of starting, receiving and processing requests on the server side.

This article is shared from Huawei cloud community《Kafka network module – server side》Original author: middleware brother.

Socketserver is a module used by Kafka server to process requests. It is created, initialized and started during Kafka startup.

Socketserver startup process:
Kafka source code analysis: the running process of the server side

  • Initialize acceptors in the order of endpoints. Each endpoint corresponds to an acceptor. Create a processor for each acceptor (the number is determined by the num.network.threads configuration item) and start the acceptor. After the acceptor is started, it will listen to the connection through the selector and hand over the newly established connection to the processor for processing (poll and select the processor)
  • Start all processors
    Acceptor starts and listens to the connection process:
    Kafka source code analysis: the running process of the server side
  • After the acceptor is started, it will create a serversocketchannel, listen on the endpoint corresponding to the acceptor, and register the op on the selector_ Accept, and then enter the dead cycle. In each cycle, obtain the ready key (i.e. the previously registered serversocketchannel) through the selector, indicating that a connection has arrived. Then create a socketchannel corresponding to the connection through accept(), poll and select one from the processors in charge of the acceptor, and hand the socketchannel to the selected processor for processing, The connection will be handed over to the processor.
  • The acceptor gives the connection to the processor for processing, which is to add the socketchannel to the processor’s connection queue newConnection, from which the processor will continuously obtain and process in the run method.
  • After the processor obtains the socketchannel from the newConnection, it registers the op on the selector_ Read and create the corresponding kafkachannel.

The process of receiving and processing requests on the server side:

  • Processor receives op_ After the read event is ready, check and try to complete the SSL handshake and SASL verification (the handshake may not be completed at this time, so after the processor receives the op_read event is ready, check and ensure that the handshake has been completed. Refer to section 9.4 for SSL / SASL)
  • After the SSL handshake and SASL verification are completed, read the data from the channel, construct the networkreceive object, and join the queue stagedreceives
  • Take out the stagedreceipts team head element (remove) and add completedreceipts
  • Take out (not remove) the elements in completedreceipts, construct a request object, add a requestqueue, and remove the Op_ Register the event of read, and set the corresponding kafkachannel to muted, and then to muted_ AND_ RESPONSE_ PENDING
  • The kafkarequesthandler takes out the element from the requestqueue (removes it) and gives it to the kafkaapi module to process the request
  • After kafkaapi processes the request, it puts the response into the responsequeue and inflightresponses of the corresponding processor and wakes up its selector
  • The processor takes out (removes) the response from the responsequeue. If the response needs to be sent back to the client, it assigns the send of the response to kafkachannel and registers the Op_ Write event
  • When the channel is ready for writing, write send to the write buffer of the channel. When send is finished, remove the Op_ Register the write event and add send to completedsends
  • Remove the corresponding response from inflightresponses, execute the response callback, set kafkachannel to muted, and then set muted to not_ Muted and add OP again_ Read event registration

Click focus to learn about Huawei cloud’s new technologies for the first time~