Kafka growth 4: principle of metadata pull source code of producer (Part 2)

Time:2021-11-22

Kafka growth 4: principle of metadata pull source code of producer (Part 2)

At the end of the previous section, we concluded that when initializing kafkaproducer, we did not pull metadata, but created the selector component, started the sender thread, blocked the select and waited for the request response. Since no request has been sent, metadata is not really pulled during initialization.

When the send method is called for the first time, it will wake up the select() blocked before waking up the selector and enter the second while loop to send the metadata pull request. It will wait for 60s through the obejct.wait mechanism. After the metadata pull from the broker is successful, it will continue to execute the request for the real production message. Otherwise, it will report a metadata pull timeout exception.

As shown below:

Kafka growth 4: principle of metadata pull source code of producer (Part 2)

After waking up the selector’s select, it should enter the second while loop. How does the second while loop send a request to pull metadata and notify all() wake up after success?

Let’s have a look today.

Second while loop – start triggering metadata pull

Wake up the blocked select. Do you remember the logic after blocking?

After wake-up, the int number of readkeys returned by nioselector. Select() will be used. If it is greater than 0, for example, some operations of pollselectionkeys will be performed. Because it is directly wakeup(), the actual readkeys is 0, so the poll method will be returned directly and will not perform the processing of pollselectionkeys.

Moreover, after the pollmethod of the selector returns, because pollselectionkeys is not executed, the subsequent methods handlecompletedsends, handlecompletedreceives, handledisconnections, handleconnections and handletimedoutrequests are not executed. (you can try the breakpoint yourself and you’ll find it.)

The above logic execution is completed, that is, the first cycle will end and the second cycle will be restarted. The overall process is as shown in the figure below: (mainly the process of gray remarks)

Kafka growth 4: principle of metadata pull source code of producer (Part 2)

Reason for execution of maybeupdate in the second cycle

Now that you enter the second loop, the methods starting with maybeupdate (), poll (), and handle will be re executed.

Do you remember the core context of maybeupdate? It mainly determines whether the metadata timeout is 0 according to three times. The code is as follows:

        @Override
        public long maybeUpdate(long now) {
            // should we update our metadata?
            long timeToNextMetadataUpdate = metadata.timeToNextUpdate(now);
            long timeToNextReconnectAttempt = Math.max(this.lastNoNodeAvailableMs + metadata.refreshBackoff() - now, 0);
            long waitForMetadataFetch = this.metadataFetchInProgress ? Integer.MAX_VALUE : 0;
            // if there is no node available to connect, back off refreshing metadata
            long metadataTimeout = Math.max(Math.max(timeToNextMetadataUpdate, timeToNextReconnectAttempt),
                    waitForMetadataFetch);

            if (metadataTimeout == 0) {
                // Beware that the behavior of this method and the computation of timeouts for poll() are
                // highly dependent on the behavior of leastLoadedNode.
                Node node = leastLoadedNode(now);
                maybeUpdate(now, node);
            }

            return metadataTimeout;
        }
        public synchronized long timeToNextUpdate(long nowMs) {
            long timeToExpire = needUpdate ? 0 : Math.max(this.lastSuccessfulRefreshMs + this.metadataExpireMs - nowMs, 0);
            long timeToAllowUpdate = this.lastRefreshMs + this.refreshBackoffMs - nowMs;
            return Math.max(timeToExpire, timeToAllowUpdate);
        }

In the first cycle, the metadata timeout gets a non-zero value, while in the second cycle, the value has actually become 0.

Because we executed metadata. Requestupdate() before sender. Wakeyup() in the previous section;

This line of code changes the needupdate flag to true. Timetonextmetadataupdate among the three time values that determine the metadata timeout will also become 0, that is, timetonextmetadataupdate, timetonextreconnectattempt and waitformetadatafetch will all become 0. Naturally, the metadata timeout is also 0.

As shown in the figure below:

Kafka growth 4: principle of metadata pull source code of producer (Part 2)

Therefore, the logic of maybeupdae will be executed in the second loop. Instead of doing nothing for the first time before.

If metadatatimeout = 0, two methods are executed:

1) As can be seen from the annotation, leastloadednode is actually selecting a broker node and pulling metadata from it. The selection criteria must be the best connected broker and nodes with less data to be sent. We will study these logic carefully.

2) The maybeupdate method is actually very critical. It is mainly the logic to establish a connection or initiate a metadata pull request

So here we mainly look at the main logic of maybeupdate:

/**
 * Add a metadata request to the list of sends if we can make one
 */
private void maybeUpdate(long now, Node node) {
    if (node == null) {
        log.debug("Give up sending metadata request since no node is available");
        // mark the timestamp for no node available to connect
        this.lastNoNodeAvailableMs = now;
        return;
    }
    String nodeConnectionId = node.idString();

    if (canSendRequest(nodeConnectionId)) {
        this.metadataFetchInProgress = true;
        MetadataRequest metadataRequest;
        if (metadata.needMetadataForAllTopics())
            metadataRequest = MetadataRequest.allTopics();
        else
            metadataRequest = new MetadataRequest(new ArrayList<>(metadata.topics()));
        ClientRequest clientRequest = request(now, nodeConnectionId, metadataRequest);
        log.debug("Sending metadata request {} to node {}", metadataRequest, node.id());
        doSend(clientRequest, now);
    } else if (connectionStates.canConnect(nodeConnectionId, now)) {
        // we don't have a connection to this node right now, make one
        log.debug("Initialize connection to node {} for sending metadata request", node.id());
        initiateConnect(node, now);
        // If initiateConnect failed immediately, this node will be put into blackout and we
        // should allow immediately retrying in case there is another candidate node. If it
        // is still connecting, the worst case is that we end up setting a longer timeout
        // on the next round and then wait for the response.
    } else { // connected, but can't send more OR connecting
        // In either case, we just need to wait for a network event to let us know the selected
        // connection might be usable again.
        this.lastNoNodeAvailableMs = now;
    }
}

The above context is relatively simple, mainly an if else.

If you can send a metadata pull request, you can call the dosend () method

Else if the request cannot be sent, it indicates that the connection has not been established and needs to be initialized. Call the initateconnection () method

The whole process is shown in the figure below:

Kafka growth 4: principle of metadata pull source code of producer (Part 2)

How to establish a connection based on NiO before pulling metadata?

Maybeupdate will use the clusterconnectionstates component according to the cansendrequest and canConnect methods to determine whether a connection has been established with the broker. As mentioned in the second section, this component is a component for networklclient to record the connection with the broker. The main codes are as follows:

NetworklClient.java;
private boolean canSendRequest(String node) {
    return connectionStates.isConnected(node) && selector.isChannelReady(node) && inFlightRequests.canSendMore(node);
}

ClusterConnectionStates
public boolean canConnect(String id, long now) {
    NodeConnectionState state = nodeState.get(id);
    if (state == null)
        return true;
    else
        return state.state == ConnectionState.DISCONNECTED && now - state.lastConnectAttemptMs >= this.reconnectBackoffMs;
}

In addition to the connection status, we also made other additional logical judgments, which are very detailed judgments. We don’t need to study here.

I mainly know that no connection has been established with the broker, so I will go to the initiateconnect () method to establish a connection. Let’s have a look.

    /**
     * Initiate a connection to the given node
     */
    private void initiateConnect(Node node, long now) {
        String nodeConnectionId = node.idString();
        try {
            log.debug("Initiating connection to node {} at {}:{}.", node.id(), node.host(), node.port());
            this.connectionStates.connecting(nodeConnectionId, now);
            selector.connect(nodeConnectionId,
                             new InetSocketAddress(node.host(), node.port()),
                             this.socketSendBuffer,
                             this.socketReceiveBuffer);
        } catch (IOException e) {
            /* attempt failed, we'll try again after the backoff */
            connectionStates.disconnected(nodeConnectionId, now);
            /* maybe the problem is our metadata, update it */
            metadataUpdater.requestUpdate();
            log.debug("Error connecting to node {} at {}:{}:", node.id(), node.host(), node.port(), e);
        }
    }

    public void connecting(String id, long now) {
        nodeState.put(id, new NodeConnectionState(ConnectionState.CONNECTING, now));
    }

The core context is very simple, just two sentences:

1) Connectionstates. Connecting() records the status as being connected. There is nothing to say about this.

2) Selector. Connect () executes the connect method through the selector encapsulated by Kafka, which is the key to establishing a connection.

Kafka growth 4: principle of metadata pull source code of producer (Part 2)

The connect method of selector is very important. Let’s see what its code is doing:

org.apache.kafka.common.network.Selector.java
@Override
public void connect(String id, InetSocketAddress address, int sendBufferSize, int receiveBufferSize) throws IOException {
    if (this.channels.containsKey(id))
        throw new IllegalStateException("There is already a connection for id " + id);

    SocketChannel socketChannel = SocketChannel.open();
    socketChannel.configureBlocking(false);
    Socket socket = socketChannel.socket();
    socket.setKeepAlive(true);
    if (sendBufferSize != Selectable.USE_DEFAULT_BUFFER_SIZE)
        socket.setSendBufferSize(sendBufferSize);
    if (receiveBufferSize != Selectable.USE_DEFAULT_BUFFER_SIZE)
        socket.setReceiveBufferSize(receiveBufferSize);
    socket.setTcpNoDelay(true);
    boolean connected;
    try {
        connected = socketChannel.connect(address);
    } catch (UnresolvedAddressException e) {
        socketChannel.close();
        throw new IOException("Can't resolve address: " + address, e);
    } catch (IOException e) {
        socketChannel.close();
        throw e;
    }
    SelectionKey key = socketChannel.register(nioSelector, SelectionKey.OP_CONNECT);
    KafkaChannel channel = channelBuilder.buildChannel(id, key, maxReceiveSize);
    key.attach(channel);
    this.channels.put(id, channel);

    if (connected) {
        // OP_CONNECT won't trigger for immediately connected channels
        log.debug("Immediately connected to node {}", channel.id());
        immediatelyConnectedKeys.add(key);
        key.interestOps(0);
    }
}

PlaintextChannelBuilder.java
    public KafkaChannel buildChannel(String id, SelectionKey key, int maxReceiveSize) throws KafkaException {
        KafkaChannel channel = null;
        try {
            PlaintextTransportLayer transportLayer = new PlaintextTransportLayer(key);
            Authenticator authenticator = new DefaultAuthenticator();
            authenticator.configure(transportLayer, this.principalBuilder, this.configs);
            channel = new KafkaChannel(id, transportLayer, authenticator, maxReceiveSize);
        } catch (Exception e) {
            log.warn("Failed to create channel due to ", e);
            throw new KafkaException(e);
        }
        return channel;
    }

The core context of the connect () method above is mainly:

1) Socketchannel. Open() creates the socketchannel of NiO

2) Set some socket parameters and initiate a connect connection through socketchannel (this is a common operation of NiO. You can search a HelloWorld of Java Native NiO by yourself, or pay attention to NiO’s growth records later, and you will certainly be familiar with this.)

3) The socketchannel registers the register with the selector and indicates that it pays attention to the connection establishment request selectionkey.op_ Connect to associate the corresponding socketchannel through the selectionkey

4) Buildchannel encapsulates the above socketchannel, selector and selectionkey into kafkachannel. What’s more, it also encapsulates an object called transportlayer. And through key. Attach (channel); Bind kafkachannel to selcetionkey

5) The kafkachannel is cached through map < string, kafkachannel > channels

The whole logic is shown in the figure below:

Kafka growth 4: principle of metadata pull source code of producer (Part 2)

Here, the initateconnect () method is completed, the mayupdate method returns, and then enters the next step of the second while loop, selector. Poll();

The following pink lines are shown:

Kafka growth 4: principle of metadata pull source code of producer (Part 2)

andSelector.poll(); We knew before that its bottom layer would call nioselector’s select() to block and wait for requests of concern.

If you are familiar with NiO, you know that if the previously sent connect connection is established successfully, the registered selectionkey has the corresponding concerned event selectionkey.op_ Connect, it will jump out of the block.

This process is shown in the following figure:

Kafka growth 4: principle of metadata pull source code of producer (Part 2)

From the above figure, the pollselectionkeys() method will be executed next:

  private void pollSelectionKeys(Iterable<SelectionKey> selectionKeys, boolean isImmediatelyConnected) {
        Iterator<SelectionKey> iterator = selectionKeys.iterator();
        while (iterator.hasNext()) {
            SelectionKey key = iterator.next();
            iterator.remove();
            KafkaChannel channel = channel(key);

            // register all per-connection metrics at once
            sensors.maybeRegisterConnectionMetrics(channel.id());
            lruConnections.put(channel.id(), currentTimeNanos);

            try {

                /* complete any connections that have finished their handshake (either normally or immediately) */
                if (isImmediatelyConnected || key.isConnectable()) {
                    if (channel.finishConnect()) {
                        this.connected.add(channel.id());
                        this.sensors.connectionCreated.record();
                    } else
                        continue;
                }

                /* if channel is not ready finish prepare */
                if (channel.isConnected() && !channel.ready())
                    channel.prepare();

                /* if channel is ready read from any connections that have readable data */
                if (channel.ready() && key.isReadable() && !hasStagedReceive(channel)) {
                    NetworkReceive networkReceive;
                    while ((networkReceive = channel.read()) != null)
                        addToStagedReceives(channel, networkReceive);
                }

                /* if channel is ready write to any sockets that have space in their buffer and for which we have data */
                if (channel.ready() && key.isWritable()) {
                    Send send = channel.write();
                    if (send != null) {
                        this.completedSends.add(send);
                        this.sensors.recordBytesSent(channel.id(), send.size());
                    }
                }

                /* cancel any defunct sockets */
                if (!key.isValid()) {
                    close(channel);
                    this.disconnected.add(channel.id());
                }

            } catch (Exception e) {
                String desc = channel.socketDescription();
                if (e instanceof IOException)
                    log.debug("Connection with {} disconnected", desc, e);
                else
                    log.warn("Unexpected error from {}; closing connection", desc, e);
                close(channel);
                this.disconnected.add(channel.id());
            }
        }
    }

The logic of this method doesn’t seem very clear. It doesn’t matter. We can debug:

Kafka growth 4: principle of metadata pull source code of producer (Part 2)

You will find that this method is mainly traversing the selectionkeys collection with response. Since only one selectioinkey has been registered previously, which is related to the connect type request, we only traversed one here.

Then you will find that the core of the while loop executes the following sentence:

private final List<String> connected;
if (channel.finishConnect()) {
     this.connected.add(channel.id());
     this.sensors.connectionCreated.record();
 } else
     continue;
 }
KafkaChannel.java
public boolean finishConnect() throws IOException {
    return transportLayer.finishConnect();
}

PlaintextTransportLayer.java
public boolean finishConnect() throws IOException {
    boolean connected = socketChannel.finishConnect();
    if (connected)
    key.interestOps(key.interestOps() & ~SelectionKey.OP_CONNECT | SelectionKey.OP_READ);
    return connected;
}

The core context of the above if else code is:

First, judge whether the connection is established through channel. Finishconnect(),The underlying essence is NiO’s socketchannel. Finishconnect();, If the connection is established and the selectionkey is modified, the main operations concerned are selectionkey.op_ Read type, no longer op_ Connect type. After that, the connected channelid is cached in a list < string > connected set.

The whole is shown in the figure below:

Kafka growth 4: principle of metadata pull source code of producer (Part 2)

The poll method is completed, and the second step of the second while loop is completed. Finally, the while loop will execute a pile of handle methods:

handleCompletedSends(responses, updatedNow);
handleCompletedReceives(responses, updatedNow);
handleDisconnections(responses, updatedNow);
handleConnections();
handleTimedOutRequests(responses, updatedNow);

In fact, you can guess which method will be executed after the connection is established? Yes, handleconnections () will be executed. Other methods cannot be executed at all. They all return directly.

What logic does handleconnections perform?

NetWorkClient.java
   private void handleConnections() {
        for (String node : this.selector.connected()) {
            log.debug("Completed connection to node {}", node);
            this.connectionStates.connected(node);
        }
    }
Selector.java
public List<String> connected() {
    return this.connected;
}
    
ClusterConnectionStates.java
    public void connected(String id) {
        NodeConnectionState nodeState = nodeState(id);
        nodeState.state = ConnectionState.CONNECTED;
    }

In fact, it traverses the node (broker) that has established the channel and records the connection status of the node as connected. (do you remember that the status was connecting when maybeupdate executed initiateconnect() before?)

At this point, the second while loop is completed, and the second loop is the same. The core executes the three steps, starting with maybeupdate() – > poll() – > handle. The main thing to do is to establish a connection with the broker through NiO.

In the previous first loop, the methods starting with maybeupdate () – > poll () – > handle are mainly blocked by the poll () method, and nothing else is done.

The overall process of the second cycle is summarized in the following large figure:

Kafka growth 4: principle of metadata pull source code of producer (Part 2)

After this second cycle of logic, do you have a more familiar understanding of producer?

After that, the third while loop or even more will be executed again, which is the same logic as the method starting with maybeupdate() – > poll() – > handle.

Send metadata pull request

The sender executes the third cycle again. The first step must be to execute maybeupdate (). This time, maybeupdate (), the connection has been established, and another section of logic, dosend () method, will be executed to really pull metadata. Let’s have a look!

         /**
         * Add a metadata request to the list of sends if we can make one
         */
        private void maybeUpdate(long now, Node node) {
            if (node == null) {
                log.debug("Give up sending metadata request since no node is available");
                // mark the timestamp for no node available to connect
                this.lastNoNodeAvailableMs = now;
                return;
            }
            String nodeConnectionId = node.idString();

            if (canSendRequest(nodeConnectionId)) {
                this.metadataFetchInProgress = true;
                MetadataRequest metadataRequest;
                if (metadata.needMetadataForAllTopics())
                    metadataRequest = MetadataRequest.allTopics();
                else
                    metadataRequest = new MetadataRequest(new ArrayList<>(metadata.topics()));
                ClientRequest clientRequest = request(now, nodeConnectionId, metadataRequest);
                log.debug("Sending metadata request {} to node {}", metadataRequest, node.id());
                doSend(clientRequest, now);
            } else if (connectionStates.canConnect(nodeConnectionId, now)) {
                // we don't have a connection to this node right now, make one
                log.debug("Initialize connection to node {} for sending metadata request", node.id());
                initiateConnect(node, now);
                // If initiateConnect failed immediately, this node will be put into blackout and we
                // should allow immediately retrying in case there is another candidate node. If it
                // is still connecting, the worst case is that we end up setting a longer timeout
                // on the next round and then wait for the response.
            } else { // connected, but can't send more OR connecting
                // In either case, we just need to wait for a network event to let us know the selected
                // connection might be usable again.
                this.lastNoNodeAvailableMs = now;
            }

When maybeupdate is executed this time, it will be executed

//NetworkClient.java
private boolean canSendRequest(String node) {
    return connectionStates.isConnected(node) && selector.isChannelReady(node) && inFlightRequests.canSendMore(node);
}
//ClusterConnectionStates.java
public boolean isConnected(String id) {
    NodeConnectionState state = nodeState.get(id);
    return state != null && state.state == ConnectionState.CONNECTED;
}
//Selector.java
public boolean isChannelReady(String id) {
    KafkaChannel channel = this.channels.get(id);
    return channel != null && channel.ready();
}
//PlaintextTransportLayer.java
public boolean ready() {
    return true;
}
//InFlightRequests.java
private final Map<String, Deque<ClientRequest>> requests = new HashMap<String, Deque<ClientRequest>>();

public boolean canSendMore(String node) {
    Deque<ClientRequest> queue = requests.get(node);
    return queue == null || queue.isEmpty() ||
        (queue.peekFirst().request().completed() && queue.size() < this.maxInFlightRequestsPerConnection);
}

Through a pile of components above, the dosend method will be executed only when all three conditions are true.

Connection states. Is connected (node): it must be true, because the connection state has been recorded as connected.

Selector. Ischannelready (node): the kafkachannel created earlier is cached in the map. Channel. Ready() returns true by default,

Inflightrequests. Cansendmore (node): the requests queue is not empty and the number of queue elements is less than the default 5 of maxinflightrequestsperconnection. This configuration is sufficient.

In the second cycle, the queue root is empty, so this condition is true.

/**
*This involves a key memory structure, map < string, deque < clientrequests > > requests in infightrequests, a memory structure composed of map and two-way queue. We mentioned this component when analyzing * network. At that time, we only knew through comments: inflightrequests is a collection of requests that have been sent or are being sent but have not received a response. I don't know what to do.
*However, we can see here that before sending a request, the request will enter this memory structure for temporary storage, which is very close to the annotation expression, and is often used to judge whether there are requests to be sent.
*/

That is, after the connection has been established, the third loop will execute the dosend method logic.

As shown in the figure below:

Kafka growth 4: principle of metadata pull source code of producer (Part 2)

Then if it passes, the following logic is executed:

if (canSendRequest(nodeConnectionId)) {
    this.metadataFetchInProgress = true;
    MetadataRequest metadataRequest;
    if (metadata.needMetadataForAllTopics())
        metadataRequest = MetadataRequest.allTopics();
    else
        metadataRequest = new MetadataRequest(new ArrayList<>(metadata.topics()));
    ClientRequest clientRequest = request(now, nodeConnectionId, metadataRequest);
    log.debug("Sending metadata request {} to node {}", metadataRequest, node.id());
    doSend(clientRequest, now);
}

public RequestSend(String destination, RequestHeader header, Struct body) {
    super(destination, serialize(header, body));
    this.header = header;
    this.body = body;
}

public static ByteBuffer serialize(RequestHeader header, Struct body) {
    ByteBuffer buffer = ByteBuffer.allocate(header.sizeOf() + body.sizeOf());
    header.writeTo(buffer);
    body.writeTo(buffer);
    buffer.rewind();
    return buffer;
}

First you can see,Before dosend, the request parameters are wrapped at various levels, and the final object is serialized into ByteBuffer. (we will not study the format of ByteBuffer. We will mention it again when we study Kafka to solve the problem of sticking and unpacking.)

I won’t show you the specific details. It can be summarized as follows: metadatarequest – > requestheader + struct – = requestsend (convert serialize method to ByteBuffer) – > clientrequest

Kafka growth 4: principle of metadata pull source code of producer (Part 2)

After packing the request, the doSend method is called.

private void doSend(ClientRequest request, long now) {
    request.setSendTimeMs(now);
    this.inFlightRequests.add(request);
    selector.send(request.request());
}
//ClientRequest.java
public RequestSend request() {
    return request;
}
// Selector.java  
public void send(Send send) {
        KafkaChannel channel = channelOrFail(send.destination());
        try {
            channel.setSend(send);
        } catch (CancelledKeyException e) {
            this.failedSends.add(send.destination());
            close(channel);
        }
    }
    
    private KafkaChannel channelOrFail(String id) {
        KafkaChannel channel = this.channels.get(id);
        if (channel == null)
            throw new IllegalStateException("Attempt to retrieve channel for which there is no open connection. Connection id " + id + " existing connections " + channels.keySet());
        return channel;
    }

// KafkaChannel.java
public void setSend(Send send) {
    if (this.send != null)
        throw new IllegalStateException("Attempt to begin a send operation with prior send operation still in progress.");
    this.send = send;
    this.transportLayer.addInterestOps(SelectionKey.OP_WRITE);
}

// PlaintextTransportLayer.java
public void addInterestOps(int ops) {
    key.interestOps(key.interestOps() | ops);
}

This method is more interesting. You will find that the main context of dosend is as follows:

1) The request is temporarily stored in the inflightrequests memory structure

2) The selector obtains the previously cached kafkachannel from the map

3) Kafkachannel records the sent request data, requestsend, and adds the focus on write requests (after the previous connection is established, the focus on op_connect is cancelled and the focus on op_read is increased. Do you remember?)

The above operations are basically the routine operations of NiO. Obtain channels and set attention events. But

What about the channel.write operation? There is no data written here? thereforeThe kafkachannel method is called setsend (send send), which only sets the objects to be sent and the OPS of interest_ Just write.

The whole process is shown in the figure below:

Kafka growth 4: principle of metadata pull source code of producer (Part 2)

After the dosend method is executed, the method of metadataupdater.maybeupdate will return, and then enter the second step of the third cycle, the selector. Poll () method, and finally execute the method starting with handle. I believe you are no stranger.

The core of selector. Poll() consists of two steps:

1) Selector. Select() blocks waiting for the server to return the concerned events

2) Execute pollselectionkeys(), traverse all selectionkeys, and perform different processing according to the events concerned by the selectionkey(previously, when establishing a connection, the channelid of successful connection was recorded only according to op_connect.)

Here, because the client is concerned about the Op_ Read and op_ Write event, so when executing the loop for the third time, the selector. Select() block will jump out and execute the logic of pollselectionkeys().

Here, I directly intercepted the key logic. When selector.java executes the third while loop, the core logic of pollselectionkeys method traversing selectionkeys is as follows:

//When selector.java executes the third while loop, the pollselectionkeys method traverses the core logic of selectionkeys
if (channel.ready() && key.isWritable()) {
    Send send = channel.write();
    if (send != null) {
    this.completedSends.add(send);
    this.sensors.recordBytesSent(channel.id(), send.size());
    }
}

The core of the above is:

1) Send the request to pull metadata through channel. Write()!

2) After sending, record the successful requests sent to list < send > completedsends; in

Here we finally see channel. Write (), and finally the bottom layer writes out the previously serialized ByteBuffer through NiO’s socketchannel. Write. And the selectionkey.op will be removed after sending_ Write’s attention, no longer write data.

//KafkaChannel.java
public Send write() throws IOException {
    Send result = null;
    if (send != null && send(send)) {
        result = send;
        send = null;
    }
    return result;
}
private boolean send(Send send) throws IOException {
    send.writeTo(transportLayer);
    if (send.completed())
        transportLayer.removeInterestOps(SelectionKey.OP_WRITE);

    return send.completed();
}

//PlaintextTransportLayer.java
    public long write(ByteBuffer[] srcs) throws IOException {
        return socketChannel.write(srcs);
    }

The data is finally sent. The whole process can be summarized as shown in the figure below:

Kafka growth 4: principle of metadata pull source code of producer (Part 2)

In the third execution of the while loop, the maybeupdat () and poll () methods have been executed. Finally, the method starting with handle is executed.

   handleCompletedSends(responses, updatedNow);
   handleCompletedReceives(responses, updatedNow);
   handleDisconnections(responses, updatedNow);
   handleConnections();
   handleTimedOutRequests(responses, updatedNow);

The positive execution of these methods is the handlecompletedsends method.

    private void handleCompletedSends(List<ClientResponse> responses, long now) {
        // if no response is expected then when the send is completed, return it
        for (Send send : this.selector.completedSends()) {
            ClientRequest request = this.inFlightRequests.lastSent(send.destination());
            if (!request.expectResponse()) {
                this.inFlightRequests.completeLastSent(send.destination());
                responses.add(new ClientResponse(request, now, false, null));
            }
        }
    }

Here, the sent request will be taken from the previously temporarily stored inflightrequests. Request. Expectresponse () is true by default, so the if condition will not be established. Handlecompletedsends is equivalent to doing nothing. From the annotation, this method is to handle: “if there is no response, return it when the sending is completed.” that is to say, this logic is not the key logic. Let’s focus on the big and let go of the small and skip it.

As you improve your experience in reading source code, you will often find that this is not the core logic. At this time, you must learn to choose and learn to focus on the big and let go of the small.

In that case, the method starting with handle is actually completed. It’s time to enter the fourth while loop

Receive the pulled metadata and wake up the kafkaproducer.send method

In fact, you can think of what the fourth while loop will do. Of course, the metadata returned by the receiving server wakes up the kafkaproducer. Send method of the previous wait. With the experience of the previous three while loops, let’s find the core logic and see how it works. Let’s have a quick look together!

1) Maybeupdate executed first:

In the fourth while loop, the waitformetadatafetch accountant in maybeupdate calculates a non-zero value, resulting in that maybeupdate will not execute anything like the first loop

long timeToNextMetadataUpdate = metadata.timeToNextUpdate(now);
long timeToNextReconnectAttempt = Math.max(this.lastNoNodeAvailableMs + metadata.refreshBackoff() - now, 0);
long waitForMetadataFetch = this.metadataFetchInProgress ? Integer.MAX_VALUE : 0;

2) Then execute selector. Poll(),It will block the select () method, but when the server returns data, we only focus on op on the selectionkey_ Read, so it will jump out of the block and execute the logic in the corresponding pollselectionkeys

/* if channel is ready read from any connections that have readable data */
if (channel.ready() && key.isReadable() && !hasStagedReceive(channel)) {
    NetworkReceive networkReceive;
    while ((networkReceive = channel.read()) != null)
    addToStagedReceives(channel, networkReceive);
}

Map<KafkaChannel, Deque<NetworkReceive>> stagedReceives;
/**
   * adds a receive to staged receives
  */
private void addToStagedReceives(KafkaChannel channel, NetworkReceive receive) {
    if (!stagedReceives.containsKey(channel))
        stagedReceives.put(channel, new ArrayDeque<NetworkReceive>());

    Deque<NetworkReceive> deque = stagedReceives.get(channel);
    deque.add(receive);
}

In fact, this logic is to accept ByteBuffer and put it into networkreceive object. The underlying essence calls socketchannel’s read() method, which is just a common NiO operation. It is similar to sending data. At the bottom, I won’t show you here. I believe you can see it.

In addition to accepting data into the networkreceive object, the accepted data is also temporarily stored in a double ended queue deque. Map<KafkaChannel, Deque<NetworkReceive>> stagedReceives;

3) After executing the poll method, it’s time to execute the method starting with handle, the handlecompletedreceives() method is executed this time:

    /**
     * Handle any completed receives and update the response list with the responses received.
     *
     * @param responses The list of responses to update
     * @param now The current time
     */
    private void handleCompletedReceives(List<ClientResponse> responses, long now) {
        for (NetworkReceive receive : this.selector.completedReceives()) {
            String source = receive.source();
            ClientRequest req = inFlightRequests.completeNext(source);
            Struct body = parseResponse(receive.payload(), req.request().header());
            if (!metadataUpdater.maybeHandleCompletedReceive(req, now, body))
                responses.add(new ClientResponse(req, now, false, body));
        }
    }

       public boolean maybeHandleCompletedReceive(ClientRequest req, long now, Struct body) {
            short apiKey = req.request().header().apiKey();
            if (apiKey == ApiKeys.METADATA.id && req.isInitiatedByNetworkClient()) {
                handleResponse(req.request().header(), body, now);
                return true;
            }
            return false;
        }

And this method is simple:

1) According to the previously staged request clientrequest, find the corresponding response from networkreceive, and then perform a series of parsing. The bufferbuffer is a struct object.

2) Execute the maybehandlecompletedreceive method of defaultmetadataupdater

What does the maybehandlecompletedreceive method of defaultmetadataupdater do?

      
  private void handleResponse(RequestHeader header, Struct body, long now) {
            this.metadataFetchInProgress = false;
            MetadataResponse response = new MetadataResponse(body);
            Cluster cluster = response.cluster();
            // check if any topics metadata failed to get updated
            Map<String, Errors> errors = response.errors();
            if (!errors.isEmpty())
                log.warn("Error while fetching metadata with correlation id {} : {}", header.correlationId(), errors);

            // don't update the cluster if there are no valid nodes...the topic we want may still be in the process of being
            // created which means we will get errors and no nodes until it exists
            if (cluster.nodes().size() > 0) {
                this.metadata.update(cluster, now);
            } else {
                log.trace("Ignoring empty metadata response with correlation id {}.", header.correlationId());
                this.metadata.failedUpdate(now);
            }
  }

    public synchronized void update(Cluster cluster, long now) {
        this.needUpdate = false;
        this.lastRefreshMs = now;
        this.lastSuccessfulRefreshMs = now;
        this.version += 1;

        for (Listener listener: listeners)
            listener.onMetadataUpdate(cluster);

        // Do this after notifying listeners as subscribed topics' list can be changed by listeners
        this.cluster = this.needMetadataForAllTopics ? getClusterForCurrentTopics(cluster) : cluster;

        notifyAll();
        log.debug("Updated cluster metadata version {} to {}", this.version, this.cluster);
    }

The core context of the above code is as follows:

1) Maybehandlecompletedreceive will convert the struts object to a metadataresponse and then to a cluster object

2) Finally, according to the nodes information in the cluster, if it is greater than 0, execute the metadata. Update() method and execute some listener callbacks. Finally, the key is that metadata. Notifyall() wakes up kafkaproducer. Send() that was blocked and waiting before

The whole process is summarized as follows:

Kafka growth 4: principle of metadata pull source code of producer (Part 2)

summary

At this point, we have finished our research on the principle of metadata pull source code. In fact, after your research, you will find that we have executed the core while loop four times. With the repeated process, it seems that the source code principle is not much difficult.

In fact, that’s it. Many times,Do simple things repeatedly. As long as you think more and think more, you will find the law and slowly understand the essence of things.This idea is much more important than our study of the source code principle of Kafka pulling metadata.

In addition, metadata pulling is not complicated. It is nothing more than connection establishment, request sending and request response. Kafka uses some interesting mechanisms, such as wait + notifyAll mechanism and NiO.

What I have been drawing for you before is a detailed logic diagram. You can draw a diagram and summarize its logic. If you can draw and explain to others, you really understand.

But in fact, Kafka has done a lot of thinking in this process. You can think about some of its highlights and advantages, just like after the previous study of ZK election principles. Your thoughts and ideas are far greater than knowledge itself. You can leave a message to me in the comments. Let’s discuss it together!

Although Kafka’s growth orientation tends to improve the technical depth, if you have skillfully used NiO, you certainly have a good understanding of NiO knowledge in the metadata pulling process.

If you don’t know much about NiO, you can baidu your basic knowledge of NiO. Or pay attention to the NiO white start camp I later published

See you next time!

This article is composed of blog one article multi posting platformOpenWriterelease!