Zookeeper’s twelfth company asked, can you do it?

Time:2020-12-12

preface

The 12th company of zookeeper, a big factory in the first line, asked, can you stand it?

This article has been included in GitHub

https://github.com/whx123/Jav…

1. Interviewer: have you ever used zookeeper in your work? Do you know what it is and what is its use?

I of chicken

  • Some of them have used zookeeper asDubbo’s registry, using zookeeperDistributed lock
  • Zookeeper, it is an open sourceDistributed coordination service, which is the manager of a cluster and provides users with an easy-to-use interface.
  • It can implement distributed service, such as call / master, publish / subscribe, distributed service, such as call / queue, distributed serviceEtc
  • Zookeeper’spurpose: naming service, configuration management, cluster management, distributed lock, queue management

Isn’t it the same meaning with function? Give me a look, let me experience

Zookeeper's twelfth company asked, can you do it?

2. Interviewer: what is naming service, configuration management and cluster management

I’m a chicken (fortunately I’ve done the interview), I’m not afraid

Zookeeper's twelfth company asked, can you do it?

    • The naming service is

    Naming services are defined asDesignated nameTo get the resource or service address. Zookeeper can create aGlobally unique pathThis path can be used as a name. Named entities can beThe address of a machine, service, or remote object in a clusterEtc. Some service address lists in distributed service frameworks (RPC, RMI), through the use of named services, client applications can obtain the entity, service address and provider information of resources according to specific names.

    • Configuration management:

    In actual project development, we often use. Properties or XML, and need to configure a lot of information, such as database connection information, FPS address port and so on. Because your program is generally distributed on different machines (if you are a stand-alone application, when I did not say), if you put these configuration information of the programZnode nodes stored in ZKNext, when you want to modify the configuration, that is, when the znode will change, you can change the content of a directory node in ZKThe watcher notifies each clientTo change the configuration.

    • Cluster management

    Cluster management includes cluster monitoring and cluster control, which is actually monitoring the status of cluster machines, eliminating machines and joining machines. Zookeeper can facilitate the management of cluster machines. It can monitor the changes of znode nodes in real time. Once a machine is found hanging, the machine will be disconnected from ZK, and the temporary directory node will be deleted, and all other machines will be informed. The addition of new machines is similar to that of soy sauce purple. All machines have been informed that a new brother directory has been added.

    Interviewer: you mentioned the znode node. How many types of znode do you know? What is zookeeper’s data model like?

    Chicken’s me (let me think about it first)

    Zookeeper's twelfth company asked, can you do it?

    Zookeeper’s data model

    Zookeeper’s view data structure, much like UNIX file system, is also tree like, so that each path can be determined to be unique. The nodes of zookeeper are calledznodeIt can be passed throughPathThe structure diagram is as follows:

    Zookeeper's twelfth company asked, can you do it?

    Four types of znode

    According to the life cycle of nodes, znodes can be divided into four types: persistent and persistent_ Sequential, ephemeral, ephemeral_ SEQUENTIAL)

    • Persistent node

    Once created, this type of node will always exist on the ZK server. Until manually deleted.

    • Persistent order node (persistent)_ SEQUENTIAL)

    Its basic feature is different from the persistent node in that it increases the order. The parent node maintains a self increasing integer number for the order in which the child nodes are created.

    • Temporary node (ephemeral)

    The lifetime of the temporary node is bound to the client session. Once the client session fails (non TCP connection is disconnected), the node will be automatically cleaned up. ZK specifies that temporary nodes can only be used as leaf nodes.

    • Temporary order node (ephemeral)_ SEQUENTIAL)

    The basic characteristics are the same as the temporary nodes, and the sequential features are added.

    4. Interviewer: do you know what is stored in the znode node? What is the maximum data size of each node?

    I of chicken

    Zookeeper's twelfth company asked, can you do it?

    What is stored in the znode node?

    The code for the znode data node is as follows

    public class DataNode implements Record {
        byte data[];                    
        Long acl;                       
        public StatPersisted stat;       
        private Set<String> children = null; 
    }

    Haha, znode containsStorage data, access rights, child node reference, node status informationAs shown in the figure:

    Zookeeper's twelfth company asked, can you do it?

    • data:Business data information stored in znode
    • ACL:Record the client’s access rights to the znode node, such as IP.
    • child:Child node reference of the current node
    • stat:Contains the state information of the znode node, such asTransaction ID, version number, timestampwait.

    What is the maximum data size of each node

    In order to ensure high throughput, low latency, and data consistency, znode is only suitable for storing very small data, no more than 1m, preferably less than 1K.

    5. Interviewer: do you know the monitoring mechanism on the znode node? Let’s talk about the zookeeper watch mechanism.

    I of chicken

    Zookeeper's twelfth company asked, can you do it?

    • Watcher mechanism
    • Working principle of monitoring mechanism
    • Summary of watcher features

    Watcher monitoring mechanism

    Zookeeper allows the client to register a watcher to listen to a znode of the server. When some specified events of the server trigger the watcher, the server will send an event notification to the designated client to realize the distributed notification function, and then the client makes business changes according to the notification status and event type of the watcher.

    Watcher can be understood as a trigger registered by the client on a znode. When the znode node changes (add, delete, modify, check), the registration event corresponding to the znode will be triggered, and the registered client will receive asynchronous notification, and then make business changes.

    How watcher sniffer works

    Zookeeper's twelfth company asked, can you do it?

    • The watcher mechanism of zookeeper mainly includes client thread, watcher manager and zookeeper server.
    • When the client registers watcher with the zookeeper server, it stores the watcher object in the watchmanager of the client.
    • When the hookeeper server triggers the watcher event, it will send a notification to the client, and the client thread will take the corresponding watcher object from the watchermanager to execute the callback logic.

    Summary of watcher features

    • disposable:A watch event is a one-time trigger. Once triggered, the client will receive such information only once.
    • Asynchronous:Zookeeper server sends the notification event of watcher to the client asynchronously. It can’t be expected to monitor every change of the node. Zookeeper can only ensure the final consistency, but not strong consistency.
    • Lightweight:Watcher notification is very simple, it just notifies that an event has occurred, and does not pass on the content of the event object.
    • Client serial:The process of executing the client watcher callback is a serial synchronization process.
    • Register the watcher with GetData, exists, getchildren methods
    • Trigger the watcher with create, delete and SetData methods

    6. Interviewer: you have a certain understanding of the data structure of zookeeper. What about the characteristics of zookeeper

    Xiaocaiji’s me: (I recited books, aha ha)

    Zookeeper's twelfth company asked, can you do it?

    Zookeeper ensures the following distributed consistency features:

    • Sequence consistency: transaction requests initiated from the same client will be applied to zookeeper in strict order.
    • Atomicity: the processing results of all transaction requests are consistent on all machines in the cluster, that is, either all machines in the cluster have successfully applied a transaction or none of them have applied it.
    • Single viewNo matter which zookeeper server the client connects to, the server-side data model it sees is consistent.
    • Reliability:Once the server successfully applies a transaction and completes the response to the client, the server state changes caused by the transaction will be retained.
    • Real time (final consistency)Zookeeper can only ensure that in a certain period of time, the client will finally be able to read the latest data status from the server.

    7. Interviewer: you just mentioned the order consistency. How does zookeeper ensure the order consistency of transactions?

    The chicken’s me: (finished this question will not)

    Zookeeper's twelfth company asked, can you do it?

    Let’s take a look at this article (the answer to this question comes from this article): let’s talk about the order consistency of zookeeper

    You need to know the transaction ID, which is zxid. Zookeeper selects the new master node by comparing the zxid and machine ID of each node. Zxid is generated by the leader node. When there is a new write event, the leader generates a new zxid and broadcasts it with the proposal. Each node locally saves the zxid of the latest transaction, and the zxid is incremental. Therefore, the larger the zxid is, the more recent the data is.

    The generation rules of zxid are as follows:

    Zookeeper's twelfth company asked, can you do it?

    Zxid consists of two parts:

    • Term of office: the same leader is responsible for coordinating the writing after this election and before the next election;
    • Transaction counter: monotonically increasing. The counter is increased by one every time a write takes effect.

    The lower 32 bits of zxid are counters. Therefore, during the same term of office, zxid is continuous, and each node keeps its latest effective zxid. By comparing whether the zxid of the new proposal is different from its latest zxid by “1”, we can ensure that transactions take effect in strict order.

    8. Interviewer: you mentioned leader. Do you know how many roles the server of zookeeper has? What are the working states of server in zookeeper?

    I of chicken

    Zookeeper's twelfth company asked, can you do it?

    Zookeeper server role

    In zookeeper cluster, there are three roles: leader, follower and observer

    Leader

    The leader server is the core of the whole zookeeper cluster working mechanism

    • The unique scheduler and processor of transaction request ensures the order of cluster transaction processing
    • The scheduler of each service in the cluster

    Follower

    The follower server is the follower of zookeeper cluster status. Its main work is as follows:

    • Process client non transaction requests and forward transaction requests to leader server
    • Participate in the voting of transaction request proposal
    • Vote in leader election

    Observer

    Observer is a server role introduced in version 3.3.0. It acts as an observer, observing the latest state changes of zookeeper cluster and synchronizing these state changes. Its work:

    • Process non transaction requests from client and forward transaction requests to leader server
    • Do not participate in any form of voting

    Server working state under zookeeper

    The server has four states, namely, looking, following, leading and observing.

    • 1. Looking: find the leader status. When the server is in this state, it will assume that there is no leader in the current cluster, so it needs to enter the leader election state.
    • 2. Following: follower status. Indicates that the current server role is follower.
    • 3. Leading: leader status. Indicates that the current server role is leader.
    • 4. Observing: observer status. Indicates that the current server role is observer.

    9. Interviewer: you said that the server role is based on zookeeper cluster. Would you like to draw the deployment diagram of zookeeper cluster? How does zookeeper ensure the data consistency of master-slave nodes?

    I of chicken

    Zookeeper's twelfth company asked, can you do it?

    Deployment diagram of keeper cluster

    Zookeeper's twelfth company asked, can you do it?

    Zookeeper cluster is a master multi slave structure

    • If the data is to be written, it should be written to the master server (master node), and then the slave server will be informed.
    • If you are reading data, you can read from both the master server and the slave server.

    How does zookeeper ensure the data consistency of master and slave nodes

    We know that a cluster is a master-slave deployment structure. To ensure the consistency of master-slave nodes, there are only two main problems:

    • The primary server hung up or restarted
    • Data synchronization between master and slave servers~

    Zookeeper uses Zab protocol to ensure the data consistency of master and slave nodes, and Zab protocol supports itCrash recovery and message broadcastingThe two models solve these two problems well

    • Crash recovery: the leader hangs up, enters this mode, and selects a new leader
    • Message broadcast: synchronize the updated data from the leader to all followers

    When the leader server is suspended, all servers in the cluster enter the logging state. First, they will elect a new leader server. Then, the new leader server will synchronize with the follower service in the cluster. When more than half of the machines in the cluster have completed data synchronization with the leader server, they will exit the recovery mode and enter the message broadcasting mode. The leader server starts to receive the transaction request from the client and generates the transaction proposal to process the transaction request.

    10. Interviewer: the leader is dead. How to elect the leader? Let’s talk about the zookeeper election mechanism

    I of chicken

    Zookeeper's twelfth company asked, can you do it?

    When the server is started or the server is running (the leader is hung), it will enter the leader election. Let’s take a look ~ suppose there are five servers in the zookeeper cluster, and their myids are servers 1, 2, 3, 4, and 5, respectively, as shown in the following figure:

    Zookeeper's twelfth company asked, can you do it?

    Server initiated leader election

    In the initialization phase of zookeeper cluster, server (myid = 1-5)successivelyLet’s go. Let’s start the zookeeper election~

    Zookeeper's twelfth company asked, can you do it?

    • Server 1 (myid = 1) is started. Currently, there is only one server and the leader election cannot be completed
    • Server 2 (myid = 2) starts. At this time, the two servers can communicate with each other and begin to enter the leader election stage
    1. Each server issues a vote

    Both server 1 and server 2 vote as leader servers. The basic elements of voting include server myid and zxid, which are expressed in the form of (myid, zxid). In the initial stage, both server 1 and server 2 vote for themselves, that is, server 1 votes for (1,0), and server 2 votes for (2,0), and then each sends the vote to all other machines in the cluster.

    1. Accept votes from various servers

    Each server accepts votes from other servers. At the same time, the server will verify the validity of the vote, whether it is the current round of voting and whether it comes from the server in the logging state.

    1. Processing votes

    After receiving the votes from other servers, they will PK their votes with their own votes. The PK rules are as follows:

    • Priority is given to check zxid. The larger zxid server is preferred as the leader.
    • If the zxid is the same, myid will be compared, and the server with larger myid will be used as the leader.

    Server 1’s vote is (1,0), it receives a vote is (2,0), both zxid is 0, because the received myid = 2, greater than its own myid = 1, so it updates its own vote to (2,0), and then sends out the vote again. For server 2, it is no longer necessary to update your own vote, just send out the last vote information.

    1. Statistical voting

    After each vote, the server will count all the votes and judge whether more than half of the machines have received the same voting information. Server 2 receives two tickets, less than 3 (n / 2 + 1, n is the total server), so it continues to keep the looking state

    • Server 3 (myid = 3) starts and continues to enter the leader election phase

    Consistent with the previous process, servers 1 and 2 vote for themselves first, because server 3 has the largest myid, so we vote for it instead. At this time, the server has 3 votes (greater than or equal to N / 2 + 1), so server 3 is selected as the leader. Change the status of server 1 and 2 to follow, and server 3 to leading;

    • Server 4 starts and an election is initiated.

    At this time, servers 1, 2 and 3 are no longer in the logging state, and the ballot information will not be changed. Vote information results: server 3 has 4 votes, server 4 has 1 vote. Server 4 and change the status to follow;

    • Server 5 starts to launch an election.

    Similarly, the server votes to server 3 and server 5 and changes the status to following;

    • At the end of the vote, server 3 is elected leader

    Leader election during server operation

    Five servers (myid = 1-5) of zookeeper cluster are running. Suddenly, leader server 3 is hung up and leader election begins~

    Zookeeper's twelfth company asked, can you do it?

    • 1. Change status

    After the leader server is suspended, the remaining non observer servers will change their server status to logging, and then start to enter the leader election process.

    • 2. Each server initiates a vote

    Each server votes for itself, because it’s runtime, so the zxid of each server may be different. Assuming that the zxid of service 1, 2, 4 and 5 are 333666999888, the votes (1333), (2666), (4999) and (5888) are generated respectively, and then the votes are sent to all other machines in the cluster.

    • 3. Accept the vote from each server
    • 4. Processing voting

    The voting rules are consistent with those during the start-up of the zookeeper cluster. The zxid is checked first, and the large priority is taken as the leader. Therefore, it is obvious that the server zxid = 999 has the priority.

    • 5. Statistical voting
    • 6. Change the server status

    11. Interviewer: you mentioned that you used zookeeper’s distributed lock in the project. Would you like to talk about the implementation principle of ZK distributed lock?

    I of chicken

    Zookeeper's twelfth company asked, can you do it?

    Zookeeper uses the temporary sequential node feature to implement distributed locking.

    • Lock acquisition process (create temporary node, check sequence number is minimum)
    • Release lock (delete temporary node, listen for notification)

    Lock acquisition procedure

    • When the first client requests it, the zookeeper client creates a persistent node / locks. If it (client1) wants to obtain the lock, it needs to create a sequence node lock1 under the locks node, as shown in the figure

    Zookeeper's twelfth company asked, can you do it?

    • Then, the client client1 will look up all the temporary order child nodes under locks to determine whether its own node lock1 is the one with the least sort. If so, the lock will be obtained successfully.

    Zookeeper's twelfth company asked, can you do it?

    • At this time, if another client, client2, comes to try to obtain the lock, it will create another temporary node lock2 under locks

    Zookeeper's twelfth company asked, can you do it?

    • Client client2 will also look up all temporary order child nodes under locks to determine whether its own node lock2 is the smallest. At this time, it is found that lock1 is the smallest, so the lock acquisition fails. If the lock acquisition fails, it will not be reconciled. Client2 registers the watcher event with lock1, which is the top node of the order, to monitor whether lock1 exists. In other words, client2 fails to grab the lock and enters the waiting state.

    Zookeeper's twelfth company asked, can you do it?

    • At this point, if another client, client3, tries to acquire the lock, it will create another temporary node, lock3, under locks

    Zookeeper's twelfth company asked, can you do it?

    • Similarly, client3 will also find all temporary order child nodes under locks to determine whether its own node lock3 is the smallest. If it finds that it is not the smallest, it will fail to obtain the lock. It will not be reconciled, it will register the watcher event with the node lock2 in front of it to listen for the existence of the lock2 node.

    Zookeeper's twelfth company asked, can you do it?

    Release the lock

    Let’s take a look at the lock release processClient business completed or failed, will delete the temporary node and release the lock. If the task is completed, client1 will explicitly call the instruction to delete lock1

    Zookeeper's twelfth company asked, can you do it?

    If the client fails, lock1 will be deleted automatically according to the characteristics of the temporary node

    Zookeeper's twelfth company asked, can you do it?

    After the lock1 node is deleted, client2 is happy because it has been listening to lock1. When the lock1 node is deleted, client2 will immediately receive a notification, and will also find all temporary order child nodes under locks. If lock2 is the smallest, it will obtain the lock.

    Zookeeper's twelfth company asked, can you do it?

    In the same way, after client2 gets the lock, client3 is also covetous for it, ha ha~

    12. Interviewer: OK, the last question, please talk about the relationship between Dubbo and zookeeper. Why choose zookeeper as the registration center?

    I have answered so many questions, won’t you give it to me :

    Zookeeper's twelfth company asked, can you do it?

    The registry of Dubbo can choose zookeeper, memcached, redis, etc. Why choose zookeeper because of its features~

    • Name the service, and the service provider writes the URL to the designated node of zookeeper to complete the service publishing.
    • Load balancing, the bearing capacity of the registry is limited, and zookeeper cluster with web applications is easy to achieve load balancing.
    • ZK supports listening to events, which is especially suitable for publish / subscribe scenarios, such as Dubbo producers and consumers.
    • The data model is simple, and the data exists in memory, which can be described as high performance
    • Other features of zookeeper can be described~

    Official account number

    Zookeeper's twelfth company asked, can you do it?

    Reference and thanks

    • < principle and practice of distributed consistency from Paxos to zookeeper > >
    • Interview questions for zookeeper
    • 28 advanced necessaries zookeeper interview questions (recommended collection!)
    • Cartoon: what is zookeeper?
    • Let’s talk about the order consistency of zookeeper
    • Zookeeper — Conformance protocol: Zab protocol
    • The principle of zookeeper’s election mechanism
    • Cartoon: how to realize distributed lock with zookeeper?