If you don’t know raft algorithm, how dare you say that you have studied distributed computing

Time:2021-6-27

1、 Introduction of raft algorithm

Raft is a “consensus” algorithm. Consensus means to ensure that all participants have the same cognition, which is simply how to achieve strong consistency.

“Consensus” includes two aspects: between servers and between clients and servers

1. Between servers

It means that all servers need to reach a “consensus”. For example, a kV system like redis, if the server is a cluster of three nodes: A, B and C, the client first sends a command of set key1 a to node a, then sends a command of set key1 B to node B, and finally executes get key1 on all servers to get B;

2. Between client and server

Let’s go back to the example above. When the client executes the command set key1 a, and the server returns success, the client thinks it is successful. In the middle, the client cannot return null or not a when fetching key1 due to network or other reasons.

Consensus algorithm mainly solves the problem of consistency in distributed system

1. Paxos algorithm

This is the earliest and very complicated one. To be honest, I have read a lot of articles, but I only have a general understanding. The details are still not very clear. I still need to watch some implementation code to deepen my understanding;

2. Raft algorithm

This is what we are going to introduce today. It was proposed by Diego ongaro and John ousterhout of Stanford University in 2014. Besides proving the correctness of the algorithm, it also provides relevant implementation and reference code. Therefore, the media has been promoting that this algorithm is easier to understand than Paxos;

3. Zab algorithm

The algorithm used by zookeeper can be regarded as a variant of Paxos algorithm. If you are interested, you can learn from Apache official website and zookeeper source code.

2、 Application scenarios and cases of raft algorithm

The above describes the distributed related algorithms. Raft is relatively easy to use, and paxox is inevitable if we want to further study; Note that it is relatively simple, because after entering the analysis, you will find that it is far from as simple as you think. Judging from my current learning progress, it took me three weeks to learn some details.

The way to learn is to read the official introduction first. The following is its official website:

https://raft.github.io/raft.pdf

There is also an animation introduction:

http://thesecretlivesofdata.c…

Of course, there is also the Chinese version. Just search the Chinese version of raft paper.

At present, there are many open source implementations

Etcd (go language implementation)

Pingcap (rust language, CODIS author’s company)

Soft jraft (Java language, Ali open source)

The above are all implemented by large companies, which can be used in production. The functions are relatively complete, and the code is relatively complex. If you want to do a simple demo analysis first, you can see the following:

1. Open source Java implementation of Baidu Engineer

https://github.com/wenweihu86…

2. Implementation of Alipay Engineer

It’s also written in Java, but it has few functions. It doesn’t have prevote and other functions. The author has also published a Book: “development practice of distributed consistency algorithm”. It’s recommended to learn from the book:

https://github.com/xnnyygn/xraft

The following is a personal learning route for reference only: first read the paper, that is, theory, then run the code, and then look at the theory in reverse, and finally you can master the corresponding rules without reading the paper.

3、 Overall introduction of raft algorithm

Because raft algorithm has a lot of content, it will be introduced in several parts later. This article will first give a general introduction to the whole thing.

Officially, raft algorithm is divided into the following three parts: election, log replication and security. From the perspective of implementation, raft algorithm is divided into election, log and new node joining.

In addition, if we want to realize a complete application, we also need to realize the part of state machine; What is a state machine? A state machine means that the system / object itself has multiple states, and then various states are switched and transformed. In fact, most distributed systems can be transformed into a state machine, and then the states of the nodes are transformed in consistent steps. Everyone’s states are consistent, and for applications, they are strongly consistent.

Take redis as an example. Suppose that the command flow from the client is as follows:

Set key1 a status: key1: a

Set key2 B status: key1: A, key2: B

That is to say, every operation of the client will cause the change of the state of the server, and then treat the whole modification as a sequence, then the whole is the state machine.

Back to the raft algorithm, let’s look at the problems to be solved in each block

1. Elections

Raft algorithm only supports single master, that is to say, it can only read and write from the master at any time

Who is the current master node;

How to ensure that only one master node will be selected;

How to quickly and correctly sense the master node to hang up;

How to quickly select the master node;

How to ensure that the main node will be selected without infinite cycle election;

How to ensure that the data of the master node is up-to-date;

2. Log

Raft algorithm achieves consistency through logs. The log part needs to solve the following problems:

Whose log is up-to-date;

How the master node copies logs to other nodes;

How to ensure the correctness of log copy, that is, no string and no error;

How to copy the log of new node joining;

How to achieve the consistency of each node log;

3. New node join

There is no brain fissure when new nodes join the cluster;

How to achieve strong consistency in configuration, there will be temporary inconsistency in the middle, and how it does not affect the normal operation of the cluster;

To be continued

Recommended Today

OSG compilation and installation – MSVC

OSG compilation and installation – MSVC 1. Preparatory work 1.1 environment Windows10 Qt5.14.2 Microsoft Visual Studio2017 cmake-3.20.4-windows-x86_64.msi   1.2 OSG Download osgOfficial website github OSG source code(OpenSceneGraph) Download URL You can also use git to download: git clone –branch OpenSceneGraph-3.6.5 https://github.com/openscenegraph/OpenSceneGraph.git   Dependency Library(3rdParty) Download address Here, select the compiledVisualStudio 2017Version Library Data resources(OpenSceneGraph-Data) Download […]