Distributed – distributed lock



  • Preface
  • Idempotency
  • Lock properties
  • Distributed lock
  • design goal
  • Design thinking
  • boundary condition
  • Main points of design
  • Different implementations
  • Concluding remarks


Suddenly I feel that it is almost imaginary to want to live a safe life. Most of the prosperous times in history are only thirty or forty years. How to ensure that I live a long time must be in those thirty or forty years (but I really hope that the future will be better and better, the general situation will be better and the individual will be better). A stable job, people who love themselves, people they love feel a little extravagant. I know that these kinds of things will be found slowly in life. Maybe in the blink of an eye, I will look back on the past and find that the choice of these things is not free, but driven by time. It seems that with the growth of age, I feel more and more insignificant. The past things are fixed and more and more. The less choices I have in the future, the less choices I have.
Alas, back to the truth, I had written that CAP last time, and I felt I had checked a lot of information, and I wrote it well. Looking back now, what is it about? Or is the brain too dumb, belongs to the kind of people who remember slowly, understand slowly and forget quickly. So think about how good it is this time, at least better than last time. By the way, when I have time to fill in the last article, I’ll start with a flag.



The definition of idempotency in HTTP/1.1 is: one or more requests for a resource forResources themselvesThe same results should be achieved (except for network timeouts, etc.). In terms of functions, f…. F (f (x)= f (x).


1. Prevent the disastrous consequences of requesting retries in important operations such as transactions and transfers.

The scope of idempotency


1. Reading Request – Natural Idempotency
2. Writing requests – non-idempotent, need to be controlled.

Database level

1. INSERT is non-idempotent and needs to be controlled by content.
2. UPDATE – Controls idempotency through WHERE conditions, and operates as little as possible with relative values. For example, UPDATE table1 SET column1 = column1 + 1 WHERE column2 = 2;
The results of each execution change, which is not idempotent. UPDATE table1 SET column1 = 1 WHERE column2 = 2; no matter how many times the execution is successful, the state is the same, so it is idempotent operation.
3. DELETE – Similarly controlled by WHERE conditions, minimizing the use of relative values. if there be
DELETE table1 WHERE column1 < now (); should be changed to DELETE table1 WHERE column1 < “2019-10-01”; better.

Business level

1. In the case of redundant deployment of multiple services, requests are consumed concurrently, so requests need to be converted from parallel to serial.

Strategies to Guarantee Identity

The essence of guaranteeing idempotency is to do a good job of serialization of resources, which is multi-directional.
Needless to say, new requests need to do a good job of weight-proof processing of important control attributes, and update requests can also be well controlled by optimistic locks.
In distributed systems, requests are processed from parallelism to serialization through distributed locks.

Lock properties

1. reentrant/non-reentrant

Reentrant:Same threadAfter the outer function acquires the lock, the code contained in the inner function to acquire the lock is unaffected. It is not necessary to apply for the lock again, but can be directly invoked for execution.
Non-reentrant: On the contrary, locks are acquired inside and outside the same thread before execution. It’s easy to cause deadlocks.
Both synchronized and Reentrantlock are reentrant locks

2. Fairness/Inequity

Fairness: First come, first served, threads requesting locks form queue consumption locks.
Unfair: When the request comes, it requests the lock first, then executes it, and throws it at the end of the queue waiting for the lock to be acquired.
Synchronized unfairness
Reentrantlock Options

3. read / write

Read Lock: It can be read by many people, but it can’t write when reading. Read Lock.
Write locks: When writing, you can’t read, and you can only write by yourself. Write locks.

4. Sharing/monopoly

Exclusive Locks: Only one thread can hold locks at a time, and ReentrantLock is a mutex implemented in an exclusive manner. Monopoly is a pessimistic locking strategy.
Shared Lock: This lock can be held by multiple threads, such as ReadWriteLock. The locking policy is relaxed to allow multiple threads of read operations to access shared resources simultaneously.
Note: AQS (AbstractQueued Synchronized) provides two different synchronization logic, exclusive and shared.

5. Interruptible/non-interruptible

Interruptible: The operation of a lock can be interrupted. ReentrantLock is interruptible.
Uninterruptible: contrary to the above. Synchronized is an uninterruptible lock.

Distributed lock

It is used to ensure mutually exclusive access of a particular shared resource in a multi-process environment in a different system. The essence is to avoid duplication of processing by serializing requests for this resource. However, it can not solve the idempotency problem of requests, and it still needs to control the idempotency in business code. A shared storage server needs to be created outside the system to store lock information.

design goal

Security attribute

1.mutex。 Only one client can hold the same lock at any time. That is, the existence of locks is globally strongly consistent.
2.Deadlock freeIt has lock failure mechanism. First, the system providing lock service is highly available and robust. Multi-node service, any node downtime or network partition does not affect the acquisition of locks; the second is the automatic renewal and release of locks by the client.
3.SymmetricFor any lock, the lock and unlock must be the same client.

Efficiency attribute

1.fault-tolerantHigh availability, high performance. The service itself is highly available and the system is robust. Multi-node service, any node downtime or network partition does not affect the acquisition of locks.
2.ReentrantIt can effectively reduce the occurrence of deadlock.
3. Multiple choices, you can choose the time to try to acquire the lock.
4. Client calls are simple. The code of lock is highly abstract and the service access is minimal.

Design thinking

1. Control of shared resources. The identifier of a resource can be used as the key of the lock, so that the resource can be controlled by a unique process.
2. Uniqueness control of locks. The process that acquires the lock obtains a unique identifier (value) separately as a condition for the release of the lock, so as to avoid the release of the lock by other processes. At the same time, the atomicity of judgment and del operation should be guaranteed to prevent mistake deletion. Usually Lua scripts are used.
3. Avoid releasing the process without executing the lock and accessing the resources in parallel. The process renews the contract in time to avoid the release of uncompleted business locks.
4. Prevent deadlock. The lock is released regularly and the release time is set.

boundary condition

1. The stability and consistency of the service itself that provides lock registration.
2. The lock failed to renew the lease as scheduled. Such as unsuccessful renewal of heartbeat, service startup GC, service suspension time during GC exceeds the effective time of lock, etc.
3. Business fake death, TTL is still continuing.

Main points of design

1. Registration of locks. To ensure that the registration of locks is atomic, that is, to determine whether locks exist and registration is serialized.
2. The renewal of locks. How to renew the lease of a lock with minimal invasion of business code? CAS atomicity.
3. The release of locks. The user of the lock releases only the lock he holds.

Different implementations

Redis implementation

1. single instance implementation principle

The unique thread of redis is serial processing, that is, it is an idempotent linear system. But this can cause a single point of failure. If redis uses master-slave mode, because redis replication is asynchronous, it will occur:

1. Client A acquires the lock at the master node.
2. The master node crashes before writing information to slave.
3. Slave was promoted to master.
4. Client B acquires the lock of the same resource as A. At this time, the resource enters the multi-process concurrent consumption state. It goes against the principle of mutual exclusion.

Of course, the probability of this situation is very low, and resources are generally acceptable if they are not sensitive to this situation.
Or the redis implementation of a single instance is used, in which case the program needs to be more tolerant of single point failures.

2. Implementation of RedLock algorithm

The essence of this implementation is to implement a consistency protocol in redis cluster to realize the unique key. However, all nodes are required to be redis master nodes, and completely independent of each other. There is no master-slave replication or other cluster coordination mechanism.

If a lock is to be acquired, the operation performed by the client requires:

1. Client gets the current time (milliseconds)
2. Use the same key and different values to request N master nodes in order. (
This step requires the client to set a multi-request timeout time that is less than the automatic release time. For example, when the automatic release time is 10 seconds, set the timeout time to 5-50 milliseconds. This is used to prevent excessive consumption of time on the downsized nodes. One node is unavailable and the next node should be requested immediately.
3. The client calculates the time consumed to acquire the lock (the time consumed by step 2 - the current time - the time acquired by step 1). If and only if the lock is acquired at N/2+1 node, the acquisition time of the lock is less than the failure time of the lock, then the acquisition of the lock is considered successful.
4. If a lock is acquired, its effective time can be regarded as the initial expiration time-the consumption time of the acquired lock.
5. if the client acquires the lock failure, no matter what causes (in fact, for two reasons, one is the success node is less than N/2+1, the two is timeout), it will release the lock at all nodes (even those nodes that do not get the lock at all).

According to the above algorithm description, redis cluster needs at least three master nodes. And the implementation is also more cumbersome. The overhead of locking is too high. The requirement of redis operation is also high. Overall, the cost of implementation is too high. Fortunately, somebody has already done it – Redisson

Implementation of Single Instance

Previous data are generally registered with the command setNX to lock, but this command can not set the automatic expiration time, only by setting value as a timestamp to control expiration, in fact, it is not very good. Now redis officially recommends using the SET command.

SET resource-name anystring NX EX max-lock-time

Since version 2.6.12, SET commands support multiple operations:
    EX seconds -- Set the specified expire time, in seconds.
    PX milliseconds -- Set the specified expire time, in milliseconds.
    NX -- Only set the key if it does not already exist.
    XX -- Only set the key if it already exist.

When releasing locks, we can use Lua scripting language to guarantee atomicity for the release of locks.

if redis.call("get",KEYS[1]) == ARGV[1]
        return redis.call("del",KEYS[1])
        return 0

Specific Code Implementation
Letuce is used here.

1. The Realization of Lock
 * If the lock is idle, the current thread gets the lock 
 * If the lock has been held by another thread, disable the current thread until the current thread acquires the lock 
 public void lock() {    
 // Selected Synchronization Mode    
    while (true){        
        if (tryLock())return;        
        this.sleepByMillisecond(renewalTime >> 1);    
 * Attempt to acquire locks 
 * If the lock is available, return true or false 
 * @return 
 public boolean tryLock() {    
    // Use ThreadLocal to save the current lock object as a re-entrant lock control
    if (threadLocal.get() != null) return true;
    String set = statefulRedisConnection.sync().set(lockKey, lockValue, new SetArgs().nx().ex(lockTime));   
    if (set != null && "OK".equals(set)){        
        System.out.println ("thread id:"+Thread.current Thread(). getId ()+ "lock success! Time:"+LocalTime.now ()));        
        isOpenExpirationRenewal = true; 
        return true;    
    return false;
 * Attempt to acquire locks 
 * Return true within a specified time or false 
 * @param time 
 * @param unit 
 * @return 
 * @throws InterruptedException 
 public boolean tryLock(long time, TimeUnit unit) throws InterruptedException {   
     // Get access time    
     LocalTime now = LocalTime.now();    
     while (now.isBefore(now.plus(time, (TemporalUnit) unit))){       
        if (tryLock())return true;    
     return false;
2. Rental renewal of locks

There may be two problems, one is that the main thread hangs and the renewal thread cannot be closed; the other is that the renewal itself fails. At present, it seems that the best way for the first case is to set exception capture in the lock code, and finally unlock in the final code block. The second way is to do a good job of log analysis code problems to solve. But if you have a better way to achieve it, you are welcome to preach.

protected void scheduleExpirationRenewal() {    
    Thread thread = new Thread(new ExpirationRenewal());    
private class ExpirationRenewal implements Runnable{    
    public void run() {        
        while (isOpenExpirationRenewal){            
            try {                
            } catch (InterruptedException e) {               
            String expirFromLua = "if redis.call('get', KEYS[1]) == ARGV[1]"                    
                    + " then "                    
                    + "return redis.call('expire',KEYS[1],ARGV[2])"                    
                    + " else "                   
                    + "return 0"                   
                    + " end";            
            Object eval = statefulRedisConnection.sync().eval(expirFromLua, 
                ScriptOutputType.INTEGER, new String[]{lockKey}, lockValue, lockTime.toString());            
            System.out.println ("Release Acquisition Result Value:" + Eval + ((long) eval==1), "Rent Success", "Rent Failure");        
3. Release of locks
public void unlock() {    
// Closing Rent Rental    
isOpenExpirationRenewal = false;    
    // delete lock    
    String delFromLua = "if redis.call(\"get\", KEYS[1]) == ARGV[1]"    
        + " then "    
        + "return redis.call(\"del\",KEYS[1])"    
        + " else "    
        + "return 0"    
        + " end";    
    Long eval = statefulRedisConnection.sync().eval(delFromLua, ScriptOutputType.INTEGER, new String[]{lockKey}, lockValue);    
    if (eval == 1){       
        System. out. println ("lock release success");    
    }else {       
        // Better keep a log
        System. out. println ("The lock has been released");    
4. summary

It only realizes reentrant. We need to consider how to achieve fair and unfair switching, and read-write locks. The key to achieve fair lock is to maintain a cross-process request lock queue, which can only be implemented by redis itself.

2. Implementation of RedLock algorithm
1. Introduction to Redisson

Get lazy and move directly to an official profile

Based on high-performance async and lock-free Java Redis client and Netty framework.

Redisson function is still very strong, recommended to have time to go to see (there is a Chinese document Oh).

2. Implementation of RedLock algorithm

Redissond’s RedissonRedLock object implements the lock algorithm introduced by RedLock. The method of use is as follows.

RLock lock1 = redissonInstance1.getLock("lock1");
RLock lock2 = redissonInstance2.getLock("lock2");
RLock lock3 = redissonInstance3.getLock("lock3");

RedissonRedLock lock = new RedissonRedLock(lock1, lock2, lock3);
// Simultaneous Locking: Lock1 lock2 lock3//Red Lock locks successfully on most nodes.

Multiple redissonInstance1 and RLock objects need to be created to form distributed locks. The data structure chosen in redis is hash, which is convenient to realize fair lock.

It’s still troublesome to use.

Zookeeper implementation


Zookeeper is a distributed coordination service, which implements the consistency protocol. It is divided into Sever and Client. Server is a distributed application. Zab conformance protocol, temporary sequential node of znode and watchs mechanism are the key to realize distributed lock in zookeeper.

1. Data model of zookeeper

The hierarchal name space (hierarchical namespace) can be seen as a distributed file system. It has a tree structure, and each node is separated by “/”. Most Unicode characters can be used as node names.
The node in zookeeper tree is called znode. A stat data structure is maintained in znode node, which stores data changes, ACL (Access Control List) – access control list. Each node exists. This list restricts who can do what. It is equivalent to the permission list of the node, including the version number of CREATE, READ, WRITE, DELETE, ADMIN changes and the corresponding timestamp.
The reading and writing of znode is atomic.

The nodes of znode can be divided into:

  • Persistent Nodes
    Even after the client that created the particular znode disconnects, the persistent node still exists. By default, all znodes are persistent unless otherwise specified.
  • Temporary node Ephemeral Nodes
    Temporary nodes are created with client and server sessions. When a session ends, the corresponding znode will be deleted. Therefore, temporary nodes cannot create child nodes. Temporary nodes play an important role in leader election.
  • Sequence Nodes
    Sequential nodes can be persistent or temporary. When a new znode is created as a sequential node, ZooKeeper sets the path of the znode by appending a 10-bit serial number to the original name. The serial number is a monotonically increasing sequence. For example, if a znode with a path / mynode is created as a sequential node, ZooKeeper changes the path to / mynode 0000000001 and sets the next serial number to 0000000002. When the increment exceeds 2147483647 (2 ^ 31 – 1), the counter overflows (resulting in the name “- 2147483648”). Sequential nodes play an important role in locking and synchronization
  • Container Nodes Version 3.5.3 Added
    This type of node is set for special cases, such as leader, lock, etc. When the last child node in the container is deleted, the container will become a candidate for deletion by the server sometime in the future (personal feeling is similar to regular deletion in redis). Therefore, KeeperException. NoNoNodeException may be thrown when creating Container node subnodes, so it is necessary to determine whether the exception is thrown and recreate the container node after the exception is thrown.
  • TTL Nodes Version 3.5.3 Added
    This type of node can only be a persistent node. When creating a persistent node or a persistent sequential node, you can choose to set the TTL of the node. If the TTL time of the node has not been modified and there are no child nodes, then the node will be deleted by the server at some time in the future. TTL nodes must be enabled through system properties because they are disabled by default.

Clients can acquire node change information by setting up a Watchs to listen for znode node information.

Implementation steps

The key to lock is the temporary sequential node and the Watchs mechanism. That is to say, a queue to acquire locks is created by temporary sequential nodes, and the consumption of the previous data is monitored through the watchs mechanism to determine whether they are consumed or not.

1. Create a persistent node lock as the parent of the lock.
2. Client A requesting lock creates temporary sequential node lock000000N under lock node.
3. Get the list of nodes under lock, get the node whose number is smaller than client A. If it does not exist, it means that the current thread has the smallest serial number and gets the lock.
4. If the node whose number is smaller than client A exists, set Watchs to listen on the sub-node. Secondary nodes are deleted to determine whether they are the smallest node or not, and then the lock is acquired.

Apache’s open source library Curator provides the corresponding implementation of distributed locks.
However, there is time to implement one on your own, not only using the usual way above, but also using Container nodes or TTL nodes.


Compared with redis, the implementation of zookeeper saves a lot of trouble without considering the consistency of locks. Wait until I have time to look at the Curator source code.

Etcd implementation

What is etcd?

A distributed key-value pair (K-V) storage service. It implements the consistency algorithm Raft (bad and forgotten, this consistency algorithm feels that it is not written in enough detail, until I have time to study it carefully), so the high availability must be 2N+1. Mainly used to store data that rarely changes, and provide monitoring queries. The service API can be invoked using HTTP / HTTPS / grpc, and the grpc mode has its own api, so it is not necessary to care about renewal when using it. Grpc is available in V3 version. V3 version still supports rest mode invocation of HTTP / https, but it should be noted that V2 version is not compatible with V3 version, commands are mostly modified, data are not interoperable, such as data created by V2 version, V3 version is not visible. And the V3 version of rest call mode support is not very good.

data model

Etcd is the basic key-value structure.

  • Logical view:
    The logical view of the storage area is a flat binary.To be honest, I don’t understand the word very well. I need to check the data. Personally, I understand that it is a hierarchical structure of data, similar to the structure of trees. If you have knowledge, you can give advice and hold your fist.) Key space. Key space maintains multiple reviews. Each transaction creates a new Revision in the Key space. Key’s life cycle is from creation to destruction. Each Key may have one or more life cycles (meaning that it keeps detailed records of key creation, modification and destruction multiple times). If Key does not exist in the current modification, then the current version of Key starts from 1. Deleting Key generates a flag that contains the proxy information for the currently closed key (by resetting version 0). Each version of the key changes itself. The version increases monotonously over the life cycle of a key. Once a compaction occurs, any key life cycle record (closed generation) before compaction revision is deleted, and the values set before compaction revision are deleted (except the latest).

  • Physical view
    The KV data of etcd is persisted to disk in the form of B + book. When updating data, it does not update the original data structure, but generates an updated data structure. So its reading and writing performance can not be compared with redis and the like.

The foundation of etcd implementing distributed lock
  • Lease (A short-lived renewable contract that deletes keys associated with it on its expiry) mechanism: the lease mechanism (TTL, Time to Live). Rental renewal.
  • Revision (A 64-bit cluster-wide counter that is incremented each time the key space is modified) mechanism: Each Key has a Revision, and each transaction creates a new Revision in the keyspace, which is monotonically incremental with an initial value of zero. The order of write operations is known by the size of the review. When implementing distributed locks, the order of locks acquisition can be determined according to the size of Revision to achieve fair locks.
  • Prefix mechanism: Prefix mechanism, also known as directory mechanism. Keys can be created as directories, such as key1 = “/ mylock / 00001” and key2 = “/ mylock / 00002”. Through prefix / mylock range query, the KV list is obtained, and the order of acquisition of locks is controlled by the size of Revision to achieve fair locks.
  • Watch mechanism: Watch mechanism, Watch can monitor a single key, but also support scope monitoring.
Implementation steps

Comparing zookeeper with etcd, we find that the overall mechanism is the same. So its implementation is roughly the same.
The steps are as follows:

1. Create the prefix of the global key for the distributed lock of the corresponding resource, such as / mylock, then the key created by the client of the first request is / mylock / uuid001, and the second is / mylock / uuid002.
2. Create lease Lease for the corresponding resource at a specific appropriate time.
3. The client puts its own key, such as / mylock / uuid001. Value here is not empty. Get the revision number of your put and record it.
4. Get all the key value pair information under the key prefix, and sort it according to the size of revision number, the smaller is in the first place.
5. Contrast whether the revision value of put is the smallest. If it is, get the lock and open the renewal thread. No, get the previous lock operation index and create a Watcher to listen on the previous key (which will be blocked).

The latest version of etcd provides the implementation of distributed locks. There is no need to care about the release and renewal of the lock. It is automatically released when unlock is called. The key word lock is basically the same as the above principle.
Etcd java client has etcd4j, jetcd and so on.


The implementation of etcd is relatively simple, and it has realized the distributed lock itself. It is easy to acquire the lock with only one lock command. But etcd seems not to be very common in other ecological applications of java, and it is not very cost-effective if it is only used to acquire distributed locks.

Concluding remarks

Finally, it’s finished. In fact, it’s in a hurry. The zookeeper in the middle hasn’t been written out in detail. The redis one hasn’t implemented a fair lock. Now these things are more and more comprehensive, some do not have to write, but the principles must be understood. In fact, these are the concrete realization of comparison – renewal – release. If you have a chance, you still need to look at the source code and expand your thinking. I hope you have time to look at the source code of Redisson, Curator and Jetcd locks. Alas, always standing flag has been cool, always dragging has been cool, I do not know which one can be better.
Sleep, sleep.
Distributed – distributed lock

Reference material:

Identity: https://www.cnblogs.com/javalyy/p/8882144.html
Implementation of Zookeeper Distributed Lock: https://www.cnblogs.com/javalyy/p/8882144.html
Etcd lock implementation: http://blogspring.cn/view/120