Three ways to implement distributed lock

Time:2021-6-12

1、 Basic concepts

1. Introduction
Traditional locks are all solutions with JDK official locks, that is to say, these locks can only be valid in one JVM process. We call this kind of lock a single application lock. However, with the rapid development of the Internet, can single application lock meet our needs?
New reading experience:http://www.zhouhong.icu/post/143

All codes of this article:https://github.com/Tom-shushu/Distributed-system-learning-notes/

2. Evolution of Internet system architecture
At the beginning of the development of Internet system, the system is relatively simple, consumes less resources, and has less user visits. We can only deploy a Tomcat application to meet the needs. The system architecture is as follows:
A Tomcat can be regarded as a JVM process. When a large number of requests arrive at the system concurrently, all requests fall on the only Tomcat. If some request methods need to be locked, such as second kill to deduct inventory, they can meet the demand, which is the same as what we said in the previous chapter. However, with the increase of traffic, it is difficult to support a Tomcat. At this time, we need to deploy Tomcat in a cluster and use multiple Tomcat to support the whole system
In the figure above, we have deployed two Tomcat to support the system. When a request arrives at the system, it will go through nginx first. Nginx mainly does load forwarding, and it will forward the request to one of Tomcat according to its own load balancing policy. When a large number of requests are accessed concurrently, the two Tomcats share all the visits. At this time, we also use the single application lock in the scenario of second kill and inventory deduction. Can we still meet the requirements?
3. Limitations of single application lock
As shown in the figure above, there are two Tomcat in the whole system architecture, and each Tomcat is a JVM. In the seckill business, because everyone is rushing to buy seckill products, a large number of requests arrive at the system at the same time and are distributed to two Tomcat through nginx. Through an extreme case scenario, we can better understand the limitations of single application lock. If there is only one second kill item, then only one of these requests can successfully grab the item, which requires a lock on the method of inventory deduction. The action of inventory deduction can only be executed one by one, but not at the same time. If it is executed at the same time, the item may be grabbed by multiple people at the same time, This leads to oversold. After the lock is added, the action of deducting the inventory is executed one by one. If the inventory is deducted to a negative number, an exception will be thrown, indicating that the user did not grab the goods. Lock seems to solve the problem of second kill, but is it really so?
We can see that there are two Tomcats in the system. The locks we add are provided by JDK. This kind of lock can only work under one JVM, namely               There is no problem in a Tomcat. When there are two or more Tomcats, a large number of concurrent requests are distributed to different Tomcats. Concurrency can be prevented in each Tomcat. However, among multiple Tomcats, the request to obtain a lock in each Tomcat generates concurrency, which leads to oversold. This is also the limitation of single application lock. It can only lock in one JVM, but not from this application level.
How to solve this problem? This requires the use of distributed locks to lock the entire application level. What is distributed lock? How do we use distributed locks?
4. What is distributed lock
Before talking about distributed locks, let’s take a look at the characteristics of single application locks. Single application locks are effective in a single JVM process and cannot cross JVM or process. Then the definition of distributed lock comes out. Distributed lock is a lock that can span multiple JVMs and processes. This kind of lock is called distributed lock.
5. Design idea of distributed lock
In the figure above, since Tomcat is started by Java, each Tomcat can be regarded as a JVM, and the internal lock of the JVM cannot span multiple processes. Therefore, if we want to implement distributed lock, we can only look outside these JVMs and implement distributed lock through other components. The architecture of the system is shown in the figure

The two Tomcats implement distributed locks across JVMs and processes through third-party components. This is the solution of distributed lock. Find the third-party components that all JVMs can access together, and implement distributed lock through the third-party components.
6. The existing distributed solutions
Distributed locks are implemented by third-party components. At present, the popular solutions of distributed locks are as follows:
  • Database, through the database can achieve distributed lock, but in the case of high concurrency on the database pressure, so rarely used.
  • Redis, with the help of redis, can also realize distributed locking. Moreover, there are many kinds of redis Java clients, and the methods used are not the same.
  • Zookeeper, zookeeper can also implement distributed lock. Similarly, zookeeper also has multiple Java clients, and its usage is different.

2、 Solutions to oversold in e-commerce platform

① Solution for oversold under monomer architecture

1. Oversold phenomenon 1
What is oversold: a product has 10 units in stock and 15 units are sold.

 

 

A and B place an order at the same time, read the inventory at the same time, reduce the inventory at the same time, and update the database at the same time. This is inventory minus 1, but they place an order for two pieces.
terms of settlement:
The inventory deduction is not carried out in the program, and is solved through the database; Transfer the inventory increment to the database, deduct one inventory, and the increment is – 1; The inventory is calculated in the database update statement, and the concurrency problem is solved through the update row lock
2. Oversold phenomenon 2
Using the above solution through the database row lock, there will be the following second phenomenon: the inventory of the database will be reduced to a negative number.

 

Solution 1
After the inventory is updated successfully, retrieve the commodity inventory again. If the commodity is negative, an exception will be thrown.
Solution 2
Add a lock, bind the database inventory verification and database inventory update together and add a lock. Only one can get this lock at a time, so as to avoid the situation that the inventory is reduced to a negative number (- 1).

3. Specific code implementation

  • Create database table
CREATE DATABASE /*!32312 IF NOT EXISTS*/`distribute` /*!40100 DEFAULT CHARACTER SET utf8mb4 */;
USE `distribute`;
DROP TABLE IF EXISTS `order`;
CREATE TABLE `order` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `order_status` int(1) NOT NULL,
  `receiver_name` varchar(255) NOT NULL,
  `receiver_mobile` varchar(11) NOT NULL,
  `order_amount` decimal(11,0) NOT NULL,
  `create_time` time NOT NULL,
  `create_user` varchar(255) NOT NULL,
  `update_time` time NOT NULL,
  `update_user` varchar(255) NOT NULL,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=16 DEFAULT CHARSET=utf8mb4;
DROP TABLE IF EXISTS `order_item`;
CREATE TABLE `order_item` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `order_id` int(11) NOT NULL,
  `product_id` int(11) NOT NULL,
  `purchase_price` decimal(11,0) NOT NULL,
  `purchase_num` int(3) NOT NULL,
  `create_time` time NOT NULL,
  `create_user` varchar(255) NOT NULL,
  `update_time` time NOT NULL,
  `update_user` varchar(255) NOT NULL,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=12 DEFAULT CHARSET=utf8mb4;
DROP TABLE IF EXISTS `product`;
CREATE TABLE `product` (
  `id` int(11) NOT NULL AUTO_INCREMENT COMMENT 'id',
  `product_ Name ` varchar (255) not null comment 'trade name',
  `Price ` decision (11,0) not null comment 'price',
  `Count ` int (5) not null comment 'inventory',
  `product_ Desc ` varchar (255) not null comment 'description',
  `create_ Time ` time not null comment 'creation time',
  `create_ User ` varchar (255) not null comment 'created by',
  `update_ Time ` time not null comment 'update time',
  `update_ User ` varchar (255) not null comment 'updated by',
  PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=100101 DEFAULT CHARSET=utf8mb4;
insert  into `product`(`id`,`product_ name`,`price`,`count`,`product_ desc`,`create_ time`,`create_ user`,`update_ time`,`update_ User ') values (100100,' test product ',' 1 ', 1,' test product ',' 18:06:00 ',' Zhou Hong ',' 19:19:21 ',' xxx ');
/**Subsequent distributed locks need to be used**/
DROP TABLE IF EXISTS `distribute_lock`;
CREATE TABLE `distribute_lock` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `business_ Code ` varchar (255) not null comment 'according to the business code, different services use different locks',
  `business_ Name ` varchar (255) not null comment 'comment, mark encoding purpose',
  PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=2 DEFAULT CHARSET=utf8mb4;
insert  into `distribute_lock`(`id`,`business_code`,`business_name`) values (1,'demo','test');
  • Order creation, inventory minus 1 and other main logic codes (here you use reentrantlock, of course, you can also use other locks)
//Note: you can't use annotation to roll back, otherwise the next thread will come in before the transaction is committed
//    @Transactional(rollbackFor = Exception.class)
    public Integer createOrder() throws Exception{
        Product product = null;
        lock.lock();
        try {
            //Open transaction
            TransactionStatus transaction1 = platformTransactionManager.getTransaction(transactionDefinition);
            //Find the goods you want to buy
            product = productMapper.selectByPrimaryKey(purchaseProductId);
            if (product==null){
                platformTransactionManager.rollback(transaction1);
                Throw new exception ("purchased product: + purchaseproductid +" does not exist ");
            }
            //Get current inventory of goods
            Integer currentCount = product.getCount();
            System. Out. Println (thread. Currentthread(). Getname() + "stock number" + currentcount) ";
            //Check the inventory (if the quantity of purchased goods is greater than the inventory quantity, an exception will be thrown)
            if (purchaseProductNum > currentCount){
                platformTransactionManager.rollback(transaction1);
                Throw new exception ("product" + purchaseproductid + "only" + currentcount + "items left, cannot be purchased");
            }
            productMapper.updateProductCount(purchaseProductNum,"xxx",new Date(),product.getId());
            platformTransactionManager.commit(transaction1);
        }finally {
            lock.unlock();
        }
        //Create order
        TransactionStatus transaction = platformTransactionManager.getTransaction(transactionDefinition);
        Order order = new Order();
        order.setOrderAmount(product.getPrice().multiply(new BigDecimal(purchaseProductNum)));
        order.setOrderStatus(1);// Pending
        order.setReceiverName("xxx");
        order.setReceiverMobile("15287653421");
        order.setCreateTime(new Date());
        Order. Setcreateuser ("no");
        order.setUpdateTime(new Date());
        Order. Setup update user ("ha ha ha ha");
        orderMapper.insertSelective(order);
        //Create order明细
        OrderItem orderItem = new OrderItem();
        orderItem.setOrderId(order.getId());
        orderItem.setProductId(product.getId());
        orderItem.setPurchasePrice(product.getPrice());
        orderItem.setPurchaseNum(purchaseProductNum);
        Orderitem. Setcreateuser ("no");
        orderItem.setCreateTime(new Date());
        orderItem.setUpdateTime(new Date());
        Orderitem. Setupdateuser ("ha ha ha ha");
        orderItemMapper.insertSelective(orderItem);
        //Transaction commit
        platformTransactionManager.commit(transaction);
        return order.getId();
    }
  • Testing (using five concurrent threads)
@Test
public void concurrentOrder() throws InterruptedException {
    Thread.sleep(60000);
    CountDownLatch cdl = new CountDownLatch(5);
    CyclicBarrier cyclicBarrier = new CyclicBarrier(5);
    //Create 5 threads to execute order placing operation
    ExecutorService es = Executors.newFixedThreadPool(5);
    for (int i =0;i<5;i++){
        es.execute(()->{
            try {
                //Wait for five threads to reach await() at the same time, and then execute the create order service. At this time, the five threads will stack up to execute at the same time
                cyclicBarrier.await();
                Integer orderId = orderService.createOrder();
                System. Out. Println ("order ID:" + OrderID ");
            } catch (Exception e) {
                e.printStackTrace();
            }finally {
                //After the execution of each thread, it will be reduced by one
                cdl.countDown();
            }
        });
    }
    cdl.await();
    es.shutdown();
}

② Implementation of distributed lock in distributed architecture

1、 Implementation of distributed lock based on Database

Multiple processes and threads access the common component database
Access the same data by select… For update and lock the data by for update
  • Add the following custom SQL in mapper. XML
select * from distribute_lock
  where business_code = #{businessCode,jdbcType=VARCHAR}
  for update
  • Main logic implementation
@RequestMapping("singleLock")
/**
 *Before adding transactional annotation, querying distributed lock and sleep are not atomic operations. After obtaining distributed lock, transaction will be submitted automatically,
 *Therefore, the second request is not prevented from acquiring the lock. After adding the annotation, the transaction has not been committed until the end of the sleep, so the transaction will be committed after the end of the sleep,
 *Only then can the second request obtain the distributed lock from the database
 */
@Transactional(rollbackFor = Exception.class)
public String singleLock() throws Exception {
    Log. Info ("I entered the method!");
    DistributeLock distributeLock = distributeLockMapper.selectDistributeLock("demo");
    If (distributelock = = null) throw new exception;
    Log. Info ("I entered the lock!");
    try {
        Thread.sleep(20000);
    } catch (InterruptedException e) {
        e.printStackTrace();
    }
    Return "I have executed it!";
}
  • Another project is the same as this one, just change the port number
advantage:
  • Simple and convenient, easy to understand, easy to operate
Disadvantages:
  • Large amount of concurrency, great pressure on the database
Suggestions:
  • Database as lock is separated from business database

2、 Implementation of distributed lock based on redis setnx

① Redis command to get lock
  • Set resource_name my_random_value NX PX 30000
  • resource_ Name: resource name. Different locks can be distinguished according to different businesses
  • my_ random_ Value: random value. The random value of each thread is not the same. It is used for checking when releasing the lock
  • Nx: if the key does not exist, the setting is successful; if the key exists, the setting is unsuccessful
  • PX: automatic failure time, in case of abnormal situation, the lock can be expired
② Implementation principle
  • With the atomicity of NX, when multiple threads are concurrent, only one thread can be set successfully
  • If the lock is set successfully, the subsequent business processing can be performed
  • If an exception occurs, the lock will be released automatically after the expiration date of the lock.
  • Redis’s delete command is used to release the lock
  • When releasing the lock, check the random number of the value setting, and only the same number can be released
  • Lua script to release lock
if redis.call("get",KEYS[1])==argv[1] then 
    return redis.call("del",KEYS[1])
else
    return 0
end
③ Why add Lua script validation:
If there is no verification, the lock may be confused, as shown in the figure above: A may release B’s lock, which may cause problems.
④ Redis distributed lock key code encapsulation
package com.example.distributelock.lock;
import lombok.extern.slf4j.Slf4j;
import org.springframework.data.redis.connection.RedisStringCommands;
import org.springframework.data.redis.core.RedisCallback;
import org.springframework.data.redis.core.RedisTemplate;
import org.springframework.data.redis.core.script.RedisScript;
import org.springframework.data.redis.core.types.Expiration;
import java.util.Arrays;
import java.util.List;
import java.util.UUID;
@Slf4j
public class RedisLock implements AutoCloseable {
    private RedisTemplate redisTemplate;
    private String key;
    private String value;
    //Expiration time unit: seconds
    private int expireTime;
    public RedisLock(RedisTemplate redisTemplate,String key,int expireTime){
        this.redisTemplate = redisTemplate;
        this.key = key;
        this.expireTime=expireTime;
        this.value = UUID.randomUUID().toString();
    }
    /**
     *Get distributed lock
     * @return
     */
    public boolean getLock(){
        RedisCallback redisCallback = connection -> {
            //Set nx
            RedisStringCommands.SetOption setOption = RedisStringCommands.SetOption.ifAbsent();
            //Set expiration time
            Expiration expiration = Expiration.seconds(expireTime);
            //Serialize key
            byte[] redisKey = redisTemplate.getKeySerializer().serialize(key);
            //Serialize value
            byte[] redisValue = redisTemplate.getValueSerializer().serialize(value);
            //Perform setnx operation
            Boolean result = connection.set(redisKey, redisValue, expiration, setOption);
            return result;
        };
        //Get distributed lock
        Boolean lock = (Boolean)redisTemplate.execute(redisCallback);
        return lock;
    }
    //Release the lock
    public boolean unLock() {
        String script = "if redis.call(\"get\",KEYS[1]) == ARGV[1] then\n" +
                "    return redis.call(\"del\",KEYS[1])\n" +
                "else\n" +
                "    return 0\n" +
                "end";
        RedisScript redisScript = RedisScript.of(script,Boolean.class);
        List keys = Arrays.asList(key);

        Boolean result = (Boolean)redisTemplate.execute(redisScript, keys, value);
        Log.info ("result of releasing lock): + result];
        return result;
    }
    @Override
    public void close() throws Exception {
        unLock();
    }
}
⑤ Testing
@RequestMapping("redisLock")
public String redisLock(){
    Log. Info ("I entered the method!");
    try (RedisLock redisLock = new RedisLock(redisTemplate,"redisKey",30)){
        if (redisLock.getLock()) {
            Log. Info ("I entered the lock!");
            Thread.sleep(15000);
        }
    } catch (InterruptedException e) {
        e.printStackTrace();
    } catch (Exception e) {
        e.printStackTrace();
    }
    Log. Info ("method execution completed");
    Return "method execution completed";
}
⑥ It’s no problem to use esjob timing task in the project, but if spring task is used in the project for timing, there may be repeated task execution in the cluster.
Solution: after the distributed lock arrives at a fixed time, before executing the task, which node obtains the lock, which node will execute the task.
⑦ Code implementation
public class SchedulerService {
    @Autowired
    private RedisTemplate redisTemplate;
    @Scheduled(cron = "0/5 * * * * ?")
    public void sendSms(){
        try(RedisLock redisLock = new RedisLock(redisTemplate,"autoSms",30)) {
            if (redisLock.getLock()){
                Log. Info ("run this program every five seconds!");
            }
        } catch (Exception e) {
            e.printStackTrace();
        }
    }
}

3、 Implementation of distributed lock based on zookeeper

Zookeeper’s observer
  1. There are three ways to set an observer: getdata(); getChildren(); exists();
  2. The node data changes and is sent to the client;
  3. The observer can only monitor once, and it needs to be reset to monitor again;
Implementation principle

  1. Using the characteristics of zookeeper’s instantaneous ordered nodes;
  2. When multithreading creates instantaneous nodes concurrently, it gets ordered sequence;
  3. The thread with the smallest sequence number obtains the lock;
  4. Other threads listen to the previous sequence number of their own sequence number;
  5. The previous thread completes execution and deletes its own serial number node;
  6. The thread with the next sequence number is notified and continues to execute;
  7. And so on, when creating nodes, the execution order of threads has been determined;
Code implementation:
package com.example.distributelock.lock;
import lombok.extern.slf4j.Slf4j;
import org.apache.zookeeper.*;
import org.apache.zookeeper.data.Stat;

import java.io.IOException;
import java.util.Collections;
import java.util.List;
@Slf4j
public class ZkLock implements Watcher,AutoCloseable {
    private ZooKeeper zooKeeper;
    private String businessName;
    private String znode;
    public ZkLock(String connectString,String businessName) throws IOException {
        this.zooKeeper = new ZooKeeper(connectString,30000,this);
        this.businessName = businessName;
    }
    public boolean getLock() throws KeeperException, InterruptedException {
        Stat existsNode = zooKeeper.exists("/" + businessName, false);
        if (existsNode == null){
            zooKeeper.create("/" + businessName,businessName.getBytes(),
                    ZooDefs.Ids.OPEN_ACL_UNSAFE,
                    CreateMode.PERSISTENT);
        }
        znode = zooKeeper.create("/" + businessName + "/" + businessName + "_", businessName.getBytes(),
                ZooDefs.Ids.OPEN_ACL_UNSAFE,
                CreateMode.EPHEMERAL_SEQUENTIAL);
        znode = znode.substring(znode.lastIndexOf("/")+1);
        List childrenNodes = zooKeeper.getChildren("/" + businessName, false);
        Collections.sort(childrenNodes);
        String firstNode = childrenNodes.get(0);
        if (!firstNode.equals(znode)){
            String lastNode = firstNode;
            for (String node:childrenNodes){
                if (!znode.equals(node)){
                    lastNode = node;
                }else {
                    zooKeeper.exists("/"+businessName+"/"+lastNode,true);
                    break;
                }
            }
            synchronized (this){
                wait();
            }
        }
        return true;
    }
    @Override
    public void process(WatchedEvent watchedEvent) {
        if (watchedEvent.getType() == Event.EventType.NodeDeleted){
            synchronized (this){
                notify();
            }
        }
    }
    @Override
    public void close() throws Exception {
        zooKeeper.delete("/"+businessName+"/"+znode,-1);
        zooKeeper.close();
        Log.info ("I released the lock");
    }
}
Test:
@RequestMapping("zkLock")
    public String zkLock(){
        Log. Info ("I entered the method!");
        try (ZkLock zkLock = new ZkLock("localhost:2181","order")){
            if (zkLock.getLock()) {
                Log. Info ("I entered the lock!");
                Thread.sleep(15000);
            }
        } catch (InterruptedException e) {
            e.printStackTrace();
        } catch (Exception e) {
            e.printStackTrace();
        }
        Log. Info ("method execution completed");
        Return "method execution completed";
    }