Technical solution of interface anti duplicate submission

Time:2021-5-7

 

Whether it’s the HTTP interface or the RPC interface, anti duplicate submission (anti duplicate interface) is an unavoidable topic.

There are differences and connections between duplicate submission and idempotent. Idempotent means that one request for a resource has the same effect as multiple requests. For example, the post method of HTTP is non idempotent. If the program is not handled well, repeated submission will lead to non idempotent, causing system data failure. Anti duplicate submission, when it belongs to the category of idempotent, is first realized by technical means. Secondly, it has to verify the uniqueness of business data.

 

In common B / s scenarios, the server receives the same HTTP request twice or more times in a very short time due to repeated submission, user shaking or network problems.

One of the duplicate submission of RPC interface is improper program call, that is, program vulnerability leads to duplicate submission. In one case, for example, Dubbo, because of network transmission problems, will trigger a call to try again.

 

Lock is a common scheme to prevent duplicate submission. Distributed system, generally with the help of redis or ZK distributed lock. For Java monomer applications, some netizens said that the synchronized lock mechanism of the language itself can be used. Strictly speaking, this is not appropriate, because synchronized is a synchronous lock under multithreading, which will only block the thread execution, but will not block the thread execution.

 

[notes]

  1. Setting of lockkey
  2. Locking is to intercept duplicate requests. The key must be related to the request data of the business operation request, and has the global uniqueness of the system. The common naming standard is business operation prefix + business data, such as key = “user. Add.” + uservo. Tostring(); key=”withdraw.”+userId。

    Counterexample: key = withdraw;. It’s a beat. The granularity of the lock is too large. During the execution of a user’s withdrawal operation, no other user can perform the withdrawal operation. For the general system cash out scenario, it is obviously unreasonable.
    Counter example: key = “withdraw.” + userid + dateutils. Format (New date(), “yyyy MM DD HH: mm: SS”);. Because the time stamp is added, the withdrawal request of the same user not at this time can be re locked, so the effect of distributed lock can not be achieved. Similarly, the unique ID generated by UUID or snowflake algorithm can not achieve the control effect of distributed lock.

  1. The validity of the lock
  2. As will be mentioned in the following code, the code to release the lock is executed in finally to ensure that the lock can still be released when the program is interrupted due to an exception. However, the JVM also causes finally not to be executed. The locking time needs to be evaluated as a conservative value according to the execution time of business logic. If it is too short, it will lead to reentry fault; If the lock is not released in time for too long, other requests cannot enter, and deadlock will occur.

  3. The atomicity of locking
  4. Some locking schemes use setnx (lockkey, requestid) and expire (lockkey, expireTime) of jedis, because they are two commands, so they are not atomic. If the program crashes suddenly after executing setnx, the lock will exist all the time and eventually lead to deadlock.

  5. About release lock
  6. The principle of releasing the lock: it is necessary to tie the bell to release the lock. In other words, client a (thread a) is locked and can only be unlocked by client a (thread a). The operation of releasing the lock should also be atomic. In addition, as mentioned above, the code to release the lock is executed in finally code.

 

Implementation of redis distributed lock

Class diagram:

 

RedisDistributedLock

package com.emax.zhenghe.rpcapi.provider.config.distributeRedisLock;

import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.data.redis.core.RedisTemplate;
import org.springframework.data.redis.core.script.DefaultRedisScript;
import org.springframework.stereotype.Component;

import javax.annotation.Resource;
import java.util.Arrays;
import java.util.UUID;
import java.util.concurrent.TimeUnit;

@Component
@Slf4j
public class RedisDistributedLock extends AbstractDistributedLock {

    @Autowired
    @Resource
    private RedisTemplate redisTemplate;

    private ThreadLocal lockFlag = new ThreadLocal();

    public static final String UNLOCK_LUA;

    static {
        StringBuilder sb = new StringBuilder();
        sb.append("if redis.call(\"get\",KEYS[1]) == ARGV[1] ");
        sb.append("then ");
        sb.append("    return redis.call(\"del\",KEYS[1]) ");
        sb.append("else ");
        sb.append("    return 0 ");
        sb.append("end ");
        UNLOCK_LUA = sb.toString();
    }


    public RedisDistributedLock() {
        super();
    }

    @Override
    public boolean lock(String key, long expire, int retryTimes, long sleepMillis) {
        boolean result = setRedis(key, expire);
        //If the lock acquisition fails, it will be retried according to the number of incoming retries
        while ((!result) && retryTimes-- > 0) {
            try {
                log.debug("lock failed, retrying..." + retryTimes);
                Thread.sleep(sleepMillis);
            } catch (InterruptedException e) {
                return false;
            }
            result = setRedis(key, expire);
        }
        return result;
    }

    /**
     *
     * @param key
     * @param expire MILLISECONDS
     * @return
     */
    private boolean setRedis(final String key, final long expire) {
        try {
            String uuid = UUID.randomUUID().toString();
            lockFlag.set(uuid);
            return redisTemplate.opsForValue().setIfAbsent(key,uuid,expire,TimeUnit.MILLISECONDS);
        } catch (Exception e) {
            log.info("redis lock error.", e);
        }
        return false;
    }


    @Override
    public boolean releaseLock(String key) {
        //When releasing the lock, it is possible that the execution time of the method after holding the lock is longer than the validity period of the lock. At this time, the lock may have been held by another thread, so it cannot be deleted directly
        try {
            DefaultRedisScript defaultRedisScript = new DefaultRedisScript(UNLOCK_LUA,Boolean.class);
            return redisTemplate.execute(defaultRedisScript,Arrays.asList(key),lockFlag.get());
        } catch (Exception e) {
            log.error("release lock occured an exception", e);
        } finally {
            //Clear the data in ThreadLocal to avoid memory overflow
            lockFlag.remove();
        }
        return false;
    }
}

 

AbstractDistributedLock

package com.emax.zhenghe.rpcapi.provider.config.distributeRedisLock;

public abstract class AbstractDistributedLock implements DistributedLock {
 
    @Override
    public boolean lock(String key) {
        return lock(key , TIMEOUT_MILLIS, RETRY_TIMES, SLEEP_MILLIS);
    }
 
    @Override
    public boolean lock(String key, int retryTimes, long sleepMillis) {
        return lock(key, TIMEOUT_MILLIS, retryTimes, sleepMillis);
    }
 
    @Override
    public boolean lock(String key, long expire) {
        return lock(key, expire, RETRY_TIMES, SLEEP_MILLIS);
    }
 
    @Override
    public boolean lock(String key, long expire, int retryTimes) {
        return lock(key, expire, retryTimes, SLEEP_MILLIS);
    }
 
}

 

DistributedLock 

package com.emax.zhenghe.rpcapi.provider.config.distributeRedisLock;

public interface DistributedLock {
    
     long TIMEOUT_MILLIS = 30000;
    
     int RETRY_TIMES = 2;
    
     long SLEEP_MILLIS = 500;
    
     boolean lock(String key);boolean lock(String key, int retryTimes, long sleepMillis);
    
     boolean lock(String key, long expire);
    
     boolean lock(String key, long expire, int retryTimes);
    
     boolean lock(String key, long expire, int retryTimes, long sleepMillis);
    
     boolean releaseLock(String key);
}

 

Call:

    @Autowired
    private RedisDistributedLock distributedLock;

    @Test
    public void lock11() throws Exception {
        String key = "examplekey" + System.currentTimeMillis();
        try {
            boolean lock = distributedLock.lock(key, 2000L, 1, 100L);
            log.info("===================" + lock);
        } finally {
            distributedLock.releaseLock(key);
        }
    }

 

 

Further encapsulation to achieve code decoupling

The above lock and release lock are exposed to the business caller, which increases the responsibility of the business caller. At the same time, if it is used improperly, it will cause bugs.

Next, let’s do a little refactoring. Take a look at the redislocktemplate below

package com.emax.zhenghe.rpcapi.provider.config.distributeRedisLock;

import lombok.extern.slf4j.Slf4j;
import org.apache.commons.lang3.StringUtils;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Component;

/**
 *Redis distributed lock concurrency control template class
 *
 * @author zhangguozhan
 */
@Slf4j
@Component
public class RedisLockTemplate {
    @Autowired
    private RedisDistributedLock redisDistributedLock;

    /**
     *Redis distributed lock control
     *
     *@ param key lock name
     *Life cycle of @ param expirems lock, unit: ms
     *@ param redislockcallback method
     * @return
     */
    public Object execute(String key, long expireMS, RedisLockCallback redisLockCallback) {
        return execute(key, expireMS, redisLockCallback, true, 2);
    }

    /**
     *Redis distributed lock control
     *
     * @param key
     * @param expireMS
     * @param redisLockCallback
     *The @ paramisautoreleaselock callback method automatically releases the lock after execution
     * @return
     */
    public Result execute(String key, long expireMS, RedisLockCallback redisLockCallback,
                          boolean isAutoReleaseLock,
                          int retryTimes) {
        Log. Info ("redis distributed lock control key = {}", key ");
        if (StringUtils.isBlank(key)) {
            log.info("try lock failure:key is null");
            return null;
        }
        boolean lock = redisDistributedLock.lock(key, expireMS, retryTimes);
        if (lock) {
            try {
                Result o = redisLockCallback.doInRedisLock();
                return o;
            } finally {
                if (isAutoReleaseLock) {
                    redisDistributedLock.releaseLock(key);
                }
            }
        } else {
            Log. Info ("##########################;
            return Result.error(ResultCodeEnum.GET_ LOCK_ Fail, "do not initiate repeatedly");
        }

    }
}

 

Redislockcallback is a functional interface

package com.emax.zhenghe.rpcapi.provider.config.distributeRedisLock;

@FunctionalInterface
public interface RedisLockCallback {
    Result doInRedisLock();
}

 

In this way, the business call becomes very easy. For example:

Result result = redisLockTemplate.execute(key, 5000L, () -> {
            List billVos = batchInsert(list);
            return Result.ok(billVos);
});

 

 

About Ajax asynchronous request

Nowadays, web projects generally adopt the development mode of front-end and back-end separation, and the front-end program framework is in full bloom, such as Vue, nodejs and so on.

For the repeated submission caused by the user’s shaking hands, the server’s approach is to use the above distributed control. The processing of the non first request is interrupted because of the locking failure, and the front end receives the prompt of “do not repeat the submission”. I thought it might affect the user experience. After consulting front-end colleagues, it turned out that this was not the case.

I wrote a demo to simulate repeated submission. The page asynchronously initiates the same request repeatedly, and the server handles it repeatedly. The first time is to add a lock and process the request normally. The second time is to find that the lock already exists and the lock fails. The prompt of “do not submit repeatedly” will be returned directly. The page will receive two responses. However, the response is earlier than the first one because the second request fails to lock and the error prompt is returned directly. The logic for Ajax to judge the response is that if the response is successful (the normal response is regarded as successful), the corresponding subsequent processing will be triggered; if it is failed (do not submit repeatedly is regarded as failure), the toast prompt will be given. Therefore, although toast is done for a moment, the page logic will be processed normally after the response of the first request comes.

 

Therefore, the above anti weight mechanism is also a more suitable scheme.

Of course, there must be business logic that should be verified, especially data verification. It’s part of the business.

 

The code of this article has been put into GitHub https://github.com/buguge/api-idempotent.git

For the front-end page asynchronous request API, please refer to the code: https://github.com/buguge/api-idempotent/blob/master/mideng/src/main/webapp/user.jsp