Analysis of current limiting

Time:2021-7-17

preface

Each system has a processing peak when doing pressure test. When it is close to the peak and continues to accept requests, the whole system will respond slowly; In order to protect the system, we need to refuse to handle the overload request. This is the current limiting that we will introduce below. We can protect the system by setting a peak threshold to limit the request to reach this peak; Some of our common middleware, such as tomcat, mysql, redis, have similar restrictions.

Current limiting algorithm

When doing current limiting, we have some common current limiting algorithms, including: counter current limiting, token bucket current limiting, leakage bucket current limiting;

  • 1. Token bucket current limiting

The principle of token bucket algorithm is that the system puts the token into the bucket at a certain rate, and discards the token when it is full; When the request comes, the token will be taken out from the bucket first. If the token can be obtained, the request can be completed continuously, otherwise, the service will be refused; The token bucket allows a certain degree of burst traffic, which can be handled as long as there is a token, and supports multiple tokens at one time;

  • 2. Leakage bucket current limiting

The principle of leaky bucket algorithm is to flow out requests according to a fixed constant rate and flow in requests at any rate. When the number of requests exceeds the capacity of the bucket, new requests will wait or be rejected; It can be seen that the leaky bucket algorithm can forcibly limit the data transmission speed;

  • 3. Current limiting of counter

Counter is a relatively simple and crude algorithm, which is mainly used to limit the total number of concurrency, such as database connection pool, thread pool and seckill; If the total number of requests in a certain period of time exceeds the set threshold, the counter will limit the current;

How to limit current

After understanding the current limiting algorithm, we need to know where and how to limit the current; For a system, we can often limit the current in the access layer. In most cases, we can use nginx, openresty and other middleware to deal with it directly; We can also limit the current in the business layer. We need to use the relevant current limiting algorithm according to our different business needs.

Business layer current limiting

For the business layer, we may be single node, multi node user bound, or multi node unbound; At this time, we need to distinguish between in-process flow restriction and distributed flow restriction.

In process current limiting

For in-process current limiting, it is relatively simple. Guava is a sharp tool that we often use. Let’s see how to limit the total concurrency of the interface, the number of requests in a certain time window, and how to use token bucket algorithm and leaky bucket algorithm for smoother current limiting;

  • Limit the total concurrency of the interface

You only need to configure a total concurrency, then use a calculator to record each request, and then compare it with the total concurrency

private static int max = 10;
private static AtomicInteger limiter = new AtomicInteger();

if (limiter.incrementAndGet() > max){
    System. Err. Println ("maximum limit exceeded");
    return;
}
  • Limit time window requests

To limit the number of requests of an interface within a specified time, we can use the cache of guava to cache the counter, and then set the expiration time; For example, set the maximum request per minute as 100:

LoadingCache<Long, AtomicLong> counter = CacheBuilder.newBuilder().expireAfterWrite(1, TimeUnit.MINUTES).build(new CacheLoader<Long, AtomicLong>() {
        @Override
        public AtomicLong load(Long key) throws Exception {
            return new AtomicLong(0);
        }
});

private static int max = 100;
long curMinutes = System.currentTimeMillis() / 1000 * 60;
if (counter.get(curMinutes).incrementAndGet() > max) {
    System. Err. Println ("the number of time window requests exceeds the upper limit");
    return;
}

The expiration time is one minute, and it will be cleared automatically every minute; For example, there are no messages in the first 59 seconds, and 200 messages arrive at 60 seconds. At this time, 100 messages are accepted, and the counter is cleared to 0, and then 100 messages are accepted; This situation can be solved by referring to the idea of TCP sliding window.

  • Smooth current limiting request

The way of counter is rough. The token bucket and leaky bucket current limiting algorithms are relatively smooth. You can directly use the ratelimiter class provided by guava

RateLimiter limiter = RateLimiter.create(2);
System.out.println(limiter.acquire(4));
System.out.println(limiter.acquire());
System.out.println(limiter.acquire());
System.out.println(limiter.acquire(2));
System.out.println(limiter.acquire());
System.out.println(limiter.acquire());

Create (2) indicates that the bucket capacity is 2 and 2 tokens are added every second, that is, a token is added in 500 ms. acquire() indicates that a token is obtained from the bucket. The return value is the waiting time. The output result is as follows:

0.0
1.998633
0.49644
0.500224
0.999335
0.500186

It can be seen that this algorithm allows certain emergencies. The waiting time for acquiring four tokens for the first time is 0, and it needs to wait 2 seconds for subsequent acquisition, and it takes 500 milliseconds for each subsequent acquisition.

Distributed current limiting

Now most systems adopt multi node deployment, so a business may be processed in multiple processes, so distributed current limiting is essential. For example, a common second kill system may have n business logic nodes at the same time;
The conventional way is to use redis + Lua and openresty + Lua to realize the current limiting service, making it atomized, while ensuring high performance; Both redis and openresty are well-known for their high performance. At the same time, they also provide atomization solutions, as shown below;

  • Redis+lua

Redis processes messages in a single thread on the server side and supports the execution of lua scripts. The logic related to current limiting can be implemented with Lua scripts to ensure atomicity. The general implementation is as follows:

--Current limiting key
local key = KEYS[1]
--Current limiting size
local limit = tonumber(ARGV[1])
--Expiration time
local expire = tonumber(ARGV[2])

local current = tonumber(redis.call('get',key) or "0")

if current + 1 > limit then
    return 0;
else
    redis.call("INCRBY", key, 1)
    redis.call("EXPIRE", key, expire)
    return current + 1
end

The counter algorithm is used to realize the current limiting. The current limiting key, the current limiting size and the validity period of the key can be passed in the place where Lua is called; If the return result is 0, it means that the current limit is exceeded; otherwise, the current accumulated value is returned.

  • OpenResty+lua

The core of openresty is nginx, but many third-party modules, NGX, are added to it_ Lua module embeds Lua into nginx, which makes nginx a web server; There are other common development modules, such as Lua rest lock, Lua rest limit traffic, Lua rest memcached, Lua rest mysql, Lua rest redis, etc;
In this section, we first use Lua rest lock module to implement a simple counter current limiting. The relevant Lua code is as follows:

local locks = require "resty.lock";

local function acquire()
    local lock = locks:new("locks");
    local elapsed, err = lock:lock("limit_key");
    local limit_counter = ngx.shared.limit_counter;
    --Get client IP
    local key = ngx.var.remote_addr;
    --Current limiting size
    local limit = 5; 
    local current = limit_counter:get(key);
    
    --Print key and current value
    ngx.say("key="..key..",value="..tostring(current));
    
    if current ~= nil and current + 1 > limit then 
       lock:unlock();
       return 0;
    end
    
    if current == nil then 
       limit_ counter:set(key,1,5); -- Set the expiration time to 5 seconds
    else 
       limit_counter:incr(key,1);
    end
    lock:unlock();
    return 1;
end

The above is an example of IP current limiting. Due to the need to ensure atomicity, the rest.lock module is used. At the same time, similar to redis, the expiration time is reset. In addition, we need to pay attention to the release of the lock; You also need to set up two shared dictionaries

http {
    ...
    #lua_ shared_ Dict < name > < size > defines a shared memory space named name, and the memory size is size; The shared memory object defined by this command is visible to all worker processes in nginx
    lua_shared_dict locks 10m;
    lua_shared_dict limit_counter 10m;
}

Access layer current limiting

The access layer is usually the traffic entrance. Nginx is used as the traffic entrance by many systems. Of course, openresty is no exception, and openresty provides more powerful functions. For example, Lua resty limit traffic module, which will be introduced here, is a powerful current limiting module; Before using Lua rest limit traffic, let’s take a general look at how to use openrest;

Openresty installation and use

  • Download installation configuration

You can download it directly from the official website http://openresty.org/en/download.html , start, overload and stop commands are as follows:

nginx.exe
nginx.exe -s reload
nginx.exe -s stop

Open the IP + port, you can see: welcome to openresty! That is to say, it is started successfully;

  • Lua script instance

First, you need to configure the following in the HTTP directory of nginx.conf:

http {
    ...
    lua_ package_ path "/lualib/?. lua;;";  # Lua module  
    lua_ package_ cpath "/lualib/?. so;;";  # C module   
    include lua.conf;   # Import custom Lua profile
}

A lua.conf is defined here. All requests related to Lua are configured in the same path as nginx.conf; Take a test.lua as an example. Lua.conf is configured as follows:

#lua.conf  
server {  
    charset utf-8; # Set encoding
    listen       8081;  
    server_name  _;  
    location /test {  
        default_type 'text/html';  
        content_by_lua_file lua/api/test.lua;
    } 
}

Here, all Lua files are placed in the Lua / API directory, such as the simplest Hello World:

ngx.say("hello world");

Lua rest limit traffic module

Lua rest limit traffic provides three ways to limit the maximum number of concurrent connections, the number of time window requests, and the number of smooth limit requests, corresponding to rest.limit.conn, rest.limit.count, and rest.limit.req; Related documents can be found directly in pod / Lua rest limit traffic with complete examples;

The following three shared dictionaries will be used and configured under HTTP in advance:

http {
    lua_shared_dict my_limit_conn_store 100m;
    lua_shared_dict my_limit_count_store 100m;
    lua_shared_dict my_limit_req_store 100m;
}
  • Limit the maximum number of concurrent connections

The provided rest.limit.conn limits the maximum number of connections. The specific script is as follows:

local limit_conn = require "resty.limit.conn"

--B<syntax:> C<obj, err = class.new(shdict_name, conn, burst, default_conn_delay)>
local lim, err = limit_conn.new("my_limit_conn_store", 1, 0, 0.5)
if not lim then
    ngx.log(ngx.ERR,
            "failed to instantiate a resty.limit.conn object: ", err)
    return ngx.exit(500)
end

local key = ngx.var.binary_remote_addr
local delay, err = lim:incoming(key, true)
if not delay then
    if err == "rejected" then
        return ngx.exit(502)
    end
    ngx.log(ngx.ERR, "failed to limit req: ", err)
    return ngx.exit(500)
end

if lim:is_committed() then
    local ctx = ngx.ctx
    ctx.limit_conn = lim
    ctx.limit_conn_key = key
    ctx.limit_conn_delay = delay
end

local conn = err

if delay >= 0.001 then
    ngx.sleep(delay)
end

The new () parameters are: dictionary name, maximum number of concurrent requests allowed, number of burst connections allowed, and connection delay;
Commit in incoming() is a Boolean value. When true, it means to record the number of current requests, otherwise it will be run directly;
Return value: if the request does not exceed the conn value specified in the method, the method returns 0 as the number of concurrent requests (or connections) with delay and current time;

  • Limit time window requests

The rest.limit.count provided can limit the number of requests within a time window. The specific script is as follows:

local limit_count = require "resty.limit.count"

--B<syntax:> C<obj, err = class.new(shdict_name, count, time_window)>
--The rate is limited to 20 / 10s
local lim, err = limit_count.new("my_limit_count_store", 20, 10)
if not lim then
    ngx.log(ngx.ERR, "failed to instantiate a resty.limit.count object: ", err)
    return ngx.exit(500)
end

local local key = ngx.var.binary_remote_addr
--B<syntax:> C<delay, err = obj:incoming(key, commit)>
local delay, err = lim:incoming(key, true)

if not delay then
    if err == "rejected" then
        return ngx.exit(503)
    end
    ngx.log(ngx.ERR, "failed to limit count: ", err)
    return ngx.exit(500)
end

The three parameters specified in new () are: dictionary name, specified request threshold, number of requests, window time before reset, in seconds;
Commit in incoming() is a Boolean value. When true, it means to record the number of current requests, otherwise it will be run directly;
Return value: if the number of requests is within the limit, the delay of the current request and the remaining number of requests to be processed are returned;

  • Smooth limit requests

The resty.limit.req provided can limit requests in a smoother way. The specific scripts are as follows:

local limit_req = require "resty.limit.req"

--B<syntax:> C<obj, err = class.new(shdict_name, rate, burst)>
--It is limited to 200 requests per second and gives 100 requests per second; In other words, the maximum number of requests per second can be between 200-300, and an error will be reported if it exceeds 300
local lim, err = limit_req.new("my_limit_req_store", 200, 100)
if not lim then
    ngx.log(ngx.ERR,
            "failed to instantiate a resty.limit.req object: ", err)
    return ngx.exit(500)
end

local key = ngx.var.binary_remote_addr
local delay, err = lim:incoming(key, true)
if not delay then
    if err == "rejected" then
        return ngx.exit(503)
    end
    ngx.log(ngx.ERR, "failed to limit req: ", err)
    return ngx.exit(500)
end

if delay >= 0.001 then
    local excess = err
    ngx.sleep(delay)
end

The three parameters of new () are: dictionary name, request rate (per second) threshold, and the number of allowed delayed requests per second;
Commit in incoming() is a Boolean value. When true, it means to record the number of current requests. Otherwise, it will run directly, which can be understood as a switch;
Return value: if the number of requests is within the limit, this method returns 0 as the delay of the current time and the (zero) number of excessive requests per second;

For more information, see the official document: pod / Lua rest limit traffic directory

summary

This paper first introduces the common current limiting algorithms, and then introduces how to limit the current in the process and distributed applications of the service layer. Finally, the access layer uses the Lua rest limit traffic module of openrest to limit the current.

Thank you for your attention

WeChat official account “Roll back the code“The first time to read, articles continue to update; Focus on Java source code, architecture, algorithm and interview.

Recommended Today

The selector returned by ngrx store createselector performs one-step debugging of fetching logic

Test source code: import { Component } from ‘@angular/core’; import { createSelector } from ‘@ngrx/store’; export interface State { counter1: number; counter2: number; } export const selectCounter1 = (state: State) => state.counter1; export const selectCounter2 = (state: State) => state.counter2; export const selectTotal = createSelector( selectCounter1, selectCounter2, (counter1, counter2) => counter1 + counter2 ); // […]