High concurrency and elegant current limiting

Time:2021-2-25

High concurrency and elegant current limiting

technical analysis

If you pay more attention to the current technology form, you will know that microservices are in a mess. Of course, things have two sides, and microservices are not the master key to solve the problems of technology and architecture. If the advantages of servitization outweigh the disadvantages, it is recommended that the system be servitized. With the continuous evolution of the service-oriented process, various concepts and technologies follow. Any solution exists to solve the problem. For example: fuse design, interface idempotent design, retrial mechanism design, and today’s current limiting design, and so on, these technologies are almost full of every system.

As for today’s current limiting, the written meaning and function are the same, which is to limit the speed of concurrent access or requests or requests within a time window to protect the system. Once the critical point is reached, the existing system can be protected by denial of service, queuing, or waiting to avoid avalanche.

Traffic restriction is like the subway in the imperial capital. If you live in Xierqi or Tiantongyuan, you may have a deeper understanding. I’m more used to expounding from the perspective of technology and consumers. The general reason for current limiting is the limited ability of consumers. The purpose is to avoid system failure beyond the ability of consumers. Of course, there are other similar situations that can be solved by current limiting.

Most of the forms of current limiting can be divided into two categories

  1. Limit the number of consumers. It can also be said that the maximum capacity value of consumption. For example: the database connection pool focuses on the total number of connections. There is also the thread pool previously written by Caicai, which essentially limits the maximum consumption power of consumers.
  2. The number of requests that can be consumed. The number here can be instantaneous concurrency or total concurrency over a period of time. It’s the same dish that I’m going to help YY make today.

In addition, there are other forms of current limiting, such as current limiting according to network traffic and CPU utilization. According to the scope of current limiting, it can be divided into distributed current limiting, application current limiting, interface current limiting, etc. No matter how it changes, the current limit can be represented by the following figure:

High concurrency and elegant current limiting

Implementation of common technologies

Token Bucket

The token bucket is a bucket for storing fixed capacity tokens. A token is added to the bucket at a fixed rate. When the bucket is full, the token is discarded. Whether the request is processed depends on whether the token in the bucket is enough. When the number of cards is reduced to zero, the new request is rejected. The token bucket allows a certain degree of burst traffic, which can be handled as long as there is a token. It supports taking more than one token at a time. The token bucket is filled with tokens.
High concurrency and elegant current limiting

Leaky bucket algorithm

Leaky bucket is a leaky bucket with a fixed capacity. The requests flow out at a constant rate, and the incoming requests flow in at any rate. When the number of incoming requests accumulates to the leaky bucket capacity, the new incoming requests are rejected. The leaky bucket can be regarded as a queue with fixed capacity and fixed outflow rate, and the leaky bucket limits the outflow rate of requests. The leaky bucket is filled with requests.
High concurrency and elegant current limiting

Counter

Sometimes we also use counter to limit current, which is mainly used to limit the total concurrency in a certain period of time, such as the concurrency of database connection pool, thread pool and seckill. Counter current limiting is a simple and crude total quantity current limiting rather than average rate current limiting as long as the total number of requests in a certain period of time exceeds the set threshold.

In addition, in fact, according to different business scenarios, there are many different current limiting algorithms, but the general rule is only one: as long as the current business scenario of the current current current limiting strategy is the best

Other basic knowledge of current limiting, please Baidu!!

Another way to solve the sister problem

Regression problem, YY sister’s problem, Caicai is not going to use the above algorithms to help her. Cai Cai is going to limit the flow by limiting the total number of requests according to the time period. The general idea is as follows:

  1. A ring is used to represent the passing request container.
  2. A pointer is used to point to the location index of the current request to judge the time difference between the current request time and the last request time of the current location, so as to judge whether it is restricted or not.
  3. If the request passes, the current pointer moves forward one position. If it does not pass, it does not move
  4. Repeat the above steps until forever

High concurrency and elegant current limiting

Code is the way to speak

The following code can be used in the production environment with no or slight modifications

The core idea of the following code is as follows: the difference between the time element of the current position and the current time of the pointer determines whether to allow this request, so that the request passed is more smooth in time.

Thinking is far more important than language. Any language can do it. Please implement it by PHPer, golanger and javaer

//The current limiting component uses an array as a ring
    class LimitService
    {
        //The position of the current pointer
        int currentIndex = 0;
        //The number of seconds of the limited time, i.e. how many requests are allowed in x seconds
        int limitTimeSencond = 1;
        //The container array of the request ring
        DateTime?[] requestRing = null;
        //Lock when the container changes or moves the pointer
        object objLock = new object();

        public LimitService(int countPerSecond,int  _limitTimeSencond)
        {
            requestRing = new DateTime?[countPerSecond];
            limitTimeSencond= _limitTimeSencond;
        }

        //Can the program continue
        public bool IsContinue()
        {
            lock (objLock)
            {
                var currentNode = requestRing[currentIndex];
                //If the value of the current node plus the set second exceeds the current time, the limit is exceeded
                if (currentNode != null&& currentNode.Value.AddSeconds(limitTimeSencond) >DateTime.Now)
                {
                    return false;
                }
                //The current node is set to the current time
                requestRing[currentIndex] = DateTime.Now;
                //The pointer moves one position
                MoveNextIndex(ref currentIndex);
            }            
            return true;
        }
        //Change the number of requests that can be passed per second
        public bool ChangeCountPerSecond(int countPerSecond)
        {
            lock (objLock)
            {
                requestRing = new DateTime?[countPerSecond];
                currentIndex = 0;
            }
            return true;
        }

        //The pointer moves forward one position
        private void MoveNextIndex(ref int currentIndex)
        {
            if (currentIndex != requestRing.Length - 1)
            {
                currentIndex = currentIndex + 1;
            }
            else
            {
                currentIndex = 0;
            }
        }
    }

The test procedure is as follows:

static  LimitService l = new LimitService(1000, 1);
        static void Main(string[] args)
        {
            int threadCount = 50;
            while (threadCount >= 0)
            {
                Thread t = new Thread(s =>
                {
                    Limit();
                });
                t.Start();
                threadCount--;
            }           

            Console.Read();
        }

        static void Limit()
        {
            int i = 0;
            int okCount = 0;
            int noCount = 0;
            Stopwatch w = new Stopwatch();
            w.Start();
            while (i < 1000000)
            {
                var ret = l.IsContinue();
                if (ret)
                {
                    okCount++;
                }
                else
                {
                    noCount++;
                }
                i++;
            }
            w.Stop();
            Console.WriteLine ($"share {w.elassedmilliseconds}, allow: {okcount}, intercept: {NOCOUNT}");
        }

The results are as follows

High concurrency and elegant current limiting

High concurrency and elegant current limiting

It takes up to 15 seconds to process 1000000 * 50 = 50000000 requests
There is no GC operation, and the memory utilization rate is very low, processing 3 million + requests per second. The above program is modified to 10 threads, which takes about 4 seconds
High concurrency and elegant current limiting

If it is a strong server or a small number of threads, the processing speed will be faster

Write at the end

Although the above code is simple, it is the core code of flow limitation (in fact, there is room for optimization). After other encapsulation, it can be applied to the filter of Web API or other scenarios. My sister’s problem is solved. Do you want her to treat me to dinner?

More wonderful articles

High concurrency and elegant current limiting