Implementation method of c# request uniqueness verification supporting high concurrency

Time:2021-7-28

Usage scenario description:

The sending request is often encountered in the network request. The server response is successful, but a network failure occurs when returning, resulting in the client unable to receive the request result. Then the client program may judge as a network failure and send the same request repeatedly. Of course, if the request result query interface is defined in the interface, the repetition will be relatively less. Especially for transaction data, this operation needs to avoid sending requests repeatedly. In another case, if the user clicks the interface button too quickly to generate continuous requests for the same content, the back-end also needs to be filtered. This generally occurs in the system docking and cannot control the business logic of the third-party system. It needs to be defined from its own business logic.

Description of other requirements:

This kind of request generally has the characteristics of time range and high concurrency, that is, repeated requests will occur in a short time. Therefore, it is necessary to support high concurrency for the module.

Technical realization:

MD5 summary is performed on the requested business content, and the MD5 summary is stored in the cache. Each request data is judged through this public called method.

Code implementation:

The common calling code uniquecheck uses the singleton mode to create a unique object, which is convenient to access only one unified cache library when calling multiple threads

/*
   *Volatile is like the more familiar const. Volatile is a type specifier.
   *It is designed to modify variables accessed and modified by different threads.
   *Without volatile, it will basically lead to the result that either you can't write multithreaded programs, or the compiler loses a lot of optimization opportunities.
   */
  private static readonly object lockHelper = new object();
 
  private volatile static UniqueCheck _instance;  
 
  /// <summary>
  ///Get single instance
  /// </summary>
  /// <returns></returns>
  public static UniqueCheck GetInstance()
  {
   if (_instance == null)
   {
    lock (lockHelper)
    {
     if (_instance == null)
      _instance = new UniqueCheck();
    }
   }
   return _instance;
  }

Note the volatile modifier here. In the actual test process, if there is no such modifier, an error will be reported in the case of high concurrency.

Customize a queue that can be processed concurrently. The code is as follows: concurrentlinkedqueue

using System;
using System.Collections.Generic;
using System.Text;
using System.Threading;

namespace PackgeUniqueCheck
{
 /// <summary>
 ///Non locked concurrent queue, processing less than 100 concurrent queues
 /// </summary>
 /// <typeparam name="T"></typeparam>
 public class ConcurrentLinkedQueue<T>
 {
  private class Node<K>
  {
   internal K Item;
   internal Node<K> Next;

   public Node(K item, Node<K> next)
   {
    this.Item = item;
    this.Next = next;
   }
  }

  private Node<T> _head;
  private Node<T> _tail;

  public ConcurrentLinkedQueue()
  {
   _head = new Node<T>(default(T), null);
   _tail = _head;
  }

  public bool IsEmpty
  {
   get { return (_head.Next == null); }
  }
  /// <summary>
  ///Enter queue
  /// </summary>
  /// <param name="item"></param>
  public void Enqueue(T item)
  {
   Node<T> newNode = new Node<T>(item, null);
   while (true)
   {
    Node<T> curTail = _tail;
    Node<T> residue = curTail.Next;

    //Judge_ Is the tail changed by other processes
    if (curTail == _tail)
    {
     //A has other processes successfully executed C_ The tail should point to the new node
     if (residue == null)
     {
      //C. other processes have changed the tail node and need to retrieve the tail node again
      if (Interlocked.CompareExchange<Node<T>>(
       ref curTail.Next, newNode, residue) == residue)
      {
       //D try to modify the tail
       Interlocked.CompareExchange<Node<T>>(ref _tail, newNode, curTail);
       return;
      }
     }
     else
     {
      //B helps other threads complete D operation
      Interlocked.CompareExchange<Node<T>>(ref _tail, residue, curTail);
     }
    }
   }
  }
  /// <summary>
  ///Queue fetch data
  /// </summary>
  /// <param name="result"></param>
  /// <returns></returns>
  public bool TryDequeue(out T result)
  {
   Node<T> curHead;
   Node<T> curTail;
   Node<T> next;
   while (true)
   {
    curHead = _head;
    curTail = _tail;
    next = curHead.Next;
    if (curHead == _head)
    {
     If (next = = null) // queue is empty
     {
      result = default(T);
      return false;
     }
     If (curhead = = curtail) // queue is in the process of the first node of enqueue
     {
      //Try to help other processes complete the operation
      Interlocked.CompareExchange<Node<T>>(ref _tail, next, curTail);
     }
     else
     {
      //The next.item must be placed before CAS
      result = next.Item;
      //If_ If the head has not changed, the_ Head points to next and exits
      if (Interlocked.CompareExchange<Node<T>>(ref _head,
       next, curHead) == curHead)
       break;
     }
    }
   }
   return true;
  }
  /// <summary>
  ///Trying to get the last object
  /// </summary>
  /// <param name="result"></param>
  /// <returns></returns>
  public bool TryGetTail(out T result)
  {
   result = default(T);
   if (_tail == null)
   {
    return false;
   }
   result = _tail.Item;
   return true;
  }
 }
}

Although it is a very simple uniqueness verification logic, in order to achieve high efficiency, high concurrency support, high reliability and low memory occupation, such requirements need to be realized and detailed simulation tests need to be done.

using System;
using System.Collections.Generic;
using System.Text;
using System.Threading;
using System.Collections;

namespace PackgeUniqueCheck
{
 public class UniqueCheck
 {
  /*
   *Volatile is like the more familiar const. Volatile is a type specifier.
   *It is designed to modify variables accessed and modified by different threads.
   *Without volatile, it will basically lead to the result that either you can't write multithreaded programs, or the compiler loses a lot of optimization opportunities.
   */
  private static readonly object lockHelper = new object();

  private volatile static UniqueCheck _instance;  

  /// <summary>
  ///Get single instance
  /// </summary>
  /// <returns></returns>
  public static UniqueCheck GetInstance()
  {
   if (_instance == null)
   {
    lock (lockHelper)
    {
     if (_instance == null)
      _instance = new UniqueCheck();
    }
   }
   return _instance;
  }

  private UniqueCheck()
  {
   //Create a thread safe hash table as a dictionary cache
   _DataKey = Hashtable.Synchronized(new Hashtable());
   Queue myqueue = new Queue();
   _DataQueue = Queue.Synchronized(myqueue);
   _Myqueue = new ConcurrentLinkedQueue<string>();
   _Timer = new Thread(DoTicket);
   _Timer.Start();
  }

  #Region public property settings
  /// <summary>
  ///Set the sleep time length of the timed thread: the default is 1 minute
  ///Time range: 1-7200000, with values from 1 ms to 2 hours
  /// </summary>
  /// <param name="value"></param>
  public void SetTimeSpan(int value)
  {
   if (value > 0&& value <=7200000)
   {
    _TimeSpan = value;
   }
  }
  /// <summary>
  ///Set the maximum number of records in the cache
  ///Value range: 1-5000000,1 to 5 million
  /// </summary>
  /// <param name="value"></param>
  public void SetCacheMaxNum(int value)
  {
   if (value > 0 && value <= 5000000)
   {
    _CacheMaxNum = value;
   }
  }
  /// <summary>
  ///Sets whether logs are displayed in the console
  /// </summary>
  /// <param name="value"></param>
  public void SetIsShowMsg(bool value)
  {
   Helper.IsShowMsg = value;
  }
  /// <summary>
  ///Thread request blocking increment
  ///Value range: 1-cachemaxnum. It is recommended to set it to 10% - 20% of the maximum cache value
  /// </summary>
  /// <param name="value"></param>
  public void SetBlockNumExt(int value)
  {
   if (value > 0 && value <= _CacheMaxNum)
   {
    _BlockNumExt = value;
   }
  }
  /// <summary>
  ///Request blocking time
  ///Value range: 1-max, set the requested blocking time according to the blocking increment
  ///The longer the blocking time, the larger the blocking increment can be set, but the worse the real-time response of the request
  /// </summary>
  /// <param name="value"></param>
  public void SetBlockSpanTime(int value)
  {
   if (value > 0)
   {
    _BlockSpanTime = value;
   }
  }
  #endregion

  #Region private variable
  /// <summary>
  ///Internal running thread
  /// </summary>
  private Thread _runner = null;
  /// <summary>
  ///Can handle high concurrency queues
  /// </summary>
  private ConcurrentLinkedQueue<string> _Myqueue = null;
  /// <summary>
  ///Time key value pair of unique content
  /// </summary>
  private Hashtable _DataKey = null;
  /// <summary>
  ///Content time queue
  /// </summary>
  private Queue _DataQueue = null;
  /// <summary>
  ///Sleep time length of timed thread: 1 minute by default
  /// </summary>
  private int _TimeSpan = 3000;
  /// <summary>
  ///Timer thread
  /// </summary>
  private Thread _Timer = null;
  /// <summary>
  ///Maximum number of records in cache
  /// </summary>
  private int _CacheMaxNum = 500000;
  /// <summary>
  ///Thread request blocking increment
  /// </summary>
  private int _BlockNumExt = 10000;
  /// <summary>
  ///Request blocking time
  /// </summary>
  private int _BlockSpanTime = 100;
  #endregion

  #Region private method
  private void StartRun()
  {
   _runner = new Thread(DoAction);
   _runner.Start();
   Helper. Showmsg ("internal thread started successfully!");
  }

  private string GetItem()
  {
   string tp = string.Empty;
   bool result = _Myqueue.TryDequeue(out tp);
   return tp;
  }
  /// <summary>
  ///Perform loop operation
  /// </summary>
  private void DoAction()
  {
   while (true)
   {
    while (!_Myqueue.IsEmpty)
    {
     string item = GetItem();
     _DataQueue.Enqueue(item);
     if (!_DataKey.ContainsKey(item))
     {
      _DataKey.Add(item, DateTime.Now);
     }
    }
    //Helper. Showmsg ("the current array is empty, the processing thread enters sleep...);
    Thread.Sleep(2);
   }
  }
  /// <summary>
  ///Execute timer action
  /// </summary>
  private void DoTicket()
  {
   while (true)
   {
    Helper. Showmsg ("number of current data queues:" +_ DataQueue.Count.ToString());
    if (_DataQueue.Count > _CacheMaxNum)
    {
     while (true)
     {
      Helper. Showmsg (string. Format ("current number of queues: {0}, has exceeded the maximum length: {1}, start cleaning..."_ DataQueue.Count, _ CacheMaxNum.ToString()));
      string item = _DataQueue.Dequeue().ToString();
      if (!string.IsNullOrEmpty(item))
      {
       if (_DataKey.ContainsKey(item))
       {
        _DataKey.Remove(item);
       }
       if (_DataQueue.Count <= _CacheMaxNum)
       {
        Helper. Showmsg ("after cleaning, start hibernating cleaning thread...);
        break;
       }
      }
     }
    }
    Thread.Sleep(_TimeSpan);
   }
  }

  /// <summary>
  ///Thread sleep waiting
  ///If the current load pressure greatly exceeds the processing capacity of the thread
  ///Then a delayed call is required
  /// </summary>
  private void BlockThread()
  {
   if (_DataQueue.Count > _CacheMaxNum + _BlockNumExt)
   {
    Thread.Sleep(_BlockSpanTime);
   }
  }
  #endregion

  #Region public method
  /// <summary>
  ///Open service thread
  /// </summary>
  public void Start()
  {
   if (_runner == null)
   {
    StartRun();
   }
   else
   {
    if (_runner.IsAlive == false)
    {
     StartRun();
    }
   }

  }
  /// <summary>
  ///Close the service thread
  /// </summary>
  public void Stop()
  {
   if (_runner != null)
   {
    _runner.Abort();
    _runner = null;
   }
  }

  /// <summary>
  ///Add content information
  /// </summary>
  ///< param name = "item" > content information < / param >
  ///< returns > true: this value is not included in the cache, the queue is added successfully, false: this value is included in the cache, and the queue addition fails < / returns >
  public bool AddItem(string item)
  {
   BlockThread();
   item = Helper.MakeMd5(item);
   if (_DataKey.ContainsKey(item))
   {
    return false;
   }
   else
   {
    _Myqueue.Enqueue(item);
    return true;
   }
  }
  /// <summary>
  ///Judge whether the content information already exists
  /// </summary>
  ///< param name = "item" > content information < / param >
  ///< returns > true: the information already exists in the cache, false: the information does not exist in the cache < / returns >
  public bool CheckItem(string item)
  {
   item = Helper.MakeMd5(item);
   return _DataKey.ContainsKey(item);
  }
  #endregion 

 }
}

Simulation test code:

private static string _example = Guid.NewGuid().ToString();

  private static UniqueCheck _uck = null;

  static void Main(string[] args)
  {
   _uck = UniqueCheck.GetInstance();
   _uck.Start();
   _uck.SetIsShowMsg(false);
   _uck.SetCacheMaxNum(20000000);
   _uck.SetBlockNumExt(1000000);
   _uck.SetTimeSpan(6000);

   _uck.AddItem(_example);
   Thread[] threads = new Thread[20];

   for (int i = 0; i < 20; i++)
   {
    threads[i] = new Thread(AddInfo);
    threads[i].Start();
   }

   Thread checkthread = new Thread(CheckInfo);
   checkthread.Start();

   string value = Console.ReadLine();

   checkthread.Abort();
   for (int i = 0; i < 50; i++)
   {
    threads[i].Abort();
   }
   _uck.Stop();
  }

  static void AddInfo()
  {
   while (true)
   {
    _uck.AddItem(Guid.NewGuid().ToString());
   }
  }

  static void CheckInfo()
  {
   while (true)
   {
    Console. Writeline ("start time: {0}..., datetime. Now. ToString (" yyyy MM DD HH: mm: SS. Ffff ");
    Console. Writeline ("insert result: {0}"_ uck.AddItem(_ example));
    Console. Writeline ("end time: {0}", datetime. Now. ToString ("yyyy MM DD HH: mm: SS. Ffff");
          //Adjust the process sleep time to test high concurrency
    //Thread.Sleep(1000);
   }
   
  }

Test screenshot:

summary

The above is the whole content of this article. I hope the content of this article has certain reference and learning value for your study or work. Thank you for your support for developpaer.

Recommended Today

Implementation example of go operation etcd

etcdIt is an open-source, distributed key value pair data storage system, which provides shared configuration, service registration and discovery. This paper mainly introduces the installation and use of etcd. Etcdetcd introduction etcdIt is an open source and highly available distributed key value storage system developed with go language, which can be used to configure sharing […]