How to achieve fixed size of handwriting cache from scratch

Time:2020-11-10

Programmer’s three high

Some time ago, a colleague had a physical examination, and the physical examination doctor said that he was three high.

I joked, isn’t programmer three high performance, high concurrency and high availability? What are you?

Every developer who pursues performance is pursuing high performance, and cache is the only way for us to embark on this high performance road.

From CPU design to service distributed cache, we are in touch with cache all the time. Today we will learn how to develop cache and how to write a cache of specified size.

The development of cache

Ancient society – HashMap

When our application has a certain amount of traffic or query the database frequently, we can present our own HashMap or concurrent HashMap in Java.

We can write this in the code:

public class CustomerService {
    private HashMap<String,String> hashMap = new HashMap<>();
    private CustomerMapper customerMapper;
    public String getCustomer(String name){
        String customer = hashMap.get(name);
        if ( customer == null){
            customer = customerMapper.get(name);
            hashMap.put(name,customer);
        }
        return customer;
    }
}

But there is a problem with this. HashMap can’t eliminate data, and the memory will grow unlimited, so HashMap will be eliminated soon.

For example, we used to query redis, but we wanted to cache some hot data locally. Obviously, using HashMap can’t meet this demand.

Of course, you can use weak references here to solve the problem of memory growing all the time.

Of course, it doesn’t mean that it is totally useless. Just like in our ancient society, not everything is outdated. For example, the traditional virtues of our Chinese nation are never out of date,

Just like this HashMap, it can be used as a cache in some scenarios. When there is no need for elimination mechanism, for example, we use reflection. If we search method and field through reflection every time, the performance will be inefficient. At this time, we cache it with HashMap, and the performance can be improved a lot.

Modern society – lruhashmap

In ancient society, the problem that baffled us was unable to eliminate the data, which would lead to the unlimited expansion of our memory, which is obviously unacceptable to us.

Some people say that I will eliminate some data, which is not right, but how to eliminate it? Random elimination?

Of course not. Just imagine that you have just loaded a into the cache and will be eliminated the next time you access it. Then you will visit our database again. Why should we cache it?

Therefore, smart people have invented several elimination algorithms. Here are three common FIFO, LRU and LFU (as well as some arc and MRU that are interested in searching by themselves)

FIFO

In this elimination algorithm, the first in cache will be eliminated first. This is the simplest, but it will lead to a low percentage.

Imagine if we have a high frequency of data access is the first access to all data, and those not very high is later accessed, then we will squeeze out our first data, but its access frequency is very high.

LRU

The least recently used algorithm.

In this algorithm, the above problem is avoided. Every time we access the data, we will put it at the end of our team. If we need to eliminate the data, we only need to eliminate the team leader.

However, there is still a problem. If one data is accessed 10000 times in the first 59 minutes of an hour (it can be seen that this is a hot data), and the next minute there is no access to this data, but there are other data accesses, which will lead to the elimination of our hot data.

LFU

Minimum frequency of use recently.

In this algorithm, we optimize the above, use extra space to record the frequency of each data, and then select the lowest frequency to eliminate. This avoids the problem that the LRU cannot handle the time period.

Three elimination strategies are listed above. For these three kinds of elimination strategies, the implementation cost is higher than the other, and the same hit rate is better than the other.

Generally speaking, the solution we choose is in the middle, that is, if the implementation cost is not too high and the hit rate is OK, how to implement an lrumap?

We can implement a simple lrumap by inheriting LinkedHashMap and overriding the removeeldestentry method.

class LRUMap extends LinkedHashMap {
    private final int max;
    private Object lock;
    public LRUMap(int max, Object lock) {
        //No need for expansion
        super((int) (max * 1.4f), 0.75f, true);
        this.max = max;
        this.lock = lock;
    }

    /**
     *Override the removeldestentry method of LinkedHashMap
     *Judge when put, if it is true, the oldest will be deleted
     * @param eldest
     * @return
     */
    @Override
    protected boolean removeEldestEntry(Map.Entry eldest) {
        return size() > max;
    }
    public Object getValue(Object key) {
        synchronized (lock) {
            return get(key);
        }
    }
    public void putValue(Object key, Object value) {
        synchronized (lock) {
            put(key, value);
        }
    }
   
    public boolean removeValue(Object key) {
        synchronized (lock) {
            return remove(key) != null;
        }
    }
    public boolean removeAll(){
        clear();
        return true;
    }
}

A linked list of entry (the object used to put key and value) is maintained in LinkedHashMap. Every time we get or put, we will put the inserted new entry or the old query entry at the end of our linked list.

It can be noted that in the construction method, we set the size to Max * 1.4,

In the following removeeldestentry method, we only need size > Max to eliminate it. In this way, our map will never reach the logic of capacity expansion,

By rewriting LinkedHashMap, we implemented our lrumap in a few simple ways.

Modern society – guava cache

In modern society, lrumap has been invented to eliminate cache data, but there are several problems

  • Lock competition is serious. You can see that in my code, lock is a global lock. When the call amount is large, the performance will be lower.
  • Expiration time is not supported
  • Auto refresh is not supported

So the big guys of Google can’t help but invent guava cache. In guava cache, you can use it as easily as the following code:

public static void main(String[] args) throws ExecutionException {
    LoadingCache<String, String> cache = CacheBuilder.newBuilder()
            .maximumSize(100)
            //30 ms after writing
            .expireAfterWrite(30L, TimeUnit.MILLISECONDS)
            //30 ms after visit
            .expireAfterAccess(30L, TimeUnit.MILLISECONDS)
            //Refresh after 20ms
            .refreshAfterWrite(20L, TimeUnit.MILLISECONDS)
            //Turn on weakkey key. When garbage collection is started, the cache is also recycled
            .weakKeys()
            .build(createCacheLoader());
    System.out.println(cache.get("hello"));
    cache.put ("hello1", "I am hello1");
    System.out.println(cache.get("hello1"));
    cache.put ("hello1", "I'm hello2");
    System.out.println(cache.get("hello1"));
}

public static com.google.common.cache.CacheLoader<String, String> createCacheLoader() {
    return new com.google.common.cache.CacheLoader<String, String>() {
        @Override
        public String load(String key) throws Exception {
            return key;
        }
    };
}

Of course, there is no limit to the pursuit of performance.

also:

Caffeine: https://houbb.github.io/2018/09/09/cache-caffeine

LevelDB: https://houbb.github.io/2018/09/06/cache-leveldb-01-start

These more superior performance implementation, we can do a bit of in-depth study.

In this article, let’s take a look at how to implement a fixed size cache.

code implementation

Interface definition

In order to be compatible with map, we define a cache interface that inherits from the map interface.

/**
 *Cache interface
 * @author binbin.hou
 * @since 0.0.1
 */
public interface ICache<K, V> extends Map<K, V> {
}

Core implementation

Let’s look at the implementation of put

@Override
public V put(K key, V value) {
    //1.1 attempt to expel
    CacheEvictContext<K,V> context = new CacheEvictContext<>();
    context.key(key).size(sizeLimit).cache(this);
    cacheEvict.evict(context);
    //2. Judge the information after expulsion
    if(isSizeLimit()) {
        Throw new cacheruntimeexception ("the current queue is full, data addition failed! "";
    }
    //3. Execute add
    return map.put(key, value);
}

Here we can let the user specify the size dynamically, but to specify the size, there must be a corresponding elimination strategy.

Otherwise, a fixed size map will definitely not fit into the element.

Elimination strategy

There are many kinds of elimination strategies, such as LRU / LFU / FIFO, etc. we implement a basic FIFO here.

All the implementations are implemented in the way of interface, which is convenient for flexible replacement later.

public class CacheEvictFIFO<K,V> implements ICacheEvict<K,V> {

    /**
     *Queue information
     * @since 0.0.2
     */
    private Queue<K> queue = new LinkedList<>();

    @Override
    public void evict(ICacheEvictContext<K, V> context) {
        final ICache<K,V> cache = context.cache();
        //Remove if limit exceeded
        if(cache.size() >= context.size()) {
            K evictKey = queue.remove();
            //Remove the initial element
            cache.remove(evictKey);
        }

        //Put the new element at the end of the team
        final K key = context.key();
        queue.add(key);
    }

}

FIFO is relatively simple. We use a queue to store the elements put each time. When the queue exceeds the maximum limit, the oldest element is deleted.

Guide class

For the convenience of users, we implement a bootstrap class similar to guava.

All parameters are provided with default values, and fluent is used to improve the user experience.

/**
 *Cache boot class
 * @author binbin.hou
 * @since 0.0.2
 */
public final class CacheBs<K,V> {

    private CacheBs(){}

    /**
     *Create an object instance
     * @param <K> key
     * @param <V> value
     * @return this
     * @since 0.0.2
     */
    public static <K,V> CacheBs<K,V> newInstance() {
        return new CacheBs<>();
    }

    /**
     *Map implementation
     * @since 0.0.2
     */
    private Map<K,V> map = new HashMap<>();

    /**
     *Size limit
     * @since 0.0.2
     */
    private int size = Integer.MAX_VALUE;

    /**
     *Expulsion strategy
     * @since 0.0.2
     */
    private ICacheEvict<K,V> evict = CacheEvicts.fifo();

    /**
     *Map implementation
     * @param map map
     * @return this
     * @since 0.0.2
     */
    public CacheBs<K, V> map(Map<K, V> map) {
        ArgUtil.notNull(map, "map");

        this.map = map;
        return this;
    }

    /**
     *Set size information
     * @param size size
     * @return this
     * @since 0.0.2
     */
    public CacheBs<K, V> size(int size) {
        ArgUtil.notNegative(size, "size");

        this.size = size;
        return this;
    }

    /**
     *Set eviction policy
     *@ param evict elimination strategy
     * @return this
     * @since 0.0.2
     */
    public CacheBs<K, V> evict(ICacheEvict<K, V> evict) {
        this.evict = evict;
        return this;
    }

    /**
     *Build cache information
     *@ return cache information
     * @since 0.0.2
     */
    public ICache<K,V> build() {
        CacheContext<K,V> context = new CacheContext<>();
        context.cacheEvict(evict);
        context.map(map);
        context.size(size);

        return new Cache<>(context);
    }

}

Test use

ICache<String, String> cache = CacheBs.<String,String>newInstance()
        .size(2)
        .build();
cache.put("1", "1");
cache.put("2", "2");
cache.put("3", "3");
cache.put("4", "4");
Assert.assertEquals(2, cache.size());
System.out.println(cache.keySet());

The default is the first in first out policy. Keys are output at this time, and the contents are as follows:

[3, 4]

Summary

At this point, a simple version of the cache can be specified size of the implementation.

Complete code for the time being, this project has not been open source, you can pay attention to [old horse roaring west wind], background reply cache, get the source code.

So far, this cache implementation is relatively simple, obviously difficult to meet our usual more flexible application scenarios.

In the next section, we’ll learn how to implement a cache that can specify expiration.

How to achieve fixed size of handwriting cache from scratch