Stage 4 notes Jingtao_ day12

Time:2021-1-28

Day 12 redis split

This document is arranged according to the class process. Please refer to Miss Liu’s column for more details and pictures

Jiang GE’s column

Cgb2008 Jingtao Day12

1、 Realization of commodity classification cache by AOP
  1. Business requirements

    Through the form of user-defined annotation, dynamic cache operation is realized. The target of interception is the method with the specified annotation. Get the key and timeout through annotation.

  2. Edit @ cachefind annotation

    @Target(ElementType.METHOD)
    @Retention(RetentionPolicy.RUNTIME)
    public @interface CacheFind {
        public String key();
        Public int seconds() default 0; // there is no need to set the timeout
    }
  3. Usage of custom annotations

    == ItemCatServiceImpl ==
    @CacheFind(key = "ITEM_ CAT_ Parent ", seconds = 7 * 24 * 60 * 60) // the specific key is filled automatically
    public List<EasyUITree> findItemCatCache(long pid) {}
  4. Edit cacheaop

    @Autowired
    private Jedis jedis;
    /*Realization idea:
        Find the type of class, method name and parameter list
        Key = user defined prefix + user parameters [0, XX]
        Judge whether the key exists. If it exists, get the data in it. JSON converts it into a specific object (return value type)
            On the contrary, the target method is executed to obtain the return value, which is converted into JSON format and saved in the cache
     */
    @Around("@annotation(cacheFind)")
    public Object around(ProceedingJoinPoint joinPoint, CacheFind cacheFind){
        System.out.println ("= = surround notification start =");
        Object result = null;
        //1. Get the prefix of key
        String key = cacheFind.key();
        System.out.println("key = "+key);
        //2. Get method parameters
        Object[] args = joinPoint.getArgs();
        String argString = Arrays.toString(args);
        key = key+"::"+argString;
        System.out.println("key = "+key);
        try {
            //3. Judge whether there is data in the cache
            if(jedis.exists(key)){
                String json = jedis.get(key);
                //5. Get the return value type
                Class clazz = ((MethodSignature)joinPoint.getSignature()).getReturnType();
                result = ObjectMapperUtil.toObj(json,clazz);
                System.out.println ("= = execution database =");
            }Else {// there is no data in the cache
                result = joinPoint.proceed();
                String resultJSON = ObjectMapperUtil.toJSON(result);
                //4. Judge whether there is a timeout in the data
                if(cacheFind.seconds()>0){
                    jedis.setex(key, cacheFind.seconds(), resultJSON);
                }else {
                    jedis.set(key, resultJSON);
                }
                System.out.println ("= = execution database =");
            }
        } catch (Throwable throwable) {
            throwable.printStackTrace();
            throw new RuntimeException(throwable);
        }
        System.out.println ("= = end of notification =");
        return result;
    }
  5. Annotate the way to query commodity categories

    @Override
    @CacheFind(key = "ITEM_CAT")
    public String findCatNameByCid(int cid) {
        return itemCatMapper.findCatNameByCid(cid);
    }
2、 Basic characteristics of redis
  1. Persistence strategy

    i. Why persistence?

    The records in redis are stored in memory. If the memory is powered off or the server is down, the memory data will be lost directly. In order to prevent this phenomenon, it is necessary to maintain the data regularly.

    II. RDB mode

    1) RDB mode is the default persistence policy of redis, which does not need to be opened manually;

    2) redis regularly performs RDB persistence operations. Disadvantages: may cause memory data loss;

    3) RDB records the snapshot of memory data. The subsequent snapshot will cover the previous snapshot and only keep the latest data each time, which is more efficient;

    4) if a small amount of data loss is allowed, RDB mode can be used.

    Command:

    #Perform persistence immediately
    Save = > will cause thread blocking
    #Persistence operation in background
    Bgsave = > will not cause blocking, it is not clear when it will be completed, and immediate execution cannot be guaranteed

    III. AOF mode

    1) it is closed by default and needs to be opened manually. After opening, RDB mode fails. However, if the Save command is executed manually, the RDB file will also be generated;

    2) it records the running process of the program, so the data is not lost;

    3) because it records the running process of the program, the persistent file will be large and need to be maintained regularly.

    Open AOF

    1) edit configuration file

    appendonly yes
    # The name of the append only file (default: "appendonly.aof")
    appendfilename "appendonly.aof" 

    2) restart the redis server

    redis-cli shutdown
    redis-server redis.conf

    IV. comparison of persistence strategies between RDB and AOF

    1) RDB mode

    Save 900 1 = > If an update operation is executed within 900 seconds, it is persisted once
    save 300 10
    The faster the save 60 1000 operation, the shorter the persistence period

    2) AOF mode

    If the appendfsync always user performs an update operation, an asynchronous operation is persisted
    Appendfsync everysec operates once per second
    Appendfsync no is not active operation, generally not used

    v. Summary

    RDB mode is preferred if a small amount of data is allowed to be lost; AOF mode is preferred if data is not allowed to be lost.

    Enterprise strategy: not only to meet the efficiency, but also to meet the data loss. Multiple redis servers are configured, and the host uses RDB mode to achieve fast reading and writing efficiency; the slave uses AOF mode to backup data.

    Vi. interview questions

    Xiao Li mistakenly executed the command flushall on the redis server. As a project manager, how would you solve this problem?

    You need to edit the AOF file from the library, delete the superfluous flushall command, and then restart.

  2. Memory optimization strategy

    i. LRU algorithm

    1) the least recently used permutation algorithm to eliminate the longest unused data;

    2) judgment dimension: time t

    II. LFU algorithm

    1) the least frequently used permutation algorithm

    2) judgment dimension: times of use

    III. random algorithm: random deletion

    IV. TTL algorithm: delete the data with short remaining time

    # volatile-lru 
    -> Evict using approximated LRU among the keys with an expire set.
    # allkeys-lru
    -> Evict any key using approximated LRU.
    # volatile-lfu 
    -> Evict using approximated LFU among the keys with an expire set.
    # allkeys-lfu 
    -> Evict any key using approximated LFU.
    # volatile-random 
    -> Remove a random key among the ones with an expire set.
    # allkeys-random 
    -> Remove a random key, any key.
    # volatile-ttl 
    -> Remove the key with the nearest expire time (minor TTL)
    # noeviction 
    -> Don't evict anything, just return an error on write operations.
  3. Common interview questions of redis

    Business scenario: when users operate on the server for a long time in a high concurrency environment, the following problems may occur:

    i. What is cache penetration?

    In a high concurrency environment, accessing data that does not exist in the database or cache. A large number of users directly access the database, resulting in abnormal database connection.

    Solution: disable IP, limit IP access / current, access 3 times per second at most, bloom filter (used to retrieve whether an element exists in a collection)

    Binary vector – > due to hash collision, multiple keys may have the same location. Therefore, if the bloom filter thinks that the data exists, then the data may exist; if the bloom filter thinks that the data does not exist, then the data must not exist.

    Solution: expand the length of two-dimensional vector, and use multiple hash algorithms for calculation

    1G memory, how to judge whether 20 billion is effective?

    II. What is cache breakdown?

    Some hot data suddenly fail in the cache, leading to a large number of users directly accessing the database, resulting in excessive concurrent pressure and database exceptions.

    Solution: set the timeout of hotspot data a little longer, and set multi-level cache (the timeout adopts random algorithm)

    III. what is cache avalanche?

    In the cache server, due to a large number of cache data failure, resulting in the user access hit rate is too low, direct access to the database. Flushall will lead to cache avalanche. To set the timeout, random algorithm should be used.

    Solution: set multi-level cache

3、 Redis fragmentation mechanism
  1. Redis performance optimization

    The memory capacity of a single redis is limited, but if there is a huge amount of data to be cached, multiple redis nodes should be used. This structure is called redis’s fragmentation mechanism.

  2. Partition mechanism configuration

    i. Configuration planning

    Prepare three redis servers with port numbers of 6379, 6380 and 6381

    #Close the redis server
    redis-cli shutdown
    #Create folder
    mkdir shards
    #Move three server profiles to a folder
    cp redis.conf shards/6379.conf
    cp redis.conf shards/6380.conf
    cp redis.conf shards/6381.conf
    #Modify port number
    cd /shards
    vim 6380.conf
    #Open service
    redis-server 6379.conf
    redis-server 6380.conf
    redis-server 6381.conf
    #Check server status
    ps -ef | grep redis

    II. Writing test methods

    /*Users can dynamically link three redis servers through client program*/
    @Test
    public void testShards(){
        List<JedisShardInfo> list = new ArrayList<>();
        list.add(new JedisShardInfo("192.168.126.129",6379));
        list.add(new JedisShardInfo("192.168.126.129",6380));
        list.add(new JedisShardInfo("192.168.126.129",6381));
        ShardedJedis shardedJedis = new ShardedJedis(list);
        shardedJedis.set ("shards", "redis splitting operation");
        System.out.println(shardedJedis.get("shards"));
    }
  3. Springboot integrates redis fragmentation

    i. Modification redis.porperties configuration file

    #Configure redis fragmentation mechanism
    redis.nodes=192.168.126.129:6379,192.168.126.129:6380,192.168.126.129:6381

    II. Modify redisconfig configuration class

    @Value("${redis.nodes}")
    private String nodes; //node,node,node
    
    @Bean
    public ShardedJedis shardedJedis(){
        String[] nodeArr = nodes.split(",");
        List<JedisShardInfo> list = new ArrayList<>();
        for (String node:nodeArr){ //node = host:port
            String host = node.split(":")[0];
            int port = Integer.parseInt(node.split(":")[1]);
            list.add(new JedisShardInfo(host,port));
        }
        ShardedJedis shardedJedis = new ShardedJedis(list);
        return shardedJedis;
    }

    III. modify cacheaop

    @Autowired
    private ShardedJedis jedis;

Recommended Today

asp.net Application of regular expression

1. Balanced group / recursive matching (?’ Group ‘), which is called the corresponding content of group, and counts it on the stack;(?’- Group ‘), and count the corresponding content named group out of the stack(?!) Zero width negative look ahead assertion. Since there is no suffix expression, attempts to match always failRegular example:,{0,1}”5″:\[[^\[\]]*(((?’Open’\[)[^\[\]]*)+((?’-Open’\])[^\[\]]*)+)*(?(Open)(?!))\],{0,1} Test […]