Jingtao project day13-14

Time:2021-5-12
  1. Implementation of cache service with AOP

=============

1.1 business requirements

1) . custom annotation @ cachefind (key = “XXX”, second = – 1)
2) Use custom annotation to identify the business method and save the return value of the method to the cache
3) AOP is used to intercept annotations, and surround notification is used to realize business

1.2 user defined annotation @ cachefind

Jingtao project day13-14

1.3 annotation identification

Jingtao project day13-14

1.4 edit AOP

package com.jt.aop;

import com.jt.anno.CacheFind;
import com.jt.util.ObjectMapperUtil;
import org.aspectj.lang.JoinPoint;
import org.aspectj.lang.ProceedingJoinPoint;
import org.aspectj.lang.annotation.Around;
import org.aspectj.lang.annotation.Aspect;
import org.aspectj.lang.annotation.Before;
import org.aspectj.lang.annotation.Pointcut;
import org.aspectj.lang.reflect.MethodSignature;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Component;
import org.springframework.stereotype.Controller;
import org.springframework.stereotype.Repository;
import org.springframework.stereotype.Service;
import redis.clients.jedis.Jedis;

import java.lang.reflect.Method;
import java.util.Arrays;

/*@Service
@Controller
@Repository*/
@Component // the component hands over the class to the spring container for management
@Aspect // indicates that I am a section
public class RedisAOP {

    @Autowired
    private Jedis jedis;

    /*
    *Implementation of AOP service call
    *1. Intercept the specified annotation
    *2. Using surround notification
    *Implementation steps:
    *1. To get the key, you must get the annotation first, and get the key from the annotation?
    *2. Check whether there is a value in redis
    *     *
    *3. Knowledge point supplement:
    *Specify the parameter name to transfer the value, and bind the parameter type at runtime to intercept the annotation
    *Joinpoint must be in the first bit of the parameter
    */
    @Around("@annotation(cacheFind)")
    public Object around(ProceedingJoinPoint joinPoint,CacheFind cacheFind){
        Object result = null;
        //Key = Business Name:: parameter
        String key = cacheFind.key();
        String args = Arrays.toString(joinPoint.getArgs());
        key = key + "::" + args;

        //2. Check whether there is a value
        if(jedis.exists(key)){
            String json = jedis.get(key);
            MethodSignature methodSignature = (MethodSignature) joinPoint.getSignature();
            Class returnType = methodSignature.getReturnType();

            result = ObjectMapperUtil.toObj(json,returnType);
            System. Out. Println ("AOP query redis cache");

        }else{
            //There is no data in redis, so you need to query the database and save the data in the cache
            try {
                result = joinPoint.proceed();
                String json = ObjectMapperUtil.toJSON(result);
                //Do you want to set the timeout
                if(cacheFind.seconds()>0){
                    jedis.setex(key, cacheFind.seconds(), json);
                }else{
                    jedis.set(key,json);
                }
                System. Out. Println ("AOP query database");
            } catch (Throwable throwable) {
                throwable.printStackTrace();
            }
        }

        return result;
    }


    /**
     *// 1. Get the key annotation method object class method name parameter
     *         Class targetClass = joinPoint.getTarget().getClass();
     *// 2. Get method object
     *         String methodName = joinPoint.getSignature().getName();
     *         Object[] args = joinPoint.getArgs();
     *         Class[] classArgs = new Class[args.length];
     *         for(int i=0;i<args.length;i++){
     *             classArgs[i] = args[i].getClass();
     *         }
     *         try {
     *// reflect instantiated objects
     *             Method method = targetClass.getMethod(methodName,classArgs);
     *             CacheFind cacheFind = method.getAnnotation(CacheFind.class);
     *             String key = cacheFind.key();
     *             System.out.println(key);
     *         } catch (NoSuchMethodException e) {
     *             e.printStackTrace();
     *         }
     */










    //Formula AOP = pointcut expression + notification method
    //@Pointcut("bean(itemCatServiceImpl)")
    //@Pointcut("within(com.jt.service.*)")
    //@Pointcut ("execution (* com. JT. Service. *. * (..)) //. * the first level subdirectory of the current package
   /*@ pointcut ("execution (* com. JT. Service.. * * (..)) //.. * all subdirectories of the current package
    public void pointCut(){

    }*/

    //How to get the related parameters of the target object?
    //ProceedingJoinPoint is only supported for around advice
   /* @Before("pointCut()")
    Public void before (joinpoint joinpoint) {// join point
        Object target = joinPoint.getTarget();
        Object[] args = joinPoint.getArgs();
        String className = joinPoint.getSignature().getDeclaringTypeName();
        String methodName = joinPoint.getSignature().getName();
        System. Out. Println ("target object: + target)";
        System. Out. Println ("method parameter: + arrays. ToString (args));
        System. Out. Println ("class name: + classname)";
        System. Out. Println ("method name: + methodname)";
    }*/
}`
  1. About redis general properties

===============

2.1 persistence strategy in redis – RDB

2.1.1 requirement description

Note: redis saves all data in memory, but the characteristics of memory are power-off and erasure. In order to ensure that the cache data in redis is not lost, it is necessary to persist the memory data regularly
Persistence: write memory data to disk

2.1.2 RDB mode

characteristic:
1. RDB mode is the default persistence rule of redis
2.RDB mode records redis memory data snapshot (only the latest data is kept)
3. Regular persistence (time adjustable) of RDB mode may lead to data loss
4. RDB mode has the highest backup efficiency
5. RDB mode backup is blocking, and other users are not allowed to operate during backup to ensure data security
Command:
1. Active backup save will block user operation
2. Background backup bgsave asynchronously for persistent operation will not block

2.1.3 about persistent configuration

1. Save 900 1 900 seconds, when the user performs an update operation, it will persist once
2. Save 300 10 300 seconds, the user has executed 10 update operations, then persistence once
3. Save 60 10000 60 seconds, the user executed 10000 times of update operations, then persistent once
4. Save 1 second, update and persist once!! The performance is very low
Jingtao project day13-14

2.1.4 about persistent file name setting

By default, the persistent file name is dump. RDB
Jingtao project day13-14

2.1.5 file storage directory

. / represents the current file directory. The meaning is written in absolute path
Jingtao project day13-14

2.2 persistence strategy in redis – AOF

2.2.1 characteristics of AOF

1) Aof mode is off by default. It needs to be turned on manually
2).Aof mode records the user’s operation process. It can realize real-time persistence and ensure that the data is not lost.
3) The persistent files maintained in. AOF mode occupy a large space, so the persistence efficiency is not high, and the persistent files need to be maintained regularly
4) Once the. AOF mode is turned on, redis reads AOF files mainly in AOF mode

2.2.2 AOF configuration

1) . turn on AOF mode
Jingtao project day13-14
2) . persistence strategy
Always: users update once, then persist once
Everysec: persistence once per second is more efficient
No: no active persistence. Operating system related. Almost no use
Jingtao project day13-14

2.3 about redis interview questions

2.3.1 about flushall operation

Business scenario:
Xiaoli is a very beautiful intern. You are his project director. Because Xiaoli is not familiar with the business, she inadvertently implemented flushall operation in the production environment. How to remedy it??
Scenario 1: the service in redis only enables the default persistent policy RDB mode
Solution:
1. Close the existing redis server
2. Check whether the RDB file is covered. If the file is not covered, restart redis
3. If flushall command and save operation are executed at the same time, RDB mode is invalid

`Scenario 2: the service in redis starts AOF mode
Solution: 
        1. Close the redis server
        2. Edit the redis persistence file and delete the flushall command
        3. Restart the redis server

General conditions: both RDB mode and AOF mode will be enabled. RDB persistence mode can be executed by Save command

2.3.2 why is single threaded redis fast

1) Redis running environment is in memory, pure memory operation
2) Single thread operation avoids frequent context switching and switch link overhead
3) Non blocking I / O (bio | NiO) is adoptedThe mechanism of multiplexing(dynamic perception)
Jingtao project day13-14
4) The latest version of redis is 6.0. Before 6.0, it is single thread operation mode. After 6.0, it supports multi thread operation mode

2.4 about redis memory optimization strategy

2.4.1 business scenario

If you use redis frequently, keep saving data to it, and do not delete, then the memory will overflow. Can you optimize the memory strategy
Can you delete unused data automatically and keep hot data in redis

2.4.2 LRU algorithm

LRU is the abbreviation of least recently usedLeast recently used, is a commonly used page replacement algorithm, select the latest and longest unused page (data) to be eliminated. The algorithm gives each page an access field,It is used to record the time t of a page since it was last visitedWhen a page needs to be eliminated, select the one with the largest t value in the existing page, that is, the least recently used page.
Calculation dimension: time t since last time
Jingtao project day13-14
Description: LRU algorithm is the best algorithm in memory optimization

2.4.3 LFU algorithm

LFU(least frequently used (LFU) page-replacement algorithm)。 NamelyLeast frequently used page replacement algorithmIt is required to replace the page with the smallest reference count during page replacement, because the frequently used page should have a larger number of references. But some pages are used many times at the beginning, but they will not be used any more. These pages will stay in memory for a long time, so the reference count register can be timedMove rightOne bit, the average number of uses that form exponential decay.
Dimension: number of references
Common sense: computer left expansion multiple
The computer moves right to reduce multiple
Jingtao project day13-14

2.4.4 random algorithm

Delete data randomly

2.4.5 TTL algorithm

Note: sort the remaining survival time, and delete the data that will be deleted in advance
Jingtao project day13-14

2.4.6 redis default memory optimization strategy

Note 1: the strategy used in redis is periodic deletion + lazy deletion
Note 2:
1. Delete regularly: redis is the defaultEvery 100 msCheck whether there are expired keys, and check them randomly. (not all the data, because the efficiency is too low.)
Problem: due to the large amount of data, it may not be selected when extracting. It may appear that the data has reached the timeout, but redis does not delete the data immediately

  1. Inertia strategy:When the user obtains the key, first check whether the data has passed the timeoutIf it has timed out, delete the data

Problem: due to the large amount of data, it is impossible for users to get all the memory data. It is inevitable that the data to be deleted will be kept in the memory all the time. It takes up the memory resources
3. You can use the above memory optimization method to delete the memory actively

Memory optimization algorithm description:
1. For volatile LRU, LRU algorithm is used to optimize the timeout data
2. All keys LRU algorithm is used to optimize all data
3. Volatile LFU uses LFU algorithm to optimize the timeout data
4. All keys LFU uses LFU algorithm to optimize all the data
5. Volatile random uses random algorithm when setting the timeout time
6. Allkeys random random algorithm is used for all data
7. Volatile TTL TTL algorithm for setting timeout
8. Noeviction does not delete data actively. If memory overflows, an error will be reported
Jingtao project day13-14

  1. Redis fragmentation mechanism

=============

3.1 business requirements

Note: the data capacity of a single redis is limited. If you need to store massive cache data, using a single redis can not meet the requirements. In order to meet the needs of data expansion, you can use the mechanism of fragmentation
Jingtao project day13-14

3.2 implementation of redis fragmentation mechanism

3.2.1 construction strategy

Prepare three redis 6379 / 6380 / 6381

3.2.2 prepare file directory

Jingtao project day13-14

3.2.2 copying configuration files

Note: put the redis configuration file into the shards directory
Jingtao project day13-14
Modify the port number of the configuration file, and then modify 6380 / 6381
Jingtao project day13-14
Start three redis:
redis-server 6379.conf
redis-server 6380.conf
redis-server 6381.conf

Verification server:
Jingtao project day13-14

3.2.3 introduction to redis segmentation

`package com.jt.test;

import org.junit.jupiter.api.Test;
import redis.clients.jedis.JedisShardInfo;
import redis.clients.jedis.ShardedJedis;

import java.util.ArrayList;
import java.util.List;

public class TestRedisShards {

    @Test
    public void testShards(){
        List<JedisShardInfo> shards = new ArrayList<>();
        shards.add(new JedisShardInfo("192.168.126.129",6379));
        shards.add(new JedisShardInfo("192.168.126.129",6380));
        shards.add(new JedisShardInfo("192.168.126.129",6381));
        ShardedJedis shardedJedis = new ShardedJedis(shards);
        //The memory capacity of three redis is expanded three times as one redis???
        Shardedjedis. Set ("shards", "redis split test");
        System.out.println(shardedJedis.get("shards"));
    }
}

3.3 consistent hash algorithm

3.3.1 algorithm Introduction

Consistent hash algorithm was proposed by MIT in 1997. It is a special hash algorithm to solve the problem of distributed cache[ 1] When a server is removed or added, the mapping relationship between the existing service request and the processing request server can be changed as little as possible. Consistent hashing solves the dynamic scaling problem of simple hashing algorithm in distributed hash table (DHT) [2].
Role: to solve the problem of cache data, where to store

3.3.2 algorithm description

Common sense:

  1. If the data is the same, the hash result must be the same
  2. Common hash values consist of 8-bit hexadecimal numbers. How many possibilities do they share? 2^32

Jingtao project day13-14

3.3.3 balance

① Balance means that the results of hash should be evenly distributed to each node, which solves the problem of load balancing algorithmically [4].
Description: realize data balance through virtual node
Jingtao project day13-14

3.3.3 monotonicity

② Monotonicity means that when adding or deleting nodes, the normal operation of the system will not be affected [4].
Principle: if the node is added / reduced, the original data should be kept unchanged as far as possible

3.3.4 dispersion

③ Decentralization means that the data should be stored in each node of the distributed cluster (the node can have its own backup), and it is not necessary for each node to store all the data [4]
If the data is stored separately, even if the server is down in the future, the impact will be only partial, not all
Don’t put eggs in one basket
Jingtao project day13-14

3.4 springboot integrates redis fragmentation mechanism

3.4.1 Edit Pro configuration file

Jingtao project day13-14

3.4.2 edit configuration class

Jingtao project day13-14

3.4.3 modifying configuration in AOP

Jingtao project day13-14

4. Redis sentinel mechanism

4.1 problems in redis fragmentation

Note: the redis fragmentation mechanism can realize the redis redis memory expansion, but the redis node is not availableNo high availabilityIf the node is down, the integrated redis partition will not be available

4.2 construction of redis master-slave structure

Regulations: 6379 master / 6380 / 6381 slave

4.2.1 copy file directory

Jingtao project day13-14

4.2.2 deleting persistent files

Jingtao project day13-14

4.2.3 start three redis servers

1.redis-server 6379.conf
2.redis-server 6380.conf
3.redis-server 6381.conf
Jingtao project day13-14

4.2.4 realize the master-slave mount of redis

Command 1: slave of host port
Command description: to execute the above command in the slave, the address of the host is mounted
Jingtao project day13-14

Command 2: Info replication
Jingtao project day13-14

The relationship of master-slave structure is as follows
Jingtao project day13-14

4.3 working principle of redis sentry

4.3.1 work flow chart

Jingtao project day13-14
Principle description:
1. The sentinel monitors the running status of the host. If the node does not respond three times in a row through the heartbeat detection mechanism (ping-pong), the sentinel concludes that the host is down and begins to vote
2. Sentinel obtains the configuration information of the host (including the master-slave structure) by linking the host, selects the slave that links the current host, selects the new host according to the random algorithm, and sets other nodes as the slaves of the new host

4.3.2 edit sentinel profile

1) . copy sentry’s configuration file
Jingtao project day13-14
2) . turn off protection mode
Jingtao project day13-14
3) . turn on the back-end operation
Jingtao project day13-14
4) . set the number of votes for sentinels
Jingtao project day13-14
5) . modify the election timeout
Jingtao project day13-14
6) . modify sentinel status
Jingtao project day13-14

4.3.3 sentry test

Sentry order:redis-sentinel sentinel.conf
Check redis service:
Jingtao project day13-14
Redis high availability test:
1. Close the redis host 6379
2. Wait 10 seconds to check who the host is
3. Restart the 6379 server to check whether it acts as the slave of the new host

4.3.4 entry cases of sentry

`@Test
    public void test01(){
        //Define the Sentinel's set information
        Set<String> sentinels = new HashSet<>();
        sentinels.add("192.168.126.129:26379");
        //Define link pool information
        JedisPoolConfig poolConfig = new JedisPoolConfig();
        poolConfig.setMaxTotal(200);    // Link pool up to 200 links
        poolConfig.setMaxIdle(20);      // Maximum free links 20
        poolConfig.setMinIdle(10);      // Minimum free links 10
        JedisSentinelPool pool = new JedisSentinelPool("mymaster",sentinels,poolConfig);
        //Dynamic access to jedis links
        Jedis jedis = pool.getResource();
        Jedis. Set ("ABC", "redis assignment operation");
        System.out.println(jedis.get("abc"));
        jedis.close();
    }

4.3.5 operation

The mechanism of Sentry will be managed by springboot

4.4 summary on fragmentation / sentry

1. Fragmentation mechanism: it can realize the expansion of memory data, but it does not achieve the effect of high availability
2. Sentry mechanism: Sentry can achieve high availability of redis nodes, but sentry itself does not achieve high availability
Requirements: 1. High availability without relying on a third party
2. Realize the expansion of memory data
3. Each node can be highly available

  1. Redis cluster building function includes the above three mechanisms. Generally, companies will deploy redis in the form of cluster

5. Redis cluster construction (key)

5.1 please refer to the work document for the specific construction steps

5.2 description of redis cluster construction

matters needing attention:
1. Start 6 redis nodes
2. Ensure that the data in the redis node is null
3. Check the IP address and firewall according to the error prompt
1) . shut down all redis servers
sh stop.sh

2) . check the redis configuration file
Jingtao project day13-14
3) . delete redundant files
Jingtao project day13-14

5.3 about redis cluster high availability test

Jingtao project day13-14
1) . shut down 7000 nodes
redis-cli -p 7000 shutdown

2) . check master-slave status
Jingtao project day13-14
3) . restart 7000 to check status
Jingtao project day13-14

Recommended Today

Looking for frustration 1.0

I believe you have a basic understanding of trust in yesterday’s article. Today we will give a complete introduction to trust. Why choose rust It’s a language that gives everyone the ability to build reliable and efficient software. You can’t write unsafe code here (unsafe block is not in the scope of discussion). Most of […]