Jingtao project day15


Cgb2010 Jingtao project day15

Jingtao project day15

1. Redis cluster

See notes for details

1.1 redis cluster outage conditions

Outage condition: when the host in redis is missing, and there is no slave replacement, redis memory data is lost. At this time, redis cluster crashes

Problem 1: 6 redis 3 masters and 3 slaves (1 master and 1 slave are divided into 3 groups). At least redis is down, several clusters crash, and at least 2 clusters crash
Problem 2: 9 redis 3 masters and 6 slaves (1 master and 2 slaves are divided into 3 groups). At least several redis clusters are down and at least 5 clusters are down

The condition of cluster downtime: when the number of hosts cannot be guaranteed, the cluster crashes
Features: if the host in the cluster is down, the slave can continue to provide services,
If there is no slave in the host, it will borrow the redundant slave from other hosts and continue to provide services. If there is no slave available when the host goes down, the cluster will crash
Answer: there are 9 redis nodes. The cluster will crash only when the nodes are down 5-7 times
Jingtao project day15

1.2 redis partition algorithm

1.2.1 introduction case of cluster testing

Jingtao project day15

1.2.2 hash slot algorithm

Hash slot algorithm partition algorithm
Description: rediscluster uses this partition. All keys are mapped to 0-16383 slots according to the hash function (CRC16 [key]% 16384), with a total of 16384 slots. Each node maintains part of the slots and the key value data mapped by the slots. According to the number of primary nodes, the interval is evenly divided
Algorithm: hash function: hash () = CRC16 [key]% 16384
Jingtao project day15
When inserting data into the redis cluster, first calculate the key, then match the calculation result to the interval of a specific slot, and then set the data to the node managing the slot
Jingtao project day15

1.3 about algorithmic interview questions

1.3.1 description of clustering / fragmentation algorithm

Question: one data is very big. How to save one slot??? Wrong?? A. logical error B. reasonable
1. Consistent hash algorithm hash (key) 4.3 billion in clockwise direction to find the nearest node for set operation
2. Which node manages the results calculated by the hash slot algorithm CRC16 (key)% 16384 (0-16383), the data will be saved to the node
Core knowledge: consistent hash algorithm / hash slot algorithm are used to determineWho should manage the dataThe final data will be stored in the node

1.3.2 interview question 1

Q: can a total of 16384 data be stored in the redis cluster? A right B wrong why???
Xiaoming conjectures: there are 16384 slots in redis, so each slot stores a key??
Answer: wrong
Reason: there are 16384 slots in the redis cluster, but these slots are used toWho manages the division data. is not used to store data. And according to the hash calculation rules, collision problems can occur. For example

`It shows that key1 and key2 are managed by the same node
        Since slot is only used to distinguish data, how many data can be stored is decided by redis memory`

1.3.3 interview question 2

Question: how many hosts are there at most in the redis cluster?? 16384 setshost

1.4 relevant interview questions

1.4.1 cache penetration

Note: in a high concurrency environment,Frequent access to data that does not exist in the database. causes the user to request direct access to the database. In serious cases, the database server will be down
Jingtao project day15

  1. In the API gateway of IP current limiting operation, the upper limit of user access is set, which stipulates that each IP can only send n requests per unit time

Treat the symptoms but not the root cause (IP proxy server: change one IP in one minute)

`2. Bloom filter bloom filter

Bloom filter was proposed by bloom in 1970. It is actually a very long binary vector and a series of random mapping functions. Bloom filters can be used to retrieve whether an element is in a collection. Its advantage is that the space efficiency and query time are much better than the general algorithm, but its disadvantage is that it has a certain error recognition rate and difficult to delete.

Core knowledge points:
Usage: binary vector, hash function combination
effect:Determine whether an element exists in the collection.
Advantages: smaller footprint / higher efficiency
Disadvantages: there is a certain rate of misclassification (hash collision), difficult to delete advantages

Question: if there are 10 million hotspot data to be saved in the redis cache. Question: is it feasible?
Calculation: if you need to store 10 million data, you need about 20g of space
Knowledge foreshadowing:
1 byte = 8 bit (binary).
1KB (kilobyte) = 1024b,
1MB (megabyte) = 1024KB,
1GB (Gigabyte) = 1024MB,
1TB (trillionbyte) = 1024gb,

Transformation: suppose you can judge whether the data exists by 0 / 1. At the same time, it takes up less space. Then this problem is solved
Calculation 2: suppose that one data occupies one bit, how much space does it occupy? 1.19M application scenarios of Bloom filter

Note: when users query the server, they first query the bloom filter. If the data exists in the query, the subsequent process is executed,
If the query does not have the data, it will return directly. No subsequent process is required
Jingtao project day15 introduction of Bloom filter algorithm

Jingtao project day15 description of Bloom filter optimization

1. According to the hash principle, the probability of hash collision exists in the data. Then the bloom filter container is used to cause misjudgment. How to solve this problem?
Jingtao project day15 optimize hash collision probability – add binary vector optimize hash collision probability – increase the number of hash functions

Jingtao project day15 about the use process of bloom in the project

Jingtao project day15

1.4.2 buffer breakdown

Note: in high concurrency environmentA hotspot data is invalid in the cache due to deletion / timeoutAt this time, there are a large number of requests to directly access the database
Cause database down
Jingtao project day15
How to optimize:
1. The timeout of updating hotspot data regularly
2. Add multi-level cache mechanism
Jingtao project day15

1.4.3 cache avalanche

Concept: due toA large number of memory data failure in redisA large number of requests directly access the database, resulting in database downtime

Command: flushdb / flushall should be used with caution
1. Set different timeout / dynamic update timeout
2. Set multi-level cache
Jingtao project day15

1.5 integration of redis cluster with springboot

1.5.1 Edit Pro configuration file

`#Prepare redis node information

#Prepare three redis

#Prepare 6 redis nodes

1.5.2 edit configuration class

Jingtao project day15

1.5.3 edit cacheaop

Jingtao project day15

2. Front end implementation of Jingtao project

2.1 front end architecture design of Jingtao

Jingtao project day15

2.2 jt-web project creation

2.2.1 building jt-web

Jingtao project day15

2.2.2 add Inheritance / dependency / plug in

`<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <!-- Because the dynamic web resource is added, we use war -- > for packaging

    <!-- Add dependency -- >

    <!-- Add plug-in -- >
    <!-- Add a plug-in when there is a main method


2.2.3 import configuration file

Jingtao project day15

2.2.4 edit startup configuration item

Jingtao project day15

2.2.5 page effect display

Jingtao project day15

2.2.6 notes on main startup

Note: the configuration in this boot entry indicates that the configuration related to data source will not be loaded when springboot starts
Reason: there are out of the box rules in the springboot container. However, the front-end project does not need to use a data source. Therefore, the project can be started normally through the following configuration
Jingtao project day15

2.3 add nginx configuration

2.3.1 configuration requirements

User throughhttp://www.jt.comHow to accesshttp://localhost: 8092 server

2.3.2 edit hosts file

Jingtao project day15

2.3.3 edit nginx configuration file

`#Configure Jingtao foreground server
    server {
        listen 80;
        server_name www.jt.com;

        location / {
            proxy_pass http://localhost:8092;

2.3.4 effect display

Jingtao project day15

2.3.5 error report description of Google browser’s HTTP protocol

1) About the relationship between HTTP and HTTPS
Jingtao project day15
2) . solution strategy
1. In the browser, type: chrome://net-internals/#hsts :
Jingtao project day15

3 Preview

1. Preview what is cross domain
2. What is homology strategy
3. What is jsonp