|String (string)||Binary security||It can contain any data, such as JPG pictures or serialized objects. One key can store up to 512M||It can be used to do the simplest data. It can cache a simple string or a JSON format string. The implementation of redis distributed lock uses this data structure, including the implementation of counters and distributed IDS|
|Hash (hash table / Dictionary)||Set of key value pairs, i.e. map type in programming language||It is suitable for storing objects, and only one attribute value can be modified like an update attribute in the database (in memcached, you need to take out the entire string and deserialize it into an object, modify it, serialize it, and save it back)||Store, read and modify user attributes|
|List||Linked list (bidirectional linked list)||Fast addition and deletion. It provides an API for operating certain elements||1. The latest news ranking and other functions (such as the timeline of the circle of friends)
2. Message queue
|Set||Hash table implementation, no duplicate elements||1. The complexity of adding, deleting and searching is O (1)
2. It provides operations such as intersection, union and difference for sets
|1. Common friends, friends circle praise
2. Use uniqueness to count all independent IP addresses accessing websites
3. When friends recommend, they can find the intersection according to the tag. If it is greater than a certain threshold, they can recommend
|Sorted set||Add a weight parameter score to the elements in the set, and the elements are orderly arranged by score||When the data is inserted into the collection, natural sorting has been performed||1. Leaderboard
2. Message queue with weight
What is a distributed ID? What are the solutions?
During development, we usually need a unique ID to identify the data. In the case of a single architecture, we can use the primary key of the database or directly maintain a self increasing number in memory as the ID. however, for a distributed system, ID conflicts may occur. Here are the following solutions:
- UUID, which has the lowest complexity, but will affect storage space and performance
- Using the self incrementing primary key of the stand-alone database as the generator of distributed ID, the complexity is moderate, and the ID length is shorter than UUID. However, due to the performance limitations of the stand-alone database, this scheme is not the optimal scheme when the concurrency is large
- Use the features of redis and zookeeper to generate IDS, such as
Auto increment command of redisThe sequential node of zoomeeper. Compared with the stand-alone database (MySQL), the performance of this scheme is improved. It can be selected appropriately
- The snowflake algorithm is the most appropriate if all problems can be solved directly by the algorithm. The snowflake algorithm can also be used to generate distributed IDS. The underlying principle is that a machine automatically increases a certain number in a certain millisecond. This scheme can also ensure that the system ID in the distributed architecture is unique, but it can only ensure that the trend is increasing. In the industry, tinyid, leaf and other open-source middleware have implemented the snowflake algorithm.
Redis incr commandIncrement the numeric value stored in the key by one.
If the key does not exist, the value of the key will be initialized to 0 before the incr operation.
At present, there are two mainstream implementation schemes for distributed locks:
- Zookeeper: it is implemented by using the temporary node, sequential node and watch mechanism of zookeeper. Zookeeper distributed locks are characterized by high consistency. Because zookeeper guarantees CP, the distributed locks implemented by it are more reliable and free from confusion
redis: using redis
Consumer subscription mechanismThe redis distributed lock is characterized by high availability. Because redis guarantees the AP, the distributed lock implemented by it may be unreliable and unstable (once the data in redis is inconsistent), and multiple clients may be added to the lock at the same time
Setnx (set if not exists) commandWhen the specified key does not exist, set the specified value for the key. Return value
Set successfully, return 1. Setting failed, return 0.
Lua ScriptPut it in a Lua file and execute the Lua script. Example: active user judgment: judge whether a game user is an active user. If the standard is met, the number of active users +1
if redis.call("EXISTS",KEYS) == 1 then return 1 else return 0 end
Publish subscribe mode:
- Start the local redis service and two redis cli clients.
- Input at the subscriber / consumer redis cli client
SUBSCRIBE runoobChat, which means subscribing to the runoobchat channel.
- Input at publisher / producer redis cli client
PUBLISH runoobChat "Redis PUBLISH test"Send a message to the runoobchat channel. At this time, the first redis cli client will see the test message sent by the second redis cli client.
- First, setnx is used to ensure that the lock can be obtained only if the key does not exist. If the key exists, the lock cannot be obtained
- Then, the Lua script should be used to ensure the atomicity of multiple redis operations
- At the same time, considering that the lock has expired, an additional watchdog timer task is required to monitor whether the lock needs to be renewed
- At the same time, considering the situation after the redis node hangs, it is necessary to apply for a lock from n/2+1 nodes at the same time by using the red lock method. It is proved that the lock is obtained successfully only after the application is received. In this way, even if one of the redis nodes hangs, the lock cannot be obtained by other clients
By zhengkai Blog csdn. net
- It usually occurs when the slave node is initialized
- Send the sync command from the slave node to connect to the master node
- Master node received
Sync commandStart execution after
RDBFile and use the buffer replication buffer to record the write commands received during this process
- After the bgsave command of the master node is executed, the RDB file is sent to all slave nodes, and the write command recorded in the buffer replication buffer is sent
- After receiving the RDB file from the node, discard all old data and load the data in the received RDB file
- After the master node sends the RDB file, it starts to send the write command in the buffer replication buffer to the slave node
- After loading the RDB from the node, start to receive the client command and execute the write command from the replication buffer of the primary node
Every time the redis master node executes a write command, it will asynchronously send the same write command to the slave node, and the slave node will receive and execute the received write command
Most of the hot data stored in the cache is to prevent requests from directly obtaining data from the cache without accessing mysql.
Summary: hot data expires at the same time, and the expiration time is set to a random value or high availability design; Cache breakdown is caused by hot spot aging, which leads to direct connection to the database. Set the key not to expire. Cache penetration is caused by non-existent value attacks, and bloom filtering is used.
- Cache avalanche: if a large number of hot data in the cache expire at the same time, a large number of requests may directly access mysql. The solution is to add a random value to the expiration time. In addition, building a highly available redis cluster is also an effective means to prevent cache avalanche
- Cache breakdown: similar to cache avalanche,
Cache avalanche is the failure of a large number of hot data, and
Cache breakdown refers to the sudden failure of a hot key, which also leads to a large number of requests to directly access the MySQL databaseThis is cache breakdown. The solution is to consider that the hot key does not set an expiration time
- Cache penetration: if a large number of keys accessing redis do not exist in redis at a certain time (for example, hackers deliberately forge some messy keys), it will also cause pressure on the data. This is cache penetration. The solution is to use bloom filter. Its role is that if it thinks that a key does not exist, then the key certainly does not exist, So you can add a layer of Bloom filter before caching to intercept nonexistent keys
- Update MySQL first and then redis. If the redis update fails, it may still be inconsistent
- First delete the redis cached data, then update mysql, and then add the data to the cache when querying again. This solution can solve the problem of solution 1. However, the performance is low at high speed, and there are still data inconsistencies. For example, thread 1 deleted the redis cached data and is updating MySQL. At this time, another query will be performed again, and the old data in MySQL will be found in redis
Delay double deletion, the steps are:
First delete the redis cache data, then update mysql, and then delete the redis cache data after a delay of hundreds of millisecondsIn this way, even if other threads read MySQL and read the old data to redis when updating mysql, they will also be deleted to keep the data consistent
Redis stream is a newly added data structure in redis 5.0.
Redis stream is mainly used for message queues (MQ, message queues). Redis itself has a redis publish / subscribe (pub/sub) to implement the function of message queues. However, it has a drawback that messages cannot be persisted. If the network is disconnected or redis is down, messages will be discarded.
Simply put, publish / subscribe (pub/sub) can distribute messages, but cannot record historical messages.
Redis stream provides message persistence and active / standby replication functions, enabling any client to access data at any time, remembering the access location of each client, and ensuring that messages are not lost.
The structure of redis stream is as follows. It has a message chain list that links all the added messages. Each message has a unique ID and corresponding content: