Redis’s first acquaintance

Time:2020-6-28

background

Recently, I met many interview questions about redis. Today, I’d like to summarize the use of redis. Here’s more about the use of redis API. For those who are interested, see the official website: redis official website

Key expiration policy

When the key expires in redis, will it be deleted immediately?

Let’s learn about deletion strategies, which can be divided into three types

  • Scheduled deletion(memory friendly, CPU unfriendly)
  • Lazy delete(extremely CPU friendly, memory friendly)
  • Delete periodically(compromise)

Lazy delete means that every time a key is fetched from the key space, judge whether the key has expired, and delete it if it has expired.

Redis adopts two strategies: lazy deletion and regular deletion, so,In redis, if the key is expired, it may not be deleted immediately!

Memory obsolescence mechanism

If a lot of expired keys are missed in regular deletion, and they are not checked in time (no lazy deletion), a lot of expired keys are piled up in memory, resulting in the memory block of redis running out, how to make it?

We can set the maximum memory usage. When the memory usage exceeds, it will be implementedData obsolescence strategy

Redis has the following memory elimination mechanisms:

Volatile LRU: select the least recently used data obsolescence from the data set with expiration set
Volatile TTL: select the data that will expire from the data set with expiration time set.
Volatile random: choose any data elimination from the data set with expiration time set
Allkeys LRU: select the least recently used data from the data set
Allkeys random: choose any data from the data set to eliminate
No invitation: when the memory reaches the limit, no data will be eliminated

Slow query

brief introduction

Like many relational databases, redis also provides slow query log records. Redis will put slow commands into an internal list.

Note: slow query records only the execution time of the command, excluding network transmission and queuing time.

Slow query configuration

There are two configurations for redis slow query, namelyslowlog-log-slower-thanandslowlog-max-len
slowlog-log-slower-thanIt is used to control the threshold value of slow query, and all commands with execution time exceeding this value will be recorded. The unit of the value isMicrosecond, the default value is 10000. If it is set to 0, all records will be recorded. If it is set to a value less than 0, no command will be recorded, that is, slow query is turned off. You can set it in the configuration file or with the config set command:

config set slowlog-log-slower-than 10000

slowlog-max-lenIt is used to set the size of the slow query record list. The default value is 128. When the list is full, if there are new records coming in, redis will clean up the oldest records in the team, and then store new records. In the production environment, we can increase the size appropriately, for example, to 1000, so that more records can be buffered and troubleshooting is convenient. The configuration method is similar to slowlog log slow than. It can be specified in the configuration file or set by executing config set on the command line

config set slowlog-max-len 1000

View slow query log

Query log

Redis specifically provides a set of commands to query the slow query log:

SLOWLOG GET

Redis's first acquaintance

You can see that two slow query records are found here, namely, zincrby and zrevrange commands
What do 1) 2) 3) 4) of the records represent respectively?
1) Represents log unique identifier UID
2) System timestamp at command execution time
3) The length of command execution, measured in terms of delicacy
4) Commands and command parameters

Get the current number of slow query log records

SLOWLOG LEN

Slow query log reset

SLOWLOG RESET

Pipeline

pipelineCommands are used to batch operate redis commands

Because redis isSingle threadThe next request must wait for the last request to complete before execution can continue. implementN times of command requires n times of network time + execution timeHowever, with the pipeline mode, the client can send multiple commands at one time, that is1 network time + N execution timeIn this way, the network round-trip time is greatly reduced and the system performance is improved.

Here’s the pseudo code:

Jedis redis = new Jedis("127.0.0.1", 6379);
Pipeline pipe = redis.pipelined();
       for (int i = 0; i < 10000; i++) {
            pipe.hmset ("key_ "+ I, data); // encapsulate the value to the pipe object, which is not executed at this time, and still stays on the client
        }
        pipe.sync (); // send the encapsulated pipe to redis in one time
        jedis.close;

So the question is, under what circumstances is it suitable to use pipeline?

Some systems may have high reliability requirements. Each operation needs to know immediately whether the operation is successful or not and whether the data has been written into redis. This scenario is not suitable.
Other systems may write data to redis in batches, allowing a certain proportion of write failures. Then this scenario can be used. For example, if 10000 pieces of data enter redis, two pieces of data may fail. It doesn’t matter. In the later stage, there is a compensation mechanism. For example, in the scenario of group sending of SMS, if 10000 pieces of data are sent in groups and implemented according to the first mode, the request comes , it will take a long time to respond to the client, and the delay is too long. If the client requests to set a timeout of 5 seconds, an exception will be thrown. Moreover, the real-time requirement for mass SMS is not as high, so it is best to use pipeline.

BitMap

BitmapIs a string of consecutive binary digits (0 or 1), each of which is in the position ofOffset (offset), and, or, XOR and other bit operations can be performed on bitmap.

API

SETBIT

SETBIT KEY OFFSET VALUE

This command is used to set or clear the bit on the specified offset for the string value stored by the key. Time complexity O (1)

In redis, the stored strings are allBinaryThe form exists.
How can I change ‘a’ to ‘B’ with the setbit command? To change 0110001 to 01100010 (the ASCII code of B is 98), is to change the offset 6 in ‘a’ from 0 to 1, and the offset 7 from 1 to 0.
After each setbit, the original bit value of the location will be returned.

BITCOUNT

BITCOUNT KEY [START END]

This command counts the number of bits set to 1 in the specified range in the string

GETBIT

GETBIT KEY OFFSET

Returns the bit value of string corresponding to key at offset

BITOP

BITOP operation destkey key [key…]

You can intersect, union and other operations on multiple binaries, and save the results to destkey

actual combat

A simple example:Daily active users

In order to count the number of users who log in today, we set up a bitmap, each of which identifies a user ID. When a user visits our web page or performs an operation, the location identifying the user is set to 1 in bitmap. So we can get throughBITCOUNTCommand to get the number of active users of the day.

Note: bitmap can be used to count traffic for a hundred million level traffic system. If the system user is only 10W, it will be better to use set data structure.

GEO

GEOThe function is provided in redis 3.2, which supports the storage of geographic location information to realize the functions such as nearby location, shaking and depending on geographic location information. The data type of geo iszset

geoadd

  • Add latitude and longitude information
Geoadd citygeo 116.405285 39.904989 "Beijing"
Geoadd citygeo 121.472644 31.231706 "Shanghai"

geopos

  • Find the longitude and latitude information of the specified key. Multiple keys can be specified and returned in batch
127.0.0.1:6379 > geopos citygeo Beijing
1) "116.40528291463851929"
2) "39.9049884229125027"

geodist

  • The distance to return to two places, you can specify the unit, such as m, km, MI, FT
127.0.0.1:6379 > geodist citygeo Beijing Shanghai
"1067597.9668"
127.0.0.1:6379 > geodist citygeo Beijing Shanghai km
"1067.5980"

georadius

  • Take the given longitude and latitude as the center, and return all position elements of the position elements contained in the key whose distance from the center does not exceed the given maximum distance
georadius cityGeo 116.405285 39.904989 100 km WITHDIST WITHCOORD ASC COUNT 5

You can specify the return distance with dist, the longitude and latitude with coord, and the geohash value with hash
ASC or desc can be specified to sort by distance
You can specify count to limit the number of records returned

georadiusbymember

  • Like geordius, however, the center point is determined by a given location element, rather than using the entered longitude and latitude as geordius does.
Geordiusbmember citygeo Beijing 100 km with dist with cool ASC count 5

zrem

  • Geo does not provide the command to delete members, but because the underlying implementation of geo is Zset, the zrem command can be used to delete geographic location information
Zrem citygeo Beijing

Bloom filter

In essence, the bloom filter is a kind of data structure, which is more ingenious and probabilistic. It is characterized by efficient insertion and query, and can be used to tell you that “something must not exist or may exist”.

Compared with the traditional list, set, map and other data structures, it is more efficient and takes less space, but the disadvantage is that the returned results are probabilistic rather than exact.

principle

Its essence is a bit array containing only 0 and 1. Specific operation when an element is added to the set, the element obtains the value of k hash through the operation of k hash functions, then maps the K value to the corresponding position of the digit group, and sets the value of the corresponding position to 1. When querying whether there is a mapping point, we can see that if the mapping point positions are all 1, it is likely to exist (related to the number of hash functions and the design of hash functions). If there is a position of 0, then this element must not exist.

If we want to map a value to the bloon filter, we need to generate multiple hash values using multiple different hash functions, and generate the bit position 1 of each generated hash value, for example, for the value “Baidu” and three different hash functions, respectively, the hash values 1, 4 and 7 are generated, then the above figure changes to
Redis's first acquaintance

OK, let’s save another value “Tencent”, if the hash function returns 3, 4, 8

Redis's first acquaintance

It is worth noting that bit 4 is overwritten because the hash function of both values returns this bit. Now if we want to query whether the value of “Dianping” exists, the hash function returns three values: 1, 5 and 8. As a result, we find that the value of bit 5 is 0, which means that no value is mapped to the bit bit. Therefore, we can say that the value of “Dianping” does not exist. When we need to query whether the value of “Baidu” exists, then the hash function will inevitably return 1, 4, 7, and then we check that the values on the three bits are all 1, so can we say that “Baidu” exists? The answer is no, only “Baidu” may exist.

Why is that? The answer is simple. As more and more values are added, there will be more and more bits set to 1. In this way, if a value “Taobao” has not been stored, if all three bits returned by the hash function are set to 1 by other values, the program will still judge that the value “Taobao” exists.

Application scenario

Common application scenarios include using Bloom filter to reduce disk IO or network requests, because once a value must not exist, we can avoid subsequent expensive query requests, for example, it can be used to solveCache penetrationThe problem.

Conclusion

Today, I have a preliminary understanding of redis. Next, I will discuss redis: advanced redis

reference resources

Redis single row
Detailed explanation of Bloom filter

Recommended Today

What are the new methods of visual + map technology?

Last week, Ren Xiaofeng, chief scientist of Alibaba Gaode map, made a technical exchange with you on the development of computer vision related technology and the application in the field of map travel at the online live broadcast activity of “cloud dialogue” between senior students of Alibaba. The interaction between live broadcast is hot. Especially […]