At the beginning of the development of web application, relational database was widely concerned and applied at that time, the reason is that web site access and concurrency were not high and interaction was less. Later, with the increase of access, web sites using relational databases began to have some performance bottlenecks, and the source of bottlenecks is generally in the disk I / O. With the further development of Internet technology, various types of applications emerge in an endless stream, which leads to more demand for performance in today’s era of cloud computing and big data, which is mainly reflected in the following four aspects:
- Low latency reading and writing speed: fast application response can greatly improve user satisfaction
- In terms of large-scale data and traffic support, it is necessary to use the massive data level
- Large scale cluster management: system administrators hope that distributed applications can be more easily deployed and managed
- Consideration of huge operation cost: it department hopes to reduce hardware cost, software cost and labor cost greatly
In order to overcome this problem, NoSQL emerges as the times require. It has the advantages of high performance, strong scalability and high availability, which is favored by developers and warehouse managers.
What is redis
Redis is one of the most popular NoSQL databases. Redis is an open source, network-based, memory based, and optional persistence key value pair storage database written in ANSI C. It has the following features:
- Run based on memory, high performance
- It supports distributed and can be expanded infinitely in theory
- Key value storage system
- Open source uses ANSI C language to write, abides by BSD protocol, supports network, can be based on memory or persistent log type, key value database, and provides a variety of language API
Compared with other database types, redis has the following characteristics:
- C / s communication model
- Single process and single thread model
- Rich data types
- Operations are atomic
- High concurrency read and write
- Support Lua scripts
Which large factories are using redis?
- Stack Overflow
What are the application scenarios of redis?
Redis’s application scenarios include: cache system (hot data: high-frequency read, low-frequency write), counter, message queuing system, leaderboard, social network and real-time system.
Data types and main features of redis
The data types provided by redis are mainly divided into five types: string type, hash type, list type, collection type and ordered collection type.
It is a binary safe string, which means that it can not only store strings, but also store pictures, videos and other types. The maximum length of the string is 512M.
For each data type, redis provides rich operation commands, such as:
The type is a map consisting of a field and an associated value. Where field and value are of string type.
The operation command of hash is as follows:
This type is a set of string elements sorted by insertion order, which is implemented based on double linked list.
List operation commands are as follows:
Set type is a kind of unordered collection. The biggest difference between set type and list type is that the elements in the set have no order and the elements are unique.
The bottom layer of set type is implemented through hash table, and its operation commands are as follows:
Set type is mainly applied to: in some scenarios, such as social scenes, through the operation of intersection, union and difference set, it is very convenient to find common friends, common concerns and common preferences through set type.
Order set type:
Zset is an ordered set type, each element will be associated with a double type fractional weight, through which the members of the set are sorted from small to large. Like the set type, its underlying layer is implemented through a hash table.
Data structure of redis
The data structure of redis is shown in the following figure:
Some definitions in the table above:
- Compressed list is one of the underlying implementations of list key and hash key. When a list key contains only a small number of list items, and each list item is either a small integer or a short string, redis will use the compressed list as the underlying implementation of the list key
- The integer set is one of the low-level implementation of the set key. When a set contains only integer value elements and the number of elements in the set is not large, redis will use the integer set as the low-level implementation of the set key
The following is an example of defining a struct data structure:
Simple dynamic string SDS (simple dynamic string)
Based on the defects of traditional strings in C language, redis constructs an abstract type called simple dynamic string, referred to as SDS. Its structure is as follows:
SDS runs through almost all the data structures of redis and is widely used.
Features of SDS
Compared with C string, SDS has the following features:
1. Get string length by constant complexity
In redis, the length of the saved string can be obtained directly by using the len attribute of SDS string
The complexity of obtaining the length of a string is reduced directly from O (n) of C string to o (1).
2. Reduce the number of memory reallocation caused by string modification
Through the characteristics of C string, we know that for a C string containing n characters, its underlying implementation is always an array of N + 1 characters (with an additional null character ending)
If the string needs to be modified at this time, the program needs to reallocate the memory of the C string array in advance (it may be extended or released)
Memory reallocation is a time-consuming operation.
Redis skillfully uses SDS to avoid the defects of C string. In SDS, the length of the buf array is not necessarily the number of characters in the string plus one. The buf array can contain unused bytes, which are recorded by the free attribute.
At the same time, SDS adoptedSpace pre allocationTo avoid the time-consuming operation of memory reallocation every time the C string is modified, the memory reallocation is reduced from the original n times of modification to N times of modification and the maximum number of times of memory reallocation.
The following is a simple definition of SDS by redis:
Redis feature 1: transaction
- Commands are serialized and executed sequentially
- Three phases: start transaction, command join queue and execute transaction
- Command: multi / exec / discard
Redis feature 2: publish subscribe (Pub / sub)
- Pub / sub is a mode of message communication
- Pub sends the message and sub receives the message
- Redis client can subscribe to any number of channels
- “Fire and forget”
- Command: publish / subscribe / psubscribe / unsubscribe
Redis feature 3: stream
- Redis 5.0 NEW
- Waiting for consumption
- Consumer group (intra group competition)
- Consumption history data
The above is the basic concept of redis. Next, we will introduce the “pit” that may be stepped on in the development process.
Analysis of redis common problems: breakdown
Concept: when redis obtains a key, it must send a request to DB because the key does not exist, which is called “redis breakdown”.
The causes of breakdown are as follows:
- First visit
- Malicious access to nonexistent key
- Key expired
Reasonable evasion scheme:
- Write ahead when the server starts
- The naming of key is standardized and intercepted by middleware
- For some high-frequency access keys, set a reasonable TTL or never expire
Redis FAQ: Avalanche
Concept: after the redis cache layer is down for some reason, all requests will flow to the storage layer. High concurrency requests in a short period of time may cause the storage layer to hang up, which is called “redis avalanche”.
Reasonable evasion scheme:
- Using redis cluster
- Current limiting
Application of redis in product development
Therefore, I am very happy to introduce to you that Jim, the architect of grape City, will bring you an open class at 14:00 on November 27, 2019 In addition to explaining the foundation of redis, Jim will also demonstrate the problems and solutions encountered by his project team when using redis. For students who have just met redis, it is of more reference significance and learning value. Welcome to attend the open class at: https://live.vhall.com/661463644 。
- The back end adopts nodejs
- Using redis service of azure
- Usage scenarios of redis
-Token cache for token verification
- “Network jitter” or redis service exception causes redis access timeout
- The stability of redis client driver
-Connection pool “broken connection” problem
-Redis reset problem caused by promise of JS
Let’s take a brief look at the advanced knowledge of redis.
Introduction of advanced redis protocol
Redis client communication protocol: resp (redis serialization protocol)
- The analysis speed is fast
- Good readability
Redis cluster internal communication protocol: RECP (redis cluster protocol), which is characterized by:
- Two TCP connections per node
- One is responsible for client server communication (P: 6379)
- One is responsible for communication between nodes (P: 10000 + 6379)
Data types supported by redis protocol:
- Simple character (first byte: +)
- Error (first byte: ‘-‘)
- Number (first byte::)
- Batch character (first byte: $)
“&hello\r\nWhoa re you\r\n”
- Array (first byte: *)
Besides redis, what are the NoSQL databases
There are many databases similar to redis but also NoSQL type. As shown in the figure below, in addition to redis, there are Memcache, cassadra and Mongo. Now, we will give a brief introduction to these databases
Memcache：This is a database very similar to redis, but its data type is not as rich as redis. Memcache is developed by Brad Fitzpatrick of livejournal. As a distributed cache system, Memcache is used by many websites to improve the access speed of websites. For some large websites that need frequent access to databases, Memcache can improve the access speed significantly.
Apache Cassandra：This is an open source distributed NoSQL database system. It was originally developed by Facebook to store data in simple formats such as inboxes. It integrates the data model of Google BigTable and the fully distributed architecture of Amazon dynamo. Facebook opened Cassandra in 2008. Due to its good scalability and performance, it has been adopted by apple, Comcast, instagram, spotify, eBay, Rackspace, Netflix and other well-known websites, and has become a popular distributed structured data storage scheme.
MongoDB: is a NoSQL database based on distributed file storage and document oriented, written by C + +, aiming to provide scalable and high-performance data storage solutions for web applications. Mongodb is a product between relational database and non relational database. Mongodb is the most functional and relational database among non relational databases. It supports a very loose data structure and is a bson format similar to JSON.
The above is the introduction to redis. If you want to know more, please let me know through comments and private messages.