Elasticsearch Foundation

Time:2022-1-10

1、 Basic concepts of ES

1.1 index

An index is a collection of documents with somewhat similar characteristics. An index is identified by a name (required)
It must be all lowercase letters), and we should use this name when we want to index, search, update and delete the documents in this index. Any number of indexes can be defined in a cluster.

The index structure mainly includes mapping and setting

1.1.1 mapping

Mapping makes some restrictions on the way and rules of processing data, such as the data type, default value, analyzer, whether to be indexed, etc. These can be set in the mapping. Others are some use rule settings for processing data in ES, also known as mapping. Processing data according to the optimal rules will greatly improve the performance. Therefore, it is necessary to establish a mapping and think about how to establish a mapping to improve the performance.

It mainly includes the following contents:
1. Defines the name of the field in the index
2. Define the data type of the field, such as string, number, Boolean, etc
3. The related configuration of inverted index can be set for the field, such as whether word segmentation is required and what word splitter is used

Elasticsearch supports the following simple domain types
String; Integer (byte, short, integer, long); Floating point number (float, double); Boolean; Date (date, format is strict_date_optional_time|epoch_millis by default, which means yyyy HH mmthh: mm: SSZ or timestamp format); Geographical location (geo_point);

Through the put request, you can add or modify the mapping of type. Modifying the mapping can only add fields, but cannot modify fields.

PUT /gb/_mapping/tweet
{ "properties" : { "tag" : { "type" : "string", "index": "not_analyzed" } }}

Field definitions in mappings have the following options

"field": {  
    "Type": "text"
    "Index": "analyzed", // the attribute has three valid values: analyzed, no and not_ Analyzed, which is analyzed by default; Analyzed: indicates that the field is analyzed and indexed, and the generated token can be searched; not_ Analyzed: indicates that the field will not be analyzed, and the original value will be indexed as a single word in the index; No: this field cannot be searched because it is not indexed;
    "Analyzer": "IK" // specify a word breaker  
    "Boost": 1.23 // score weighting at field level  
    "Doc_values": false // not_ The analyzed field is enabled by default. The analyzed field cannot be used. Sorting and aggregation can improve performance and save memory. If you are sure that you do not need to sort or aggregate fields, or access field values from script, you can disable the doc value to save disk space
    "Fielddata": {"loading": "eager"} // elasticsearch the default behavior of loading memory fielddata is to delay loading. When elasticsearch queries a field for the first time, it will fully load the inverted indexes in all segments of the field into memory, so that future queries can obtain better performance.
    "Fields": {"keyword": {"type": "keyword", "ignore_above": 256} // multiple index modes can be provided for a field. The value of the same field, one word segmentation and one word non segmentation  
    "Ignore_above": 100 // text with more than 100 characters will be ignored and not indexed
    "Include_in_all": true // set whether this field is included in_ In the all field, the default is true unless index is set to the no option  
    "Index_options": "docs" // 4 optional parameters docs (index document number), freqs (document number + word frequency), positions (document number + word frequency + position, which is usually used for distance query), offsets (document number + word frequency + position + offset, which is usually used in the highlighted field). The word segmentation field is position by default, and others are docs by default  
    "Norms": {"enable": true, "loading": "lazy"} // the word segmentation field is configured by default. The non word segmentation field is {"enable": false} by default. It is boost when storing the length factor and index. It is recommended to use the field that needs to participate in scoring, which will increase the memory consumption  
    "Null_value": "null" // set the initialization values of some missing fields. Only strings can be used. The null value of the word segmentation field will also be used by the word segmentation "position_increment_gap": 0 // affect the distance query or approximate query. It can be set on the word segmentation field on the data of the multi value field. When querying, you can specify the slop interval. The default value is 100  
    "Store": false // whether to set the storage of this field separately from_ The source field is separated from the source field. The default value is false. You can only search and cannot obtain values  
    "Search_analyzer": "IK" // set the word splitter during search. By default, it is consistent with the analyzer. For example, standard + NGram is used for index and standard is used for automatic prompt during search  
    "Similarity": "BM25" // the default is TF / IDF algorithm. It specifies a field scoring strategy, which is only valid for string and word segmentation types  
    "Term_vector": "no" // vector information is not stored by default. The parameter Yes (term storage) and with are supported_ Positions (term + position), with_ Offsets (term + offset), with_ positions_ Offsets (term + position + offset) can improve the performance of fast vector highlight, but opening it will increase the index volume, which is not suitable for large amount of data  
}

1.1.2 Setting

Setting is the configuration property of the ES index.

The configuration items of the index can be divided into static attribute and dynamic configuration according to whether they can be changed. The so-called static configuration means that the index cannot be modified after it is created. The static configuration can only be set when the index is created or on the index in closed index status (closed index):

  • Index static configuration
index. number_ of_ Shards: the number of primary shards, which cannot be modified. The default number of slices to create an index library is 1. In elastic search before 7.0.0, the default number is 5.

index. shard. check_ on_ Startup: whether to check whether the partition is damaged before the index is opened. When the partition is detected to be damaged, the partition will not be opened
Optional value: false: do not detect; Checksum: check only the physical structure; True: check for physical and logical damage, and compare CPU consumption; Fix: similar to false. It will be discarded after version 7.0. Default: false.

index. Codec: compression algorithm for data storage. The default algorithm is lz4, which can also be set to best_ compression,best_ Compression compression is better, but the storage performance is worse than lz4

index. routing_ partition_ Size: the number of routing partitions. The default value is 1. It can only be set when the index is created. This value must be less than index number_ of_ Shards, if this parameter is set, its routing algorithm is: (hash (_routing) + hash (_id)% index routing_ parttion_ size ) % number_ of_ shards。 If this value is not set, the routing algorithm is hash (_routing)% number_ of_ shardings,_ The default value of routing is_ id。
  • Index dynamic configuration
index. number_ of_ Replicas: the number of replicas of each primary partition. The default value is 1. This value must be greater than or equal to 0

index. auto_ expand_ Replicas: automatically allocate the number of replicas based on the number of available nodes. The default value is false (that is, this function is disabled)

index. refresh_ Interval: the frequency at which refresh operations are performed, which makes the most recent changes to the index searchable. The default is 1s. You can set it to - 1 to disable refresh.
 
index. max_ result_ Window: the maximum value of from + size used for index search. The default is 10000
 
index. max_ rescore_ Window: the window in which the rescore is searched in this index_ Maximum size
 
index. blocks. read_ Only: set to true to make the index and index metadata read-only, and false to allow writing and metadata change.

index. blocks. read_ only_ allow_ Delete: and index blocks. read_ Only is basically similar. The only difference is that delete actions are allowed.
 
index. blocks. Read: set to true to disable reading of the index
 
index. blocks. Write: set to true to disable writing to the index.
 
index. blocks. Metadata: set to true to disable reading and writing of index metadata.
 
index. max_ refresh_ Listeners: the maximum number of refresh listeners available on each slice of the index

index. max_ docvalue_ fields_ Search: a query can contain at most open docs_ The number of values fields. The default value is 100.

1.2 type

In an index, you can define one or more types.
A type is a logical classification / partition of your index, and its semantics is entirely up to you. Usually, there will be
A document with a common set of fields defines a type. Different versions and types have changed

edition Type
5.x Support multiple types
6.x There can only be one type
7.x Custom index types are no longer supported by default (the default type is: _doc)

1.3 document

A document is a basic information unit that can be indexed, that is, a piece of data
For example, you can have a customer’s document, a product’s document, and of course, you can also have a customer’s document
A document of the order. Documents are represented in JSON (JavaScript object notation) format, and JSON is a
Ubiquitous Internet data interaction formats.
In an index / type, you can store as many documents as you want.

1.4 field

It is equivalent to the field of the data table, which classifies and identifies the document data according to different attributes.

1.5 shards

An index can store a large amount of data that exceeds the hardware limit of a single node. To solve this problem, elastic search provides the ability to divide the index into multiple copies, each of which is called sharding. When you create an index, you can specify the number of slices you want. Each partition itself is also a fully functional and independent “index”, which can be placed on any node in the cluster.

Segmentation is very important for two reasons:
1) Allows you to split / expand your content horizontally.
2) It allows you to perform distributed and parallel operations on shards to improve performance / throughput.

How a fragment is distributed, how its documents are aggregated and how search requests are managed by elasticsearch. For you as a user, these are transparent and don’t need to care too much.
The confused concept is that a Lucene index is called sharding in elastic search. An elasticsearch index is a collection of fragments. When elastic search searches in the index, it sends a query to each partition belonging to the index (Lucene index), and then combines the results of each partition into a global result set.

1.6 replicas

In a network / cloud environment, failures can occur at any time. In this case, a partition / node is somehow offline or disappears for any reason. In this case, a failover mechanism is very useful and highly recommended. For this purpose, lasticsearch allows you to create one or more copies of shards, which are called replica shards (replicas).

There are two main reasons why replication fragmentation is important:
1) High availability is provided in case of fragmentation / node failure. For this reason, it is important to note that the replication shard is never placed on the same node as the original / primary shard.
2) Expand your search volume / throughput becauseSearch can run in parallel on all replicas.

The number of shards and copies can be specified when the index is created. After the index is created, you can
In orderDynamically change the number of replications at any timeBut you can’t change the number of slices afterwards. By default, each index in elasticsearch is partitioned, 1 master partition and 1 copy.

1.7 allocation

The process of assigning shards to a node, including assigning primary shards or replicas. If it is a replica, it also includes the process of copying data from the primary shard. This process is completed by the master node.

2、 System architecture

Elasticsearch Foundation

image.png

2.1 cluster

A running elasticsearch instance is called a node, and a cluster consists of one or more clusters with the same cluster Name is composed of nodes configured by. They jointly bear the pressure of data and load. When a node joins the cluster or removes a node from the cluster,The cluster will redistribute all data equally

Responsibility of master node:

  • Manage all changes within the cluster, such as adding and deleting indexes, or adding and deleting nodes.
  • It is not responsible for document level changes and searches, so when the cluster has only one master node, it will not become a bottleneck even if the traffic increases.

As users, we can send requests to any node in the cluster, including the master node. Each node knows the location of any document and can forward our request directly to the node where we store the documents we need. No matter which node we send the request to, it can collect data from each node containing the documents we need, and return the final result to the client.

Master node and master partition are irrelevant concepts:

  • Primary node: it is responsible for managing all changes within the cluster, such as adding and deleting indexes, or adding and deleting nodes.
  • Master slice: responsible for changes to the document level.

2.2 failover (replicas)

In order to prevent data loss caused by node failure, we need to back up the data on another node. The backup of the primary node is replica fragmentation. The number of backup fragments can be specified in creating mapping or dynamically modified in cluster work.

Cluster monitoring status

  • Green: all primary and secondary partitions have been allocated.
  • Yellow: at least one replica fragment is unassigned
  • Red: at least one ink slice is not allocated

2.3 horizontal expansion

The amount of data that can be stored on each node is limited, and too much data on a single node will increase the pressure on data processing. We can expand the cluster node for horizontal capacity expansion. The benefits are as follows:

  • 1. Although the number of shards cannot be modified after it is specified when creating mapping (the number of shards determines the maximum amount of data that the index can store), you can reduce the distribution of shards on each host by adding nodes. Because a new node is started, the cluster will redistribute the fragments in order to distribute the load;
  • 2. If the number of nodes increases, more replicas can be configured. Read operations (search and return data) can be processed by the master shard or replica shard at the same time, so the more replica shards you have, the higher throughput you will have.

2.4 troubleshooting

If a node goes down, the primary partition on this node will fail. If there are replica partitions on other nodes, the cluster will promote the replica partitions on other nodes to the primary partition. This process occurs instantaneously. If there is no replica fragmentation, the cluster health status will change to red, indicating that there are primary nodes that are not allocated.

2.5 route calculation

When a document is indexed, it is stored in a main partition. When we create a document, how does it decide which slice the document should be stored in? This is determined according to the following formula:

shard = hash(routing) % number_of_primary_shards

Routing is a variable value, which is the default value of the document_ ID, which can also be set to a user-defined value.

Therefore, the cluster partition number cannot be modified after it is specified during creation, because after modification, the previous routing values will become invalid and the document will no longer be found.

All document APIs (get, index, delete, bulk, update and mget) accept a routing parameter called routing. Through this parameter, we can customize the mapping of documents to fragments. A custom routing parameter can be used to ensure that all related documents – for example, all documents belonging to the same user – are stored in the same partition.

2.6 slice control

We can send requests to any node in the cluster. Each node has the ability to handle arbitrary requests. Each node knows any document location in the cluster, so it can forward the request directly to the required node. In the following example, all requests are sent to node 1, which we call coordinating node.

When sending requests, in order to expand the load, it is better to poll all nodes in the cluster.

2.6.1 writing process

New, index and delete requests are all write operations. They must be completed on the primary partition before they can be copied to the relevant replica partition.

Elasticsearch Foundation

image.png

The sequence of steps required to create, index, and delete documents:

  1. The client sends a new, index or delete request to node 1.
  2. Node uses the of the document_ ID determines that the document belongs to fragment 0. The request will be forwarded to node 3 because the primary partition of partition 0 is currently assigned to node 3.
  3. Node 3 executes the request on the primary partition. If successful, it forwards the request in parallel to the replica shards of node 1 and node 2. Once all replica shards report success, node 3 will report success to the coordination node, and the coordination node will report success to the client.

When the client receives a successful response, the document change has been completed in the primary partition and all replica partitions, and the change is safe. There are some optional request parameters that allow you to affect this process, possibly improving performance at the expense of data security. These options are rarely used because elasticsearch is fast, but for completeness, please refer to the following table:

parameter meaning
consistency Consistency is consistency. By default, even if you’re just trying to execute awriteBefore the operation, the master shard requires that a specified number of shard copies (or in other words, most of them) must be active and available before executionwriteOperation (where the sharded replica can be a master shard or a replica shard). This is to avoid network partition failurewriteOperation, resulting in inconsistent data.Specified quantityNamely:
int( (primary + number_of_replicas) / 2 ) + 1
The value of the consistency parameter can be set toone(it is allowed to execute as long as the main partition status is OKwriteOperation),all(it is allowed to execute only when the status of the primary partition and all replica partitions is OKwriteOperation), orquorum。 The default value is quorum, which means that most partitioned copies are allowed to be executed if the status is OKwriteOperation.
Note that number is in the calculation formula of the specified quantity_ of_ Replicas refers to the number of replica fragments set in the index settings, not the number of replica fragments in the current processing active state. If your index setting specifies that the current index has three replica fragments, the calculation result of the specified number is:
int( (primary + 3 replicas) / 2 ) + 1 = 3
If you only start two nodes at this time, the number of active fragmented copies will not reach the specified number, so you will not be able to index and delete any documents.
timeout What happens if there are not enough replica shards? Elasticsearch will wait and hope that more pieces will appear. By default, it waits up to 1 minute. If you need to, you can use the timeout parameter to make it terminate earlier: 100 milliseconds, 30s is 30 seconds.

Note: the new index has 1 replica shard by default, which means that two active shard replicas should be required to meet the specified number. However, these default settings prevent us from doing anything on a single node. To avoid this problem, it is required only when number_ of_ When replicas is greater than 1, the specified quantity will be executed.

2.6.2 reading process

Elasticsearch Foundation

image.png

The sequence of steps to retrieve a document from a master slice or a replica slice:

  1. The client sends an acquisition request to node 1;
  2. Node uses the of the document_ ID to determine that the document belongs to fragment 0. Replica shards of shard 0 exist on all three nodes. In this case, it forwards the request to node 2;
  3. Node 2 returns the document to node 1, and then returns the document to the client.

When processing read requests, the coordination node will pass each requestPoll all replica shardsTo achieve load balancing. When the document is retrieved, the indexed document may already exist on the primary partition but has not been copied to the replica partition. In this case, the replica shard may report that the document does not exist, but the master shard may successfully return the document. Once the index request is successfully returned to the user, the document is available in both primary and replica Shards.

2.6.3 update process

Elasticsearch Foundation

image.png

The steps to partially update a document are as follows:

  1. The client sends an update request to node 1.
  2. It forwards the request to node 3 where the primary partition is located.
  3. Node 3 retrieves the document from the main partition and modifies it_ JSON in the source field and try to re index the document of the main fragment. If the document has been modified by another process, it will retry step 3, exceeding retry_ on_ Give up after conflict times. (optimistic lock is used to control concurrency, and one will be added every time the data _versionis modified)
  4. If node 3 successfully updates the document, it forwards the new version of the document to the replica fragments on node 1 and node 2 in parallel and re establishes the index. Once all replica fragmentation returns success, node 3 returns success to the coordination node, and the coordination node returns success to the client.

When the master shard forwards changes to the replica shard, it does not forward update requests. Instead, it forwards the new version of the complete document (only the latest _versionversion is taken to ensure that the latest version is synchronized). Remember that these changes will be forwarded asynchronously to the replica shards, and there is no guarantee that they will arrive in the same order in which they were sent. If elasticsearch only forwards change requests, the changes may be applied in the wrong order, resulting in a corrupted document.

2.6.4 multi document operation process

The schema of the mget and bulk APIs is similar to the single document schema. The difference is that the coordination node knows which partition each document exists in. It decomposes the whole multi document request into multi document requests of each fragment, and forwards these requests to each participating node in parallel.
Once the coordination node receives the response from each node, it collects and arranges the response of each node into a single response and returns it to the client;

The sequence of steps required to retrieve multiple documents with a single mget request:

Elasticsearch Foundation

image.png
  1. The client sends an mget request to node 1;
  2. Node 1 constructs multiple document acquisition requests for each partition, and then forwards these requests in parallel to the nodes hosted on each required primary partition or replica partition. Once all replies are received, node 1 builds the response and returns it to the client.
    You can set the routing parameter for each document in the docs array.

Bulk API, which allows multiple create, index, delete and update requests to be performed in a single batch request:

Elasticsearch Foundation

image.png
  1. The client sends a bulk request to node 1.
  2. Node 1 creates a batch request for each node and forwards these requests to each node host containing the main partition in parallel.
  3. The main tiles perform each operation in sequence one by one. When each operation is successful, the master shard forwards the new document (or delete) to the replica shard in parallel, and then performs the next operation. Once all replica fragmentation reports that all operations are successful, the node will report success to the coordination node, which will collect and sort out these responses and return them to the client.

2.7 shard internal principle

Slicing is the smallest work unit of elasticsearch. Traditional databases store a single value for each field, but this is not enough for full-text retrieval. Each word in the text field needs to be searched. For the database, it means that a single field needs the ability to index multiple values. The best support is that the data structure required for multiple values of a field isInverted index

2.7.1 inverted index

Elastic search uses a structure called inverted index, which is suitable for fast full-text search. An inverted index consists of a list of all non repeating words in the document. For each word, there is a document list containing it.

The inverted index consists of the following parts:

  • Doc list of a keyword
  • Number of all docs for a keyword IDF (reverse document frequency: divide the total number of documents by the number of documents with word changes, and then take the logarithm)
  • Number of times a keyword appears in each doc: TF (word frequency: frequency of words in this document)
  • The order of a keyword in this doc
  • Length of each doc: length norm
  • Average length of all docs for a keyword

The purpose of recording this information is to facilitate the efficiency and efficiency of search_ Calculation of score.

2.7.2 word segmentation and standardization (analysis)

You can only search for terms that appear in the index, so the index text and query string must be standardized to match
Same format.

Analyzer is the analyzer, which includes

  • Character filtercharacterfilters (0 or more): first, the string passes through each character filter in order. Their task is to sort out strings before word segmentation. A character filter can be used to remove HTML or convert & to and.
  • Word breaker tokenizer (1): secondly, the string is divided into a single entry by the word breaker. When a simple word splitter encounters spaces and punctuation, it may split the text into entries. For example, the whitespace word breaker will divide entries by space characters.
  • Post filter tokenfilter (0 or more than 1): finally, entries pass through each token filter in order. This process may change entries (for example, lowercase quick), delete entries (for example, useless words such as a, and, the), or add entries (for example, synonyms such as jump and leap).
Elasticsearch Foundation

image.png

2.7.3 dynamic update index

Early full-text retrieval would build a large inverted index for the whole document collection and write it to disk. Once the new index is ready, the old one will be replaced by it so that the latest changes can be retrieved.

The benefits of invariance are as follows:

  • Thread safety: thread safety, without considering multithreading.
  • If the inverted index cannot be changed, the data in memory will not change. As long as there is enough space in the file system cache, most read requests will directly request memory instead of hitting the disk. This provides a significant performance improvement.
  • Other caches (such as filter cache) are always valid throughout the life cycle of the index. They do not need to be reconstructed every time the data changes because the data does not change.
  • Writing a single large inverted index allows data to be compressed, reducing the use of disk I / O and indexes that need to be cached in memory.

How to update inverted index while preserving invariance?
Instead of directly rewriting the entire inverted index, the new index is added to reflect the latest changes. This introducesSearch by segmentEach segment is an inverted index, and each inverted index will be queried in turn. After the query is completed from the earliest, the results will be merged. The concept of commit point is also added.

  • Segment file: for files storing inverted indexes, each segment is essentially an inverted index, and a segment file will be generated every second. When there are too many files, es will automatically perform segment merge, and the deleted documents will be physically deleted at the same time;
  • Commit point: record all currently available segments, one for each commit point Del file (ES deleting data is not a physical deletion in nature). When es does delete and modify, it will be deleted at first The Del file declares that a document has been deleted. It records that a document in a segment has been deleted. When the query request comes, the deleted file in the segment can be found, but when the result is returned, it will be maintained according to the commit point Del file filters out deleted documents;
  • Translog log file: in order to prevent data loss caused by the downtime of elastic search and ensure reliable storage, es will write the data to the translog log at the same time.

2.7.4 near real time search

With the development of per segment search, the delay of a new document from indexing to searchable is significantly reduced. New documents can be retrieved in minutes, but it’s not fast enough. Disk has become a bottleneck here. Committing a segment to disk requires a fsync to ensure that the segment is physically written to disk, so that data will not be lost in case of power failure. But fsync operation is expensive; If you index a document every time, it will cause great performance problems.
This means that fsync cannot be triggered frequently, but it will take effect only after the section is really closed. Therefore, the longer the fsync time, the longer the delay that new documents can be retrieved after insertion. In order to realize near real-time search and prevent data loss, the new segment will be written to the file system cache (refresh), which will be relatively cheap, and then flushed to the disk (flush) later, which will be relatively expensive. Fsync only synchronizes to disk as a translog log file.

  • refresh: es when receiving a data request, it is stored in the memory first. By default, the data will be written to the file system cache from the memory buffer every second. This process is called refresh;
  • flush: ES by default, the data in the file system cache will be flushed to the disk every 30 minutes, and the translog log file will be emptied. This process is called flush.
  • fsync: translog will execute fsync every 5 seconds or after a change request is completed to flush the translog from the cache to the disk. This operation is time-consuming. If the data consistency requirements are not high, it is recommended to change the index to asynchronous. If the node goes down, data will be lost for 5 seconds;

Segment merging process
Because the automatic refresh process will create a new segment per second, this will lead to a sharp increase in the number of segments in a short time. Too many segments will cause great trouble. Each segment consumes file handles, memory, and CPU cycles. More importantly, each search request must check each segment in turn; So the more segments, the slower the search.
When segments are merged, those old deleted documents will be cleared from the file system. The deleted document (or the old version of the updated document) will not be copied to the new large section.

Starting segment merging doesn’t require you to do anything. Indexing and searching are performed automatically:

  1. When indexing, the refresh operation creates a new segment and opens the segment for search.
  2. The merge process selects a small number of segments of similar size and merges them into larger segments in the background. This does not interrupt indexing and searching.
  3. Once the merge is completed, the old segment is deleted.

Merging large segments requires a lot of I / O and CPU resources. If it is allowed to develop, it will affect the search performance. Elastic search imposes resource constraints on the merge process by default, so there are still enough resources to perform the search well.

The data writing process of the complete elasticsearch is as follows:

Elasticsearch Foundation

2.8 document processing

2.8.1 document conflict

When multiple clients update documents concurrently, the latest updated data will be saved as the final result and the whole document will be re indexed. In general, there is no problem.
However, concurrency control is required in some scenarios, such as inventory problems. You need to query the inventory balance first, and then reduce the number of items sold. This will cause oversold problems.

There are generally two ways to solve this concurrency problem:

  • Pessimistic concurrency control: add an exclusive lock when accessing data. After unlocking, it can be accessed by other processes.
  • Optimistic concurrency control: obtain the version number when accessing data. When modifying data, only when the version number is greater than or equal to the version number of data can it be modified. After modification, the version number of data is increased by one.

Optimistic locks are used by default in es:
When the document is successfully added, the of the current document will be returned_ version、_ seq_ No and_ primary_ Term information. These are version number related attributes.

{
    "_index": "smartcook",
    "_type": "_doc",
    "_id": "4",
    "_version": 1,
    "result": "created",
    "_shards": {
        "total": 2,
        "successful": 1,
        "failed": 0
    },
    "_seq_no": 2,
    "_primary_term": 8
}

When modifying this document, you can carry the version number information. If the version number information is wrong, the modification will fail.

http://chenjie.asia:9200/smartcook/_doc/4?if_seq_no=3&if_primary_term=8

#Return
{
    "_index": "smartcook",
    "_type": "_doc",
    "_id": "4",
    "_version": 3,
    "result": "updated",
    "_shards": {
        "total": 2,
        "successful": 1,
        "failed": 0
    },
    "_seq_no": 4,
    "_primary_term": 8
}

2.8.2 optimistic concurrency control

Elasticsearch is distributed. When a document is created, updated, or deleted, the new version of the document must be copied to other nodes in the cluster. Elastic search is also asynchronous and concurrent, which means that these replication requests are sent in parallel and may be out of order when they arrive at the destination. Elasticsearch needs a way to ensure that the old version of the document does not overwrite the new version.
When we discussed index, get and delete requests earlier, we pointed out that each document has one_ Version number, which is incremented when the document is modified. Elastic search uses this version number to ensure that changes are executed in the correct order. If the old version of the document arrives after the new version, it can be simply ignored.
We can use the version number to ensure that conflicting changes in the application will not lead to data loss. We do this by specifying the version number of the document we want to modify. If the version is not the current version number, our request will fail.
The old version of ES uses version, but the new version does not support it. The following error will be reported to prompt us to use if_ seq_ No and if_ primary_ term。

{
 "error": {
 "root_cause": [
 {
 "type": "action_request_validation_exception",
 "reason": "Validation Failed: 1: internal versioning can not be used 
for optimistic concurrency control. Please use `if_seq_no` and `if_primary_term` 
instead;"
 }
 ],
 "type": "action_request_validation_exception",
 "reason": "Validation Failed: 1: internal versioning can not be used for 
optimistic concurrency control. Please use `if_seq_no` and `if_primary_term` 
instead;"
 },
 "status": 400
}

2.8.3 version control of external system

A common setting is to use other databases as the main data storage and elasticsearch for data retrieval, which means that all changes to the main database need to be copied to elasticsearch. If multiple processes are responsible for this data synchronization, you may encounter concurrency problems similar to those described earlier.
If your primary database already has a version number – or a field value that can be used as the version number, such as timestamp   – you can add version in elastic search_ Reuse these same version numbers by type = external to the query string. The version number must be an integer greater than zero and less than 9.2e + 18 𔁑 – a positive value of long type in Java.
The processing method of external version number is somewhat different from that of internal version number discussed earlier. Elasticsearch does not check the current version number_ Whether the version and the version number specified in the request are the same, but check the current version_ Whether version is less than the specified version number. If the request is successful, the external version number is used as the new version of the document_ Version.

http://chenjie.asia:9200/smartcook/_doc/4?version=4&version_type=external

#Return
{
    "_index": "smartcook",
    "_type": "_doc",
    "_id": "4",
    "_version": 4,
    "result": "updated",
    "_shards": {
        "total": 2,
        "successful": 1,
        "failed": 0
    },
    "_seq_no": 5,
    "_primary_term": 8
}

The external version number can be specified not only in index and delete requests, but also when creating new documents.

3、 Operation command

Elasticsearch Foundation

21580557-355e7ba2dc10b507.png

3.1 basic command

To view the fragmented copy of es:

http://chenjie.asia:9200/_cat/shards
http://chenjie.asia:9200/_cat/shards/{index}

To view ES’s health:

GET http://chenjie.asia:9200/_cat/health?v
GET http://chenjie.asia:9200/_cluster/health?pretty=true

View node

GET http://chenjie.asia:9200/_cat/nodes

View master node

GET http://chenjie.asia:9200/_cat/master

View all indexes

http://chenjie.asia:9200/_cat/indices
http://chenjie.asia:9200/_cat/allocation 
http://chenjie.asia:9200/_cat/thread_pool
http://chenjie.asia:9200/_cat/segments 
http://chenjie.asia:9200/_cat/segments/{index}

3.2 index

Create index

PUT http://chenjie.asia:9200/smartcook

Query index

GET http://chenjie.asia:9200/smartcook

Delete index

PUT http://chenjie.asia:9200/smartcook

Create index with setting and mapping

PUT http://chenjie.asia:9200/smartcook
{
    "setting":{
        "analysis": {
            //Custom word breaker
        },
        "index": {
            "number_of_shards": 3,
            "number_of_replicas": 1
        }
    },
    "mappings": {
            //Definition of field name and type
            "properties": {   
                //Field name
                "menu": {  
                    //Field type
                    "Type": "text", // text indicates separable words
                    "Index": true // those that can be indexed can be queried
                },
                "author": {
                    "Type": "keyword"
                    "index": true
                },
                "food": {
                    "type": "keyword",
                    "Index": false // cannot be queried
                },
                "degree": {
                    "type": "integer",
                    "index": true
                }
            }
    }
}

**Note: if fields is not specified, it will be created dynamically when inserting data.
When creating mapping, you can specify the dynamic attribute (which can be set under type or in field)

a. True: a new field is allowed to be passed in, and a new field is automatically mapped (default configuration)
b. False: a new field is allowed to be passed in, but it will not be mapped to a new field, and the field cannot be queried
c. Strict: an error will be reported if a new field is passed in
It can be set as follows

PUT http://chenjie.asia:9200/smartcook/_mapping/
{
    "dynamic":"strict"
}

When creating mapping, you can specify the dynamic attribute (which can be set under type or in field)
a. True: a new field is allowed to be passed in, and a new field is automatically mapped (default configuration)
b. False: a new field is allowed to be passed in, but it will not be mapped to a new field, and the field cannot be queried
c. Strict: an error will be reported if a new field is passed in

3.3 data operation

insert data

If the field is not defined, it is automatically set to keyword type. If not specified_ ID, random ID will be generated automatically_ id

PUT http://chenjie.asia:9200/smartcook/_doc/1
{
    "Menu": "tomato and egg",
    "author":"CJ",
    "Food": "tomatoes, eggs",
    "degree": 3
}

PUT http://chenjie.asia:9200/smartcook/_doc/2
{
    "Menu": "leek eggs",
    "author":"ZS",
    "Food": "leeks, eggs",
    "degree": 2
}

Query data

#Query data by ID
GET http://chenjie.asia:9200/smartcook/_doc/1  
#Query all data under the index
GET http://chenjie.asia:9200/smartcook/_search

Modify inserted data

Method 1: directly put a piece of data with the same ID and overwrite it.

PUT http://chenjie.asia:9200/smartcook/_doc/1
{
    "degree": 5
}

Method 2: use post request to update the specified field

POST http://chenjie.asia:9200/smartcook/_doc/1
{
    "degree": 5
}

Delete data

http://chenjie.asia:9200/smartcook/_doc/1

3.4 analyzer

3.4.1 concept

Analyzer is the analyzer, which includes

  • Character filtercharacterfilters (0 or more): first, the string passes through each character filter in order. Their task is to sort out strings before word segmentation. A character filter can be used to remove HTML or convert & to and.
  • Word breaker tokenizer (1): secondly, the string is divided into a single entry by the word breaker. When a simple word splitter encounters spaces and punctuation, it may split the text into entries. For example, the whitespace word breaker will divide entries by space characters.
  • Post filter tokenfilter (0 or more than 1): finally, entries pass through each token filter in order. This process may change entries (for example, lowercase quick), delete entries (for example, useless words such as a, and, the), or add entries (for example, synonyms such as jump and leap).
Elasticsearch Foundation

image.png

An analyzer has and only has one tokenizer, 0 or more char filters and token filters.

Elasticsearch has provided a wealth of out of the box analyzers. We can create our own token analyzer. We can even use the existing char filter, tokenizer and token filter to recombine into a new analyzer, and define our own analyzer for each field in the document.

Elasticsearch Foundation

Word segmentation effect of different analyzers

3.4.2 effect test of word splitter

The default character filters are: HTML strip (removing HTML tags and converting HTML entities), mapping (string replacement operation), and pattern replace (regular matching replacement).

The following example filters out HTML tags through characterfilters:

POST _analyze
{
"tokenizer":  "keyword",
"char_filter" : [ "html_strip" ],
"text" : "<div><h1>B<sup>+</sup>Trees</h1></div>"
}

3.4.2.1 test the effect of common word splitter

① Test keyword segmentation
GET http://chenjie.asia:9200/smartcook/_analyze
{
  "Analyzer": "keyword", // the keyword participator will not segment text
  "Text": "fried rice with Lao Ganma"    
}
② Test standard participle

By default, standard analyzer is the default analyzer of elasticsearch:

  • No char filter
  • Using standard tokonizer
  • Change the string to lowercase, and selectively delete some stop words, etc. By default, stop words isnone, that is, no stop words are filtered.
GET http://chenjie.asia:9200/smartcook/_analyze
{
  "Analyzer": "standard": // the standard participator will segment every Chinese word
  "Text": "fried rice with Lao Ganma"
}
③ Test English participle
http://chenjie.asia:9200/smartcook/_analyze
{
    "analyzer": "english",
    "text":"running apps in a phone"
}

#Get [run] [app] [phone]

Above, the etymology of running is run, and the etymology of apps is app. The English analyzer call returns the etymology of these words using the Stemmer post filter.

④ Test IK participle
http://chenjie.asia:9200/smartcook/_analyze
{
  "analyzer": "ik_smart",  //ik_ Smart word splitter will intelligently segment the text
  "Text": "fried rice with Lao Ganma"    
}
or
{
  "analyzer": "ik_max_word",  //ik_ max_ The word splitter splits the text at the finest granularity
  "Text": "fried rice with Lao Ganma"    
}
⑤ Test Pinyin participle
http://chenjie.asia:9200/smartcook/_analyze
{
  "Analyzer": "pinyin", // Pinyin word splitter will store the Pinyin of each word and the Pinyin initial of the string
  "Text": "fried rice with Lao Ganma"    
}

3.4.2.2 test the word segmentation effect of fields in the specified index

GET http://chenjie.asia:9200/smartcook/_analyze
{
    "field":"name",
    "Text": "fried rice with Lao Ganma"
}

3.5 query command

3.5.1 simple query

Query the row whose author is CJ under the index

http://chenjie.asia:9200/smartcook/_search?q=author:CJ

3.5.2 complex query

  1. Term: query documents with specified keywords in a field without keyword analysis.
  2. Terms: query documents containing multiple keywords in a domain, and the keywords will not be analyzed, and the domain needs to contain all the specified keywords.
  3. Match: the keyword will be analyzed. If there is a match between the keyword segmentation and the word segmentation of the searched field, the match will be successful and the result will be returned.
  4. match_ All: query all documents without conditions
  5. multi_ Match: match multiple fields
  6. match_ Phrase: phrase matching query. When a string is analyzed, the parser not only returns a list of terms, but also returns the position or order information of each term of the original string. Matches documents with the correct word order, and no other words are inserted between these words. The slop parameter tells match_ Phrase queries how far apart entries can be, and still treats documents as matches. How far apart means how many times you need to move an entry to match the query with the document. As shown below, quick fox can match quick brown fox; When slop is 3, fox quick can match quick brown fox

Note: term does not segment query criteria, and keyword does not segment fields. The text type will be parsed by the word breaker, and the keyword will not be parsed by the word breaker.

Field filtering, sorting and paging

"_source": ["menu", "author"], // results are filtered. Only menu and author fields are displayed
"sort": [
        {
                "degree": {
                        "order": "desc"
                }
        }, // the query results are inverted according to the degree
        {
                 "_score":{
                        "order": "desc"
         }
}],
"From": 0, // starting from the first data
"Size": 2 // returns the number of data
"Profile": true, // displays the specific execution process
"Explain": true, // displays the score calculation

Highlight query

//Highlight
"highlight": {
    "Pre_tags": ["< p class ='key 'style ='color: Red' >"], // HTML prefix tags that can be customized
    "Post_tags": ["< / P >"], // HTML suffix tags that can be customized
    "fields": {
        "Menu": {} // the search results will be automatically added with HTML highlighted tags < EM >
    }
}

Exact query (term without word segmentation)

Term is an accurate query, and the search keyword is no longer segmented.

GET http://chenjie.asia:9200/smartcook/_search
{
    "query": {
        "term": {
            "name": "CJ"
        }
    }
}

Query term (query without word segmentation)

GET http://chenjie.asia:9200/smartcook/_search
{
  "query":{
    "term":{
      "menu":{
         "Value": "egg"
      }
    }
  }
}

Query terms (no word segmentation or)

GET http://chenjie.asia:9200/smartcook/_search
{
  "query":{
    "terms":{
      "Menu": ["chicken", "egg] // multi keyword query without word segmentation. The relationship between keywords is or
    }
  }
}

Full query

GET http://chenjie.asia:9200/smartcook/_search
{
    "query":{
        "match_all":{
        }
    },
    "_source": ["menu", "author"], // results are filtered. Only menu and author fields are displayed
    "sort": [
        {"degree": {"order": "desc"}} // query results are inverted according to degree
    ],
    "From": 0, // starting from the first data
    "Size": 2 // returns the number of data
}

Single condition word segmentation query (match)

GET http://chenjie.asia:9200/smartcook/_search
{
    "query": {
        "match": {
            "Menu": "tomato and egg"
        }
    },
    "Profile": true, // displays the specific execution process
    "Explain": true, // displays the score calculation
    //Highlight
    "highlight": {
    "Pre_tags": ["< p class ='key 'style ='color: Red' >"], // HTML prefix tags that can be customized
    "Post_tags": ["< / P >"], // HTML suffix tags that can be customized
    "fields": {
    "Menu": {} // the search results will be automatically added with HTML highlighted tags < EM >
    }
}

Multi condition word segmentation query

GET http://chenjie.asia:9200/smartcook/_search
{
    "query":{
        "bool": {
            "must": [
                {
                    "match": {
                        "Menu": "tomato"
                    }
                },
                {
                    "match": {
                        "author": "CJ"
                    }
                }
            ],
            "should": [],
            "Must_not": [], // it does not affect the correlation score
            "Filter": {// correlation scoring is not affected
                "range":{
                    "degree":{
                        "gt": 1,   // gte、lte、gt、lt
                        "lt": 5
                    }
                }
            }
        }
    }
}

Query match

GET http://chenjie.asia:9200/smartcook/_search
{
  "query":{
    "match":{
      "menu":{
        "Query": "tomato eggs",
        "Operator": "and" // [tomato] and [egg] need to be met at the same time
      }
    }
  }
}

Query match_ Phrase query

GET http://chenjie.asia:9200/smartcook/_search
{
  "query":{
    "match_phrase":{
      "menu":{
        "Query": "West Egg",
        "Slop": 3 // indicates that the relative position of the word in the phrase is allowed to deviate from 3 positions
      }
    }
  }
}

multi_query

{
    "query": {
        "multi_match": {
            "query": "CJ",
            "fields": ["menu","author"]
        }
    } 
}

Query query_ String (single or multi field query)

GET http://chenjie.asia:9200/smartcook/_search
{
  "query":{
    "query_string":{
      "Fields": ["menu", "author"], // the matching fields are menu and author
      "Query": "ZS or (tomato and egg)" // specify a keyword with "ZS" or both "tomato" and "egg"
    }
  }
}

Range query

GET http://chenjie.asia:9200/smartcook/_search
{
  "query":{
    "range":{
      "degree":{
        "gte":3,
        "lte" :4
      }
    }
  }
}

match_ phrase_ Prefix query

(recommended search, the last word will match a large number of documents, which is inefficient)

POST /my_index/my_type/_search
{
    "query": {
        "match_phrase_prefix": {
            "title": {
                "query": "this is r",
                "analyzer": "standard",
                "max_expansions": 10,
                "slop": 2,
                "boost": 100
            }
        }
    }
}

Parameter Description:
The analyzer specifies which parser will segment the phrase
max_ Expansions controls the maximum return result
Boost is used to set the weight of the query
Slop allows term spacing between phrases
Note: Match_ phrase_ Prefix and match_ Phrase is the same, but it has another feature, that is, it allows prefix matching on the last term of the text. If it is a word, such as a, it will match all documents starting with a in the document field. If it is a phrase, such as “this is Ma”, it will match first_ Phrase query to find all documents containing the phrase “this is”, and then find all documents prefixed with “Ma” in these matching documents

Aggregate query

Aggregate the degree field to get the value of each degreeStatistical times。 If you do not want to display the original data, you can add size = 0

GET http://chenjie.asia:9200/smartcook/_search
{
    "Aggs": {// aggregation operations
        "Degree_count": {// name, whatever it is
            "Terms": {// grouping statistics
                "Field": "degree" // grouping fields
            }
        }
    },
    "size": 0
}

Get the of the aggregate fieldaverage value

GET http://chenjie.asia:9200/smartcook/_search
{
    "Aggs": {// aggregation operations
        "Degree_avg": {// name, whatever it is
            "AVG": {// average value
                "Field": "degree" // grouping fields
            }
        }
    },
    "size": 0
}

Get the of the aggregate fieldMaximum

GET http://chenjie.asia:9200/smartcook/_search
{
    "aggs":{
        "degree_max":{
            "max":{
                "field":"degree"
            }
        }
    },
    "size":0
}

Get the of the aggregate fieldminimum value

GET http://chenjie.asia:9200/smartcook/_search
{
    "aggs":{
        "degree_min":{
            "min":{
                "field":"degree"
            }
        }
    },
    "size":0
}

Get the of the aggregate fieldSum

GET http://chenjie.asia:9200/smartcook/_search
{
    "aggs":{
        "degree_sum":{
            "sum":{
                "field":"degree"
            }
        }
    },
    "size":0
}

The total number is obtained after de duplication of the value of a field

GET http://chenjie.asia:9200/smartcook/_search
{
    "aggs":{
        "degree_distinct":{
            "cardinality":{
                "field":"degree"
            }
        }
    },
    "size":0
}

State aggregation, which returns count, Max, min, AVG and sum for a field at one time

GET http://chenjie.asia:9200/smartcook/_search
{
    "aggs":{
        "degree_stats":{
            "stats":{
                "field":"degree"
            }
        }
    },
    "size":0
}

Bucket aggregation query

Bucket aggregation is equivalent to the group by statement in SQL

{
    "aggs":{
        "age_groupby":{
            "terms":{
                "field":"age"
            }
        }
    },
    "size":0
}

Fuzzy query

Returns a document containing words similar to the search term.
Edit distance is the number of character changes required to convert one term to another. These changes can include:

  • Change character (box → Fox)
  • Delete characters (black → lack)
  • Insert character (SiC → sick)
  • Transpose two adjacent characters (act → cat)
    In order to find similar terms, a fuzzy query creates all possible variants or extensions of a set of search terms within a specified editing distance. The query then returns an exact match for each extension.
    Modify and edit the distance through fuzziness. Generally, the default value auto is used to generate the editing distance according to the length of the term.
GET http://chenjie.asia:9200/smartcook/_search
{
  "query": {
    "fuzzy": {
      "author": {
        "value": "CJJ",
        "fuzziness": 2
      }
    }
  } 
}

Value: keyword of query
Boost: the weight of the query. The default value is 1.0
min_ Similarity: set the minimum similarity of matching. The default value is 0.5. For strings, the value is 0-1 (including 0 and 1); For numerical values, the value may be greater than 1; For the date value of 1D, 1M, etc., 1D is equal to 1 day
prefix_ Length: indicates the common prefix length of distinguishing word items. The default is 0

4、 Cluster deployment

4.1 es 5.6.3

Deploy the ES cluster with IK and Pinyin word splitter installed on one node. The steps are as follows:

① Edit dockerfile

vim elasticsearch.dockerfile

FROM elasticsearch:5.6.3-alpine
ENV VERSION=5.6.3
ADD https://github.com/medcl/elasticsearch-analysis-ik/releases/download/v$VERSION/elasticsearch-analysis-ik-$VERSION.zip /tmp/
RUN /usr/share/elasticsearch/bin/elasticsearch-plugin install file:///tmp/elasticsearch-analysis-ik-$VERSION.zip
ADD https://github.com/medcl/elasticsearch-analysis-pinyin/releases/download/v$VERSION/elasticsearch-analysis-pinyin-$VERSION.zip /tmp/
RUN /usr/share/elasticsearch/bin/elasticsearch-plugin install file:///tmp/elasticsearch-analysis-pinyin-$VERSION.zip
RUN rm -rf /tmp/*

② Create mirror es_ ik_ py

docker build -t es_ik_py . -f elasticsearch.dockerfile

③ Create startup container

Node 1:

docker run -d -e ES_JAVA_OPTS="-Xms256m -Xmx256m" --name elasticsearch1 -p 9200:9200 -p 9300:9300 -v /root/es-cluster/es1.yml:/usr/share/elasticsearch/config/elasticsearch.yml  -v /root/es-cluster/data/esdata:/usr/share/elasticsearch/data  es_ik_py01

Node 2:

docker run -d -e ES_JAVA_OPTS="-Xms256m -Xmx256m" --name elasticsearch1 -p 9201:9200 -p 9301:9300 -v /root/es-cluster/es2.yml:/usr/share/elasticsearch/config/elasticsearch.yml  -v /root/es-cluster/data:/usr/share/elasticsearch/data es_ik_py02

Mapped profile
es1. YML is:

cluster.name: elasticsearch-cluster
node.name: node-1
network.host: 0.0.0.0
http.port: 9200
transport.tcp.port: 9300
discovery.zen.ping.unicast.hosts: ["116.62.148.11:9300","116.62.148.11:9301"]
#Enable allow cross domain requests for resources
http.cors.enabled: true
http.cors.allow-origin: "*"

es2. YML is:

cluster.name: elasticsearch-cluster
node.name: node-2
network.host: 0.0.0.0
http.port: 9200
transport.tcp.port: 9300
discovery.zen.ping.unicast.hosts: ["116.62.148.11:9300","116.62.148.11:9301"]
#Enable allow cross domain requests for resources
http.cors.enabled: true
http.cors.allow-origin: "*"

4.2 es 7.8.0

vim elasticsearch.dockerfile

FROM elasticsearch:7.8.0
ENV VERSION=7.8.0
RUN sh -c '/bin/echo -e "y" | /usr/share/elasticsearch/bin/elasticsearch-plugin install https://github.com/medcl/elasticsearch-analysis-ik/releases/download/v${VERSION}/elasticsearch-analysis-ik-$VERSION.zip'
RUN sh -c '/bin/echo -e "y" | /usr/share/elasticsearch/bin/elasticsearch-plugin install https://github.com/medcl/elasticsearch-analysis-pinyin/releases/download/v$VERSION/elasticsearch-analysis-pinyin-$VERSION.zip'

configuration file
es1.yml

#Configuration information of node 1:
#Cluster name and nodes should be consistent
cluster.name: my-elasticsearch
#The node name must be unique in the cluster
node.name: node-1
node.master: true
node.data: true
#IP address allowed to access
network.host: 0.0.0.0
#HTTP port
http.port: 9200
#TCP listening port
transport.tcp.port: 9300
#discovery.seed_hosts: ["chenjie.asia:9300", "chenjie.asia:9301"]
#discovery.zen.fd.ping_timeout: 1m
#discovery.zen.fd.ping_retries: 5
#List of nodes in the cluster that can be selected as the master node
cluster.initial_master_nodes: ["node-1", "node-2"]
#Cross domain configuration
#action.destructive_requires_name: true
#Enable allow cross domain requests for resources
http.cors.enabled: true
http.cors.allow-origin: "*"

#Control the allowable memory size of fielddata, and automatically clean up the old cache when it reaches 20% of heap. No configuration, no recycling.
indices.fielddata.cache.size: 20%
indices.breaker.total.use_real_memory: false
#The fielddata circuit breaker limits the maximum heap size of fileddata. The default setting is 60% of the heap
indices.breaker.fielddata.limit: 40%
#The request circuit breaker estimates the size of the structure that needs to complete other request parts, such as creating an aggregate bucket. The default limit is 40% of the heap memory.
indices.breaker.request.limit: 40%
#Total combines the request and fielddata circuit breakers to ensure that the combination of the two does not use more than 70% of the heap memory (the default).
indices.breaker.total.limit: 95%

es1.yml

#Configuration information of node 1:
#Cluster name and nodes should be consistent
cluster.name: my-elasticsearch
#The node name must be unique in the cluster
node.name: node-2
node.master: true
node.data: true
#IP address allowed to access
network.host: 0.0.0.0
#HTTP port
http.port: 9200
#TCP listening port
transport.tcp.port: 9300
discovery.seed_hosts: ["chenjie.asia:9300", "chenjie.asia:9301"]
discovery.zen.fd.ping_timeout: 1m
discovery.zen.fd.ping_retries: 5
#List of nodes in the cluster that can be selected as the master node
cluster.initial_master_nodes: ["node-1", "node-2"]
#Cross domain configuration
#action.destructive_requires_name: true
#Enable allow cross domain requests for resources
http.cors.enabled: true
http.cors.allow-origin: "*"

#Control the allowable memory size of fielddata, and automatically clean up the old cache when it reaches 20% of heap. No configuration, no recycling.
indices.fielddata.cache.size: 20%
indices.breaker.total.use_real_memory: false
#The fielddata circuit breaker limits the maximum heap size of fileddata. The default setting is 60% of the heap
indices.breaker.fielddata.limit: 40%
#The request circuit breaker estimates the size of the structure that needs to complete other request parts, such as creating an aggregate bucket. The default limit is 40% of the heap memory.
indices.breaker.request.limit: 40%
#Total combines the request and fielddata circuit breakers to ensure that the combination of the two does not use more than 70% of the heap memory (the default).
indices.breaker.total.limit: 95%

Start es container

docker run -d --privileged -e ES_JAVA_OPTS="-Xms256m -Xmx256m" --name es1 -p 9200:9200 -p 9300:9300 -v /root/docker/es/conf/es1.yml:/usr/share/elasticsearch/config/elasticsearch.yml  -v /root/docker/es/data/es1data:/usr/share/elasticsearch/data -v  /root/docker/es/logs:/usr/share/elasticsearch/logs -v /root/docker/es/analysis-ik:/usr/share/elasticsearch/config/analysis-ik es_ik_py:7.8.0
docker run -d --privileged -e ES_JAVA_OPTS="-Xms256m -Xmx256m" --name es2 -p 9201:9200 -p 9301:9300 -v /root/docker/es/conf/es2.yml:/usr/share/elasticsearch/config/elasticsearch.yml  -v /root/docker/es/data/es2data:/usr/share/elasticsearch/data -v  /root/docker/es/logs:/usr/share/elasticsearch/logs -v /root/docker/es/analysis-ik:/usr/share/elasticsearch/config/analysis-ik es_ik_py:7.8.0

If the startup is abnormal, the mounted file may not have write permission. Executechmod -R 777 /root/docker/es/data/es1data /root/docker/es/logs

remarks:

  1. If the container startup error is bootstrap checks failed, you need to modify the memory configuration of the host system file as follows
    ①vi /etc/security/limits.conf, amend as follows:

    * soft nofile 65536
    * hard nofile 131072
    * soft nproc 4096
    * hard nproc 4096
    

    ②vi /etc/security/limits.d/XX-nproc.conf, amend to read:

    * soft nproc 4096
    

    ③vi /etc/sysctl.conf, add the following:

    vm.max_map_count=655360
    

    ④ Validate system files

    sysctl -p
    
  1. The startup container reported an error warning: IPv4 forwarding is disabled Networking will not work. Or the network is blocked
    You need to modify / etc / sysctl conf

    net.ipv4.ip_forward=1
    

    Then restart the network service

    systemctl restart network
    

    see

    sysctl net.ipv4.ip_forward
    

5、 Elasticsearch optimization

5.1 hardware selection

Elasticsearch is based on Lucene. All index and document data are stored on local disk. The specific path can be found in the ES configuration file/ config/elasticsearch. YML is configured as follows:

#----------------------------------- Paths
------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
#path.data: /path/to/data
#
# Path to log files:
#
#path.logs: /path/to/logs
#

Disk is usually a bottleneck on modern servers. Elasticsearch heavy use of disks, the throughput your disks can handle
The larger, the more stable your node will be. Here are some tips for optimizing disk I / O:

  • Use SSD. As mentioned elsewhere, they are much better than mechanical disks.
  • Use RAID 0. Striped raid will increase disk I / O, and the cost is obviously that when one hard disk fails, the whole disk fails. Do not use mirroring or parity raid because the replica already provides this function.
  • In addition, multiple hard disks are used and elasticsearch is allowed to pass through multiple paths The data directory configuration allocates data striping to them.
  • Do not use remotely mounted storage, such as NFS or SMB / CIFS. This introduced delay is completely contrary to performance.

5.2 slicing strategy

5.2.1 reasonably set the number of slices

The design of sharding and replica provides es with the characteristics of supporting distribution and failover, but it does not mean that sharding and replica can be allocated indefinitely. Moreover, after the partition of the index is allocated, we cannot modify the number of partitions again due to the routing mechanism of the index.
Too large or too small number of slices will affect the efficiency of ES work

  • The underlying layer of a partition is a Lucene index, which will consume a certain file handle, memory and CPU operation.
  • Each search request needs to hit each partition in the index. It’s ok if each partition is on a different node, but it’s bad if multiple partitions need to compete on the same node to use the same resources.
  • The term statistics used to calculate the relevance are based on segmentation. If there are many slices, each with little data will lead to low correlation.

How many slices a business index needs to allocate may require architects and technicians to make a pre judgment on the growth of the business, and the horizontal expansion should be carried out in stages. Prepare sufficient resources for the next stage. Only when you enter the next stage will you have time to think about what changes need to be made to reach this stage. Generally speaking, we follow some principles:

  • Control the heap space setting of the maximum JVM in which the hard disk capacity occupied by each partition does not exceed es (generally no more than 32g, refer to the JVM setting principle below). Therefore, if the total capacity of the index is about 500g, the partition size can be about 16; Of course, it is best to consider principle 2 at the same time.
  • Consider the number of nodes. Generally, a node is sometimes a physical machine. If the number of partitions is too large, which greatly exceeds the number of nodes, it is likely to lead to multiple partitions on a node. Once the node fails, even if more than one replica is maintained, it may also lead to data loss and the cluster cannot recover. Therefore, it is generally set that the number of slices does not exceed 3 times the number of nodes.
  • For the maximum number of primary partitions, replicas and nodes, we can refer to the following relationship when allocating:Number of nodes < = number of primary partitions * (number of copies + 1)

5.2.2 postpone slice distribution

For the problem of instantaneous interruption of nodes, by default, the cluster will wait one minute to check whether the node will rejoin. If the node rejoins during this period, the rejoined node will maintain its existing partition data and will not trigger new partition allocation. This can reduce the huge overhead of ES in automatically rebalancing available fragments.
By modifying parametersdelayed_timeout, the rebalancing time can be extended. It can be set globally or modified at the index level:

PUT /_all/_settings 
{
 "settings": {
 "index.unassigned.node_left.delayed_timeout": "5m" 
 } 
}

5.3 routing

When we query a document, how does elastic search know which partition a document should be stored in? It is actually calculated by the following formula:
shard = hash(routing) % number_of_primary_shards
The default value of routing is the ID of the document. You can also use user-defined values, such as user ID.

Query without routing:The whole process is divided into two steps because we don’t know which slice the data to be queried is on:

  • Distribution: after the request reaches the coordination node, the coordination node distributes the query request to each partition.
  • Aggregation: the coordination node collects the query results on each slice, sorts the query results, and then returns the results to the user.

Query with routing:When querying, you can directly locate an allocation query according to the routing information. You do not need to query all the allocations, but sort through the coordination node.

For the user-defined query above, if the routing is set to userid, the data can be queried directly, and the efficiency is greatly improved.

5.4 write speed optimization

The default configuration of ES integrates data reliability, writing speed, real-time search and other factors. In actual use, we need to conduct biased optimization according to the company’s requirements.
For scenarios with low search performance requirements but high write requirements, we need to choose the appropriate write optimization strategy as much as possible. In general, the following aspects can be considered to improve the performance of writing indexes:

  • Increase translog flush to reduce IOPs and writeblock.
  • The index refresh interval is increased to reduce the number of segment merges.
  • Adjust the bulk thread pool and queue.
  • Optimize the task distribution among nodes.
  • The index establishment of Lucene layer is optimized to reduce CPU and Io.

5.4.1 batch data submission

Es provides bulk API to support batch operations. When we have a large number of write tasks, we can use bulk to write in batches.
The general policy is as follows: the volume of data submitted in batch cannot exceed 100m by default. The number of data pieces is generally determined according to the size of the document and the performance of the server, but the data size of a single batch should gradually increase from 5MB to 15MB. When the performance is not improved, take this data amount as the maximum.

5.4.2 optimizing storage devices

ES is an application that uses disks intensively. It will operate disks frequently during segment merging, so it has high requirements for disks. When the disk speed is increased, the overall performance of the cluster will be greatly improved.

5.4.3 reasonable use and consolidation

Lucene stores data in segments. When new data is written to the index, Lucene will automatically create a new segment.
As the amount of data changes, the number of segments will increase, the number of multi file handles and CPU consumed will increase, and the query efficiency will decrease.
Because Lucene segment merging requires a large amount of computation and consumes a lot of I / O, ES adopts a more conservative strategy by default to allow the background to merge segments regularly.

5.4.4 reduce the number of refreshes

When Lucene adds data, it adopts the strategy of delayed writing. By default, the index is refreshed_ The interval is 1 second.
Lucene writes the data to be written to memory first. When it exceeds 1 second (default), refresh will be triggered once, and then refresh will refresh the data in memory to the file cache system of the operating system.
If we don’t have high requirements for the effectiveness of search, we can extend the refresh cycle, such as 30 seconds.
This can also effectively reduce the number of segment refreshes, but it also means that more heap memory needs to be consumed.

5.5.5 increase flush setting

The main purpose of flush is to persist the segments in the file cache system to the hard disk. When the data volume of translog reaches 512MB or 30 minutes, flush will be triggered once.
index. translog. flush_ threshold_ The default value of the size parameter is 512MB, which we can modify.
Increasing the parameter value means that more data may need to be stored in the file cache system, so we need to leave enough space for the file cache system of the operating system.

5.5.6 reduce the number of copies

In order to ensure the availability of the cluster, ES provides replica support. However, each replica will also perform analysis, indexing and possible merging processes. Therefore, the number of replicas will seriously affect the efficiency of writing indexes.
When writing an index, you need to synchronize the written data to the replica node. The more replica nodes, the slower the efficiency of writing an index.
If we need to write in large quantities, we can disable replica replication and set index number_ of_ Replicas: 0 close replica. After the write is complete, replica is modified back to the normal state.

5.5 memory settings

The default memory set after ES installation is 1GB, which is too small for any real business.
If the ES is installed by decompressing, include a JVM in the ES installation file Option file, add the following commands to set the heap size of ES, XMS represents the initial size of the heap, Xmx represents the maximum memory that can be allocated, both of which are 1GB.
Ensure that the sizes of Xmx and XMS are the same. The purpose is to waste resources without re separating and calculating the size of the heap area after the Java garbage collection mechanism has cleaned up the heap area, so as to reduce the pressure caused by scaling the heap size.
Suppose you have a 64g memory machine. According to normal thinking, you may think it’s better to give 64g memory to es, but is this the reality? The bigger the better? Although memory is very important for ES, the answer is no!

Because the allocation of ES heap memory needs to meet the following two principles:

  • Do not exceed 50% of the physical memory: Lucene is designed to cache the data in the underlying OS into memory.
    Lucene’s segments are stored in a single file. These files will not change, so it is convenient for caching. At the same time, the operating system will cache these segment files for faster access.
    If the heap memory we set is too large, Lucene’s available memory will be reduced, which will seriously affect Lucene’s full-text query performance.

  • The size of heap memory should not exceed 32GB: in Java, all objects are allocated on the heap, and then a Klass pointer points to its class metadata.
    This pointer is 64 bits on a 64 bit operating system, which can use more memory (2 ^ 64). On a 32-bit system, it is 32 bits, and the maximum addressing space of a 32-bit operating system is 4GB (2 ^ 32).
    But a 64 bit pointer means a greater waste because your pointer itself is large. Not to mention wasting memory. What’s worse, larger pointers will occupy more bandwidth when moving data between main memory and buffer (such as LLC, L1, etc.).
    Eventually we will all use the 31 g setting
    -Xms 31g
    -Xmx 31g
    Suppose you have a machine with 128 GB of memory, you can create two nodes, and the memory allocation of each node is no more than 32 GB. That is, no more than 64 GB of memory is given to es heap memory, and the remaining more than 64 GB of memory is given to Lucene.

5.6 important configuration

Parameter name Parameter value explain
cluster.name elasticsearch Configure the cluster name of ES. The default value is es. It is recommended to change it to a name related to the stored data. Es will automatically find nodes with the same cluster name under the same network segment.
node.name node-1 Node names in a cluster cannot be duplicated in the same cluster. Once the node name is set, it cannot be changed. Of course, it can also be set to the host name of the server, such as node name:${HOSTNAME}。
node.master true Specifies whether the node is eligible to be elected as a master node. The default is true. If it is set to true, it is only eligible to become a master node. Whether it can become a master node needs to be elected.
node.data true Specifies whether the node stores index data. The default is true. Data addition, deletion, modification and query are all completed in the data node.
index.number_of_shards 1 Set the number of index slices. The default is 1 slice. You can also set this value when creating an index. The specific setting depends on the size of the data volume. If the amount of data is small, the efficiency is the highest when it is set to 1
index.number_of_replicas 1 Set the default number of index copies, which is 1 by default. The more replicas, the better the availability of the cluster, but the more data needs to be synchronized when writing the index.
transport.tcp.compress true Set whether to compress data when transmitting data between nodes. The default value is false and not compressed
discovery.zen.minimum_master_nodes 1 Set the minimum number of candidate master nodes to participate in the election of master nodes. The default is 1. If the default value is used, brain fissure may occur when the network is unstable< Br / > the reasonable value is (master_eligible_nodes / 2) + 1, where Master_ eligible_ Nodes indicates the number of candidate primary nodes in the cluster
discovery.zen.ping.timeout 3s Set the timeout of Ping connection when automatically discovering other nodes in the cluster. The default is 3 seconds< Br / > in a poor network environment, it needs to be set larger to prevent partition transfer due to misjudging the survival state of the node.

6、 Es exception record

5.1 circuit_ breaking_ Exception exception

① Query times exception

    "error": {
        "root_cause": [
            {
                "type": "circuit_breaking_exception",
                "reason": "[parent] Data too large, data for [<http_request>] would be [257665614/245.7mb], which is larger than the limit of [255013683/243.1mb], real usage: [257665384/245.7mb], new bytes reserved: [230/230b], usages [request=0/0b, fielddata=0/0b, in_flight_requests=230/230b, accounting=3516/3.4kb]",
                "bytes_wanted": 257665614,
                "bytes_limit": 255013683,
                "durability": "PERMANENT"
            }
        ],
        "type": "circuit_breaking_exception",
        "reason": "[parent] Data too large, data for [<http_request>] would be [257665614/245.7mb], which is larger than the limit of [255013683/243.1mb], real usage: [257665384/245.7mb], new bytes reserved: [230/230b], usages [request=0/0b, fielddata=0/0b, in_flight_requests=230/230b, accounting=3516/3.4kb]",
        "bytes_wanted": 257665614,
        "bytes_limit": 255013683,
        "durability": "PERMANENT"
    },
    "status": 429
}

reason:
Es protection measures to prevent the memory capacity used by the cache from exceeding the limit. The default cache setting of ES allows the cache to be only caused by no entry or exit;

Es caches index data in memory (JVM) when querying:
Expulsion line and circuit breaker. When the cached data reaches the expulsion line, it will automatically expel some data to keep the cache within a safe range. When the user is ready to perform a query operation, the circuit breaker works. When the cached data + the amount of data to be cached for the current query reaches the limit of the circuit breaker, a data too large error will be returned to prevent the user from performing the query operation. Es divides the cached data into two categories, fielddata and other data. Let’s look at fielddata in detail. It is the “culprit” of our exception this time.
Fielddata mentioned in ES configuration refers to field data. When sorting (sort) and Statistics (aggs), ES reads all the field data involved into memory (JVM heap) for operation. It is equivalent to data caching to improve query efficiency.

solve:

#Control the allowable memory size of fielddata, and automatically clean up the old cache when it reaches 20% of heap. No configuration, no recycling.
indices.fielddata.cache.size: 20%
indices.breaker.total.use_real_memory: false
#The fielddata circuit breaker limits the maximum heap size of fileddata. The default setting is 60% of the heap
indices.breaker.fielddata.limit: 40%
#The request circuit breaker estimates the size of the structure that needs to complete other request parts, such as creating an aggregate bucket. The default limit is 40% of the heap memory.
indices.breaker.request.limit: 40%
#Total combines the request and fielddata circuit breakers to ensure that the combination of the two does not use more than 70% of the heap memory (the default).
indices.breaker.total.limit: 95%

7、 Interview questions

7.1 why use elasticsearch?

With the development of business and the passage of time, there will be a lot of data in the system. In business, fuzzy query is often used for data search, and fuzzy query will cause the query engine to give up the index, resulting in full table scanning when the system queries the data. In a million level database, the query efficiency is very low, and we use es to do a full-text index, We can improve the query speed by putting some fields of frequently queried system functions, such as commodity name, description, price and ID in the commodity table of e-commerce system into the ES index database.

7.2 what is the master election process of elasticsearch?

  • The selection of elasticsearch is the responsibility of zendiscovery module, which mainly includes Ping (nodes discover each other through this RPC) and unicast (unicast module contains a host list to control which nodes need Ping).
  • All nodes that can become a master (node. Master: true) are sorted according to the nodeid dictionary. Each time, each node is elected, the nodes they know are ranked in order, and then the first (bit 0) node is selected. For the time being, it is considered a master node.
  • If the number of votes on a node reaches a certain value (it can become the number of master nodes n / 2 + 1) and the node elects itself, the node is the master. Otherwise, re-election will continue until the above conditions are met.
  • The responsibilities of the master node mainly include the management of clusters, nodes and indexes, and are not responsible for document level management; The data node can turn off the HTTP function.

7.3 elasticsearch cluster brain crack problem?

Possible causes of “brain crack”:

  • Network problem: the network delay between clusters causes some nodes to lose access to the master. They think the master is dead, so they elect a new master, mark the partitions and replicas on the master in red, and assign a new master partition.
  • Node load: the role of the master node is both master and data. When the traffic is large, the es may stop responding and cause a large area delay. At this time, other nodes cannot get the response from the master node. It is considered that the master node is dead and the master node will be re selected.
  • Memory recycling: the ES process on the data node occupies a large amount of memory, causing the JVM to reclaim large-scale memory, causing the ES process to lose response.

Solution to brain crack problem:

  • Reduce misjudgment: discovery zen. ping_ The timeout node status response time is 3S by default and can be increased appropriately. If the master does not respond within the response time range, it is judged that the node has hung up. Increasing the parameters (such as 6S, discovery. Zen. Ping_timeout: 6) can appropriately reduce misjudgment.
  • Election trigger: discovery zen. minimum_ master_ nodes:1
    This parameter is used to control the minimum number of cluster master nodes for election behavior. When the number of alternative master nodes is greater than or equal to the value of this parameter, and there are nodes of this parameter in the alternative master nodes, it is considered that the master node is hung, and the election is carried out. The official recommendation is (n / 2) + 1, n is the number of primary nodes (that is, the number of nodes qualified to become primary nodes)
  • Role separation: that is, the master node is separated from the data node, limiting roles
    The primary node is configured as node master: true node. data: false
    The slave node is configured as: node master: false node. data: true

7.4 elasticsearch index document process?

Elasticsearch Foundation

image.png
  • By default, the coordination node uses the document ID to participate in the calculation (also supports routing), so as to provide appropriate fragments for routing:
    shard = hash(document_id) % (num_of_primary_shards)
  • When the node where the partition is located receives the request from the coordination node, it will write the request to the memory buffer, and then write it to the filesystem cache regularly (every 1 second by default). This process from the memory buffer to the filesystem cache is called refresh;
  • Of course, in some cases, the data with momery buffer and filesystem cache may be lost. Es ensures the reliability of the data through the translog mechanism. The implementation mechanism is that after receiving the request, it will also be written to the translog. When the data in the filesystem cache is written to the disk, it will be cleared. This process is called flush;
  • In the flush process, the buffer in memory will be cleared and the content will be written to a new segment. The fsync of the segment will create a new commit point and refresh the content to disk. The old translog will be deleted and a new translog will be started.
  • The timing of flush trigger is timed trigger (30 minutes by default) or when the translog becomes too large (512M by default);

7.5 how does elasticsearch update and delete documents?

  • Deletion and update are also write operations, but the documents in elasticsearch are immutable, so they cannot be deleted or changed to show their changes;
  • Each segment on the disk has a corresponding Del file. When the delete request is sent, the document is not really deleted, but in Del file is marked for deletion. The document can still match the query, but it will be filtered out in the results. When segments are merged, in Documents marked for deletion in the Del file will not be written to the new segment.
  • When a new document is created, elasticsearch will specify a version number for the document. When the update is performed, the old version of the document will be updated The Del file is marked for deletion, and the new version of the document is indexed to a new segment. The old version of the document can still match the query, but it will be filtered out in the results.

7.6 elasticsearch search process?

Elasticsearch Foundation

image.png
  • Search is performed as a two-stage process, which we call query then fetch;
  • During the initial query phase, the query is broadcast to each shard copy (primary shard or replica shard) in the index. Each fragment performs a search locally and builds a priority queue with the size of matching documents as from + size. PS: the file system cache will be queried during the search, but some data is still in the memory buffer, so the search is near real-time.
  • Each fragment returns the IDS and sorting values of all documents in its own priority queue to the coordination node. It combines these values into its own priority queue to produce a global sorted result list.
  • The next step is the retrieval phase. The coordination node identifies which documents need to be retrieved and submits multiple get requests to the relevant fragments. Each fragment loads and enriches the document, and then returns the document to the coordination node if necessary. Once all the documents are retrieved, the coordination node returns the results to the client.
  • The search type of query then fetch refers to the data of this segment when scoring the relevance of documents, which may not be accurate when the number of documents is small. DFS query then fetch adds a pre query processing to query term and document frequency. This score is more accurate, but the performance will deteriorate.

7.7 when elasticsearch is deployed, what are the optimization methods for Linux settings?

  • 64 GB memory machines are ideal, but 32 GB and 16 GB machines are also common. Less than 8 GB is counterproductive.
  • If you want to choose between faster CPUs and more cores, it’s better to choose more cores. The extra concurrency provided by multiple cores is far better than a slightly faster clock frequency.
  • If you can afford SSD, it will go far beyond any rotating media. SSD based nodes have improved query and index performance. SSD is a good choice if you can afford it.
  • Even if data centers are close at hand, avoid clustering across multiple data centers. It is absolutely necessary to avoid clusters spanning large geographical distances.
  • Make sure that the JVM running your application is exactly the same as the JVM of the server. In several places in elasticsearch, Java’s native serialization is used.
  • By setting the gateway recover_ after_ nodes、gateway. expected_ nodes、gateway. recover_ after_ Time can avoid too much fragmentation switching when the cluster is restarted, which may shorten the data recovery from hours to seconds.
  • Elasticsearch is configured to use unicast discovery by default to prevent nodes from inadvertently joining the cluster. Only nodes running on the same machine will automatically form a cluster. It is better to use unicast instead of multicast.
  • Do not arbitrarily modify the size of the garbage collector (CMS) and each thread pool.
  • Give Lucene (less than) half of your memory (but no more than 32 GB!), Through es_ HEAP_ The size environment variable is set.
  • Memory swapping to disk is fatal to server performance. If memory is swapped to disk, a 100 microsecond operation may become 10 milliseconds. Think again that so many 10 microseconds of operation delay add up. It’s not hard to see how terrible switching is for performance.
  • Lucene uses a lot of files. At the same time, elasticsearch uses a large number of sockets to communicate between nodes and HTTP clients. All this requires sufficient file descriptors. You should increase your file descriptor and set a large value, such as 64000.

Supplement: performance improvement method in index stage

  • Use batch requests and resize them: 5 – 15 MB of data per batch is a good starting point.
  • Storage: using SSDs
  • Segment and merge: elasticsearch defaults to 20 MB / s, which should be a good setting for mechanical disks. If you use SSD, you can consider increasing it to 100 – 200 MB / s. If you are doing batch import and don’t care about the search, you can completely turn off the merge current limit. In addition, you can add index translog. flush_ threshold_ Set the size from the default 512 MB to a larger value, such as 1 GB, which can accumulate larger segments in the transaction log when an empty trigger is triggered.
  • If your search results don’t need near real-time accuracy, consider putting the index of each index refresh_ The interval is changed to 30s.
  • If you are doing mass import, consider setting index number_ of_ Replicas: 0 close replica.

7.8 in terms of GC, what should I pay attention to when using elasticsearch?

  • The index of inverted dictionary needs resident memory and cannot be GC. Segment memory on data node needs to be monitored;
  • All kinds of caches, such as field cache, filter cache, indexing cache, bulk queue, etc., should be set to a reasonable size, and whether the heap is sufficient according to the worst case, that is, when all kinds of caches are full, is there any heap space that can be allocated to other tasks? Avoid using “self deception” such as clear cache to free memory.
  • Avoid searches and aggregations that return a large number of result sets. Scenarios that really need to pull a large amount of data can be implemented using the scan & scroll API.
  • Cluster stats resident memory cannot be expanded horizontally. Super large clusters can be divided into multiple clusters and connected through triple nodes.
  • If you want to know whether heap is enough or not, you must combine with the actual application scenario and continuously monitor the heap usage of the cluster.

7.9 how does elasticsearch realize the aggregation of large amounts of data (hundreds of millions of magnitude)?

The first approximate aggregation provided by elastic search is the cardinality measure. It provides the cardinality of a field, that is, the number of distinct or unique values of the field. It is based on HLL algorithm. HLL will hash our input first, and then estimate the probability according to the bits in the hash result to obtain the cardinality. It is characterized by configurable accuracy, which is used to control the use of memory (more accuracy = more memory); The accuracy of small data sets is very high; We can configure parameters to set the fixed memory usage required for de duplication. Whether thousands or billions of unique values, memory usage is only related to the accuracy of your configuration.

7.10 in the case of concurrency, if elasticsearch ensures consistent reading and writing?

  • Optimistic concurrency control can be used through the version number to ensure that the new version will not be overwritten by the old version, and the application layer will handle specific conflicts;
  • In addition, for write operations, the consistency level supports quorum / one / all, and the default is quorum, that is, write operations are allowed only when most shards are available. However, even if most of them are available, there may be failure to write to the replica due to network and other reasons. In this way, the replica is considered to be faulty, and the fragment will be rebuilt on a different node.
  • For read operations, you can set replication to sync (default), which makes the operation return only after the primary partition and replica partition are completed; If replication is set to async, you can also set the search request parameters_ The preference is the primary node to query the primary partition and ensure that the document is the latest version.

7.11 how to monitor elasticsearch cluster status?

Elasticsearch head plug-in.
Elasticsearch is monitored through kibana. You can view the health status and performance of your cluster in real time, and analyze the past cluster, index and node indicators.

7.12 do you know the dictionary tree?

Common dictionary data structures are as follows:

data structure Advantages and disadvantages
Sort listarray / list Use dichotomy to find the imbalance.
HashMap/TreeMap High performance, large memory consumption, almost three times the original data.
Skip List Jump table, which can quickly find words, is implemented in Lucene, redis, HBase, etc. Compared with treemap and other structures, it is especially suitable for high concurrency scenarios.
Trie It is suitable for English dictionaries. If there are a large number of strings in the system and these strings basically have no public prefix, the corresponding trie tree will consume a lot of memory.
Double Array Trie It is suitable for making Chinese dictionaries with small memory consumption. Many word segmentation tools use this algorithm (deep double array trie).
Ternary Search Tree Trigeminal tree, each node has three nodes, which has the advantages of space saving and fast query.
Finite State Transducers(FST) A finite state transition machine, Lucene 4 has an open source implementation and is widely used.

Dictionary tree, also known as word lookup tree, trie tree, is a tree structure and a variant of hash tree. The typical application is to count, sort and save a large number of strings (but not limited to strings), so it is often used by search engine system for text word frequency statistics. Its advantages are: using the common prefix of string to reduce the query time, minimize unnecessary string comparison, and the query efficiency is higher than that of hash tree.

The core idea of trie is to exchange space for time, and use the public prefix of string to reduce the overhead of query time, so as to improve efficiency. It has three basic properties:

  • The root node does not contain characters. Except for the root node, each node contains only one character.
  • From the root node to a node, the characters passing through the path are connected to the corresponding string of the node.
  • All child nodes of each node contain different characters.

For the Chinese dictionary tree, the child nodes of each node are stored in a hash table, so there is no waste of space, and the hash complexity O (1) can be preserved in the query speed.

7.13 what are the cluster, node, index, document and type in elasticsearch?

  • Cluster: a cluster is a collection of one or more nodes (servers) that jointly store your entire data and provide federated access across all nodes
    Indexing and searching functions. The cluster is identified by a unique name, which is “elasticsearch” by default. This name is important because if the node is set to
    If it is set to join the cluster by name, the node can only be part of the cluster.
  • Node: a node is a single server that is part of a cluster. It stores data and participates in cluster indexing and search functions.
  • Index: an index is like a “database” in a relational database. It has a mapping that defines multiple types. An index is a logical namespace that maps to one or more primary shards and can have zero or more replica Shards. MySQL = > database elasticsearch = > index
  • Document class: a document is similar to a row in a relational database. The difference is that each document in the index can have a different structure (field), but
    Yes, common fields should have the same data type. MySQL => Databases => Tables => Columns / Rows
    Elasticsearch = > indexes = > types = > documents with attributes
  • Type: type is the logical category / partition of index, and its semantics completely depends on the user.

7.14 what is the inverted index in elasticsearch?

Inverted index is the core of search engine. The main goal of search engine is to provide fast search when looking for documents with search conditions. The inverted index in ES is actually the inverted index of Lucene, which is different from the traditional forward index. When storing data, the inverted index associates keywords with data and saves them in the inverted table. Then, during query, the query content is segmented and queried in the inverted table, and finally the data can be matched.

8、 References

Official documents https://www.elastic.co/guide/cn/elasticsearch/guide/current/administration.html

https://www.bilibili.com/video/BV1hh411D7sb

Recommended Today

Lagrange interpolation — Python

Data analysis Data cleaning: missing value processing, 1 deleting records2 data interpolation3 no treatment Data inhttps://book.tipdm.org/jc/219Data and code in resource package in Chapter4 \ demo \ data \ cataling_ sale. xls Common interpolation methods Interpolation Lagrange interpolation According to mathematical knowledge, for n known points on the plane (no two points), n-1 polynomial can be […]