Elasticsearch database | elasticsearch-7.5.0 application construction


Elasticsearch is a distributed and restful search and data analysis engine. ——Elastic stack official website


Building elasticsearch’s “things”“


One day, on the deck of the golden Melly, Weibo told Saab that it was necessary to use elasticsearch database in the following project development process, which was mainly used for real-time query of data collected by various devices on the golden Melly, and maybe later used in the construction and application of distributed log system, so as to let Saab do a preliminary research on technology. As a result, a large number of data were searched in Saab, and it was found that the information on the Internet was almost the same, or most of them did not have a complete process. Even, it is found that for the practical application of elasticsearch database, most pirates are only limited to elk and other situations. For example, when most pirates mention redis, they just think that this device can only be used in the cache level, but in fact, the function of others is so powerful that it exceeds the imagination of the pirates. Even Saab found a free trial resource of elasticsearch database in an Alibaba site. But for the free lunch, maybe after charging for that day, it can only use a “wave” to commemorate the evil of gold. In addition to the traditional deployment of elasticsearch database, there is a docker, which can not only run elasticsearch database stably, but also save a lot of server memory. So Saab began to explore the way to build elastic search on docker

What happened when docker deployed elasticsearch“

1. Foundation prerequisites

One VM host of Alibaba cloud or virtual machine, and the docker container is installed on its own:

Ps[ ⚠️ Precautions]:
For docker installation, please refer to:Build and deploy docker container based on centos7 in Linux Environment

2. Maximum number of open files

[2.1] parameter optimization: ulimit – a | grep open

[[email protected] ~]# ulimit -a |grep open
open files   (-n) 65535
[[email protected] ~]#

[2.2] modify the configuration file / etc / security/ limits.conf

#Add the following two lines at the bottom
* soft nofile 65536
* hard nofile 65536
* soft nproc 2048
* hard nproc 4096

[2.3] confirm whether to modify: ulimit – a | grep open

[[email protected] ~]# ulimit -a |grep open
open files   (-n) 65535
[[email protected] ~]#

3. Adjust the number of system processes

Modification method: VIM / etc / security / limits.d/20- nproc.conf

#Adjust to the following configuration
*          soft    nproc     4096
root       soft    nproc     unlimited

4. Adjust the virtual memory and the maximum number of concurrent connections

Virtual memory – > vm.max_ map_ count
Elasticsearch uses the hybrid mmapfs / niofs directory by default to store indexes. The default operating system has too low a limit on the MMAP count, which may cause an out of memory exception

[1] Temporary solution:

sysctl –w vm.max_map_count=262144
sysctl –w vm.max_map_count=262144

If it is not optimized, the following prompt will appear when elasticsearch is started:

max virtual memory areas vm.max_map_count [65535] is too low, increase to at least [262144]

[2] Permanent effective scheme: update system file / etc/ sysctl.conf Of vm.max_ map_ Count field
Execute command: VIM / etc/ sysctl.conf

[[email protected] ~]# vim /etc/sysctl.conf
[[email protected] ~]#
#Settings Elasticsearch Virtual Memory—>655360

Load to system: sysctl – P — load / etc/ sysctl.conf

[[email protected] elasticsearch]# sysctl -p --load /etc/sysctl.conf
vm.swappiness = 0
net.ipv4.neigh.default.gc_stale_time = 120
net.ipv4.conf.all.rp_filter = 0
net.ipv4.conf.default.rp_filter = 0
net.ipv4.conf.default.arp_announce = 2
net.ipv4.conf.lo.arp_announce = 2
net.ipv4.conf.all.arp_announce = 2
net.ipv4.tcp_max_tw_buckets = 5000
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_syn_backlog = 1024
net.ipv4.tcp_synack_retries = 2
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1
kernel.sysrq = 1
vm.max_map_count = 655360
fs.file-max = 655360
vm.overcommit_memory = 1
[[email protected] elasticsearch]#

5. Turn off elasticsearch self detection

stay elasticsearch.yml Add configuration item in:

bootstrap.memory_lock: false
bootstrap.system_call_filter: false

Ps[ A kind of Note:
It is mainly to avoid the following problems:
[1] Max file descriptors [1024] for elasticsearch process is too low, increase to at least [65536]
[2] Adjust virtual memory and maximum number of concurrent connections: Max virtual memory areas vm.max_ map_ count [65530] is too low, increase to at least [262144]
[3] Turn off elasticsearch self detection: system call filters failed to install; check the logs and fix your configuration or disable system call filters at your own risk
[4] Adjust the number of system processes: max number of threads [1024] for user [lish] like too low, increase to at least [2048]

Actual deployment of elasticsearch service

Basic steps for docker to deploy application services: Search Image > pull Image > run

1. Query elasticsearch image:

docker search elasticsearch


PS [precautions]:

  1. Generally, image resources are pulled from the official docker warehouse[docker-hub]Pull or build your own docker cloud warehousealiyun-dockeretc.
  2. The elk images selected in this tutorial are all based on the official elk docker warehouseelastic-io

2. Pull elasticsearch image:

docker pull docker.elastic.co/elasticsearch/elasticsearch:7.3.1


PS [precautions]:
1. This tutorial adopts version 7.3. X, and the latest version is 7.4. X [mainly built in Alibaba cloud with version 7.3. X, to avoid pit entry PROBLEMS]
2. The [net / http: TLS handshake timeout] problem may occur during the pull process. Try it several times more, mainly due to the limitation of network bandwidth

3. Modify the image name:

docker tag docker.elastic.co/elasticsearch/elasticsearch:7.3.1  elasticsearch:latest


PS [precautions]:
1. The name is too long, which makes it inconvenient to view. Through docker tag source image [source image] target image, [target image] is recommended- image:target-version ]Format definition, and does not take up space, equivalent to renaming the image
2. For pull kibana[ docker.elastic.co/kibana/kibana : 7.3.1] and logstash[ docker.elastic.co/logstash/logstash It is suggested to modify.

4. Deploy image service:
Deployment command:

docker run -itd -p 9200:9200 -p 9300:9300 --restart=always --privileged=true --name elasticsearch-server -e "discovery.type=single-node" -e ES_JAVA_OPTS="-Xms=512m -Xms=512m" elasticsearch:latest

Default configuration directory:


To view a list of containers:

docker ps --format "table {{.ID}}\t{{.Names}}\t{{.Ports}}"
![docker ps]

PS [precautions]:

1. Open ports [9200 and 9300] – > 9200 are used as HTTP protocol, mainly used for external communication, 9300 as TCP protocol, communication between jars is through TCP protocol, and cluster deployment is usually through 9300. Recommended [host custom port: 9200]
2. — restart = always: configure the container restart policy. When the host machine restarts, it is not necessary to start it manually because it is configured to start automatically
3. — privileged: configure container operation authority [true root operation authority, false current container user operation permission]
4. For the deployment network mode, the default bridging mode is recommended, and the host mode can also be customized

5. Modify the configuration:
Enter container: docker exec – it container ID [container ID] or container name [container name] / bin / Bash
For example: docker exec – it f2d2e97da375 / bin / bash ᦇ f2d2e97da375 – > container ID

Modify the configuration file:

[[email protected] elasticsearch]# ls 
LICENSE.txt  NOTICE.txt  README.textile  bin  config  data  jdk  lib  logs  modules  plugins
[[email protected] elasticsearch]# 
[[email protected] elasticsearch]# cd config  
[[email protected] config]# ls
elasticsearch.keystore  elasticsearch.yml  jvm.options  log4j2.properties  role_mapping.yml  roles.yml  users  users_roles
[[email protected] config]# vi elasticsearch.yml

Add cross domain configuration:http.cors.enabled: true && http.cors.allow-origin: "*"

cluster.name: "docker-cluster"
http.cors.enabled: true
http.cors.allow-origin: "*"

Then exit the exit container and restart the container on the host: docker restart container ID [container ID] or container name [container name]
docker restart f2d2e97da375

[[email protected] config]# exit
[[email protected] ~]# docker restart f2d2e97da375
[[email protected] ~]#

PS [precautions]:
1. Enter container mode: including using docker attach command or docker exec command,
The docker exec command is recommended. reason:

  • Docker attach: using exit to exit the container will cause the container to stop
  • Docker exec: using exit to exit the container does not cause the container to stop
  • Refer to several methods for docker to enter the container-Several methods of docker entering container
    2. If docker has installed the visual interface portainer, it is recommended to enter the container in this way

Build and deploy elasticsearch head service

Elasticsearch head: the web front-end interface of elastic search cluster, built with nodjs, is mainly used to view elasticsearch related information

1. Pull elasticsearch head image: docker pull mobz / elasticsearch- head:5

[[email protected] ~]# docker pull mobz/elasticsearch-head:5
5: Pulling from mobz/elasticsearch-head
75a822cd7888: Pull complete 
57de64c72267: Pull complete 
4306be1e8943: Pull complete 
871436ab7225: Pull complete 
0110c26a367a: Pull complete 
1f04fe713f1b: Pull complete 
723bac39028e: Pull complete 
7d8cb47f1c60: Pull complete 
7328dcf65c42: Pull complete 
b451f2ccfb9a: Pull complete 
304d5c28a4cf: Pull complete 
4cf804850db1: Pull complete 
Digest: sha256:55a3c82dd4ba776e304b09308411edd85de0dc9719f9d97a2f33baa320223f34
Status: Downloaded newer image for mobz/elasticsearch-head:5
[[email protected] ~]#

2. Modify elasticsearch head image name: docker tag mobz / elasticsearch- head:5    elasticsearch- head:latest

[[email protected] ~]# docker tag  mobz/elasticsearch-head:5        elasticsearch-head:latest
[[email protected] ~]# docker images
REPOSITORY                                      TAG                 IMAGE ID            CREATED             SIZE
grafana/grafana                                 latest              05d1bcf30d16        7 days ago          207MB
nginx                                           latest              540a289bab6c        3 weeks ago         126MB
prom/prometheus                                 latest              2c8e464e47f4        3 weeks ago         129MB
moxm/sentinel-dashboard                         latest              0ccaac81584e        4 weeks ago         167MB
portainer                                       latest              4cda95efb0e4        4 weeks ago         80.6MB
portainer/portainer                             latest              4cda95efb0e4        4 weeks ago         80.6MB
apache/skywalking-ui                            latest              fa66ca9c9862        2 months ago        123MB
apache/skywalking-oap-server                    latest              376a37cdf65c        2 months ago        190MB
docker.elastic.co/kibana/kibana                 7.3.1               b54865ba6b0b        2 months ago        1.01GB
docker.elastic.co/elasticsearch/elasticsearch   7.3.1               3d3aa92f641f        2 months ago        807MB
elasticsearch                                   latest              3d3aa92f641f        2 months ago        807MB
prom/node-exporter                              latest              e5a616e4b9cf        5 months ago        22.9MB
google/cadvisor                                 latest              eb1210707573        12 months ago       69.6MB
elasticsearch-head                              latest              b19a5c98e43b        2 years ago         824MB
mobz/elasticsearch-head                         5                   b19a5c98e43b        2 years ago         824MB
tutum/influxdb                                  latest              c061e5808198        3 years ago         290MB
[[email protected] ~]#

3. Deploy elasticsearch head container

docker run -itd --restart=always --privileged=true -p 9100:9100 --name elasticsearch-head-server elasticsearch-head:latest

To view container services:

docker ps --format "table {{.ID}}\t{{.Names}}\t{{.Ports}}"


4. Browser access:http://remote-ip:9100/

Actual test elasticsearch head service

1. Prepare data:

    "companyId": "ebcb4f99e0cb4ad781278ae636c1031f",
    "Classifyname": hormone detection,
    "Devicetype": hormone detection,
    "data": {
    	"deviceId": "20000060100000002",
        "deviceNo": "QC00020000060100000002",
        "appid": "00000000",
        "sim": "89860404191792655118",
        "csq": "30",
        "electric": "98",
        "voltage": "13279",
        "softVer": "5143000500010601-01020006|5143000500010601-01010003",
        "hardVer": "5143000500010601-00000002|5143000500010601-00000002",
        "status": "0",
        "date": "2020-09-13 11:23:52",
        "elements": [
                "key": "20",
                "value": "10",
                "time": "2020-09-13 11:23:52"
                "key": "21",
                "value": "11",
                "time": "2020-09-13 11:23:52"
                "key": "22",
                "value": "12",
                "time": "2020-09-13 11:23:52"
                "key": "23",
                "value": "13",
                "time": "2020-09-13 11:23:52"

2. Set request header: application / JSON; charset = UTF-8

3. Send put request:http://remote-ip/device/deviceData/QC00020000060100000001

    "_index": "device",
    "_type": "deviceData",
    "_id": "QC00020000060100000002",
    "_version": 1,
    "result": "created",
    "_shards": {
        "total": 2,
        "successful": 1,
        "failed": 0
    "created": true

4. Open the browser to view the results:

Summary of building elasticsearch

1. Deployment basic command:

docker run -itd -p 9200:9200 -p 9300:9300 --restart=always --privileged=true --name elasticsearch-server --network-alias elasticsearch-server --hostname elasticsearch-server -v /docker/elasticsearch/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml -v /docker/elasticsearch/data:/usr/share/elasticsearch/data -v /docker/elasticsearch/logs:/usr/share/elasticsearch/logs  -e "discovery.type=single-node" -e ES_JAVA_OPTS="-server -Xms512m -Xmx512m -Xmn256m -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=320m -XX:+AlwaysPreTouch -XX:-UseBiasedLocking " -e TZ="Asia/Shanghai"  elasticsearch:latest

2. Configuration file[ elasticsearch.yml ]:

cluster.name: elasticsearch-cluster
node.name: elasticsearch-server
http.cors.enabled: true
http.cors.allow-origin: "*"
bootstrap.memory_lock: false
bootstrap.system_call_filter: false
transport.tcp.port: 9300
transport.tcp.compress: true
http.max_content_length: 128mb

3. Installation of word segmentation plug-in:

./bin/elasticsearch-plugin install https://github.com/medcl/elasticsearch-analysis-ik/releases/download/v7.5.0/elasticsearch-analysis-ik-7.5.0.zip

4. Docker deployment elasticsearch head:

docker run -itd --restart=always --privileged=true -p 9100:9100 --network-alias elasticsearch-head-server --name elasticsearch-head-server --hostname elasticsearch-head-server -v /docker/elasticsearch-head/app:/usr/src/app elasticsearch-head:latest

5. Elasticsearch head 406 error:
Ajax settings part of the JSON configuration:

ajaxSettings: {
      url: ajaxLocation,
      isLocal: rlocalProtocol.test(ajaxLocParts[1]),
      global: true,
      type: "GET",
      //contentType: "application/x-www-form-urlencoded",
      contentType: "application/json;charset=UTF-8",
      processData: true,
      async: true,
		timeout: 0,
		data: null,
		dataType: null,
		username: null,
		password: null,
		cache: null,
		traditional: false,
		headers: {},

      accepts: {
        xml: "application/xml, text/xml",
        html: "text/html",
        text: "text/plain",
        json: "application/json, text/javascript",
        "*": "*/*",

JSON configuration of inspectdata part:

// Detect, normalize options and install callbacks for jsonp requests
  jQuery.ajaxPrefilter("json jsonp", function (s, originalSettings, jqXHR) {
    //var inspectData = s.contentType === "application/x-www-form-urlencoded" &&
    var inspectData =
      (s.contentType === "application/x-www-form-urlencoded" &&
        typeof s.data === "string") ||
      (s.contentType === "application/json;charset=UTF-8" &&
        typeof s.data === "string");

Copyright notice: This article is the original article of the blogger. It is subject to the relevant copyright agreement. If you reprint or share it, please attach the original source link and link source.