Elasticsearch is a distributed and restful search and data analysis engine. ——Elastic stack official website
Building elasticsearch’s “things”“
One day, on the deck of the golden Melly, Weibo told Saab that it was necessary to use elasticsearch database in the following project development process, which was mainly used for real-time query of data collected by various devices on the golden Melly, and maybe later used in the construction and application of distributed log system, so as to let Saab do a preliminary research on technology. As a result, a large number of data were searched in Saab, and it was found that the information on the Internet was almost the same, or most of them did not have a complete process. Even, it is found that for the practical application of elasticsearch database, most pirates are only limited to elk and other situations. For example, when most pirates mention redis, they just think that this device can only be used in the cache level, but in fact, the function of others is so powerful that it exceeds the imagination of the pirates. Even Saab found a free trial resource of elasticsearch database in an Alibaba site. But for the free lunch, maybe after charging for that day, it can only use a “wave” to commemorate the evil of gold. In addition to the traditional deployment of elasticsearch database, there is a docker, which can not only run elasticsearch database stably, but also save a lot of server memory. So Saab began to explore the way to build elastic search on docker
What happened when docker deployed elasticsearch“
1. Foundation prerequisites
One VM host of Alibaba cloud or virtual machine, and the docker container is installed on its own:
Ps[ ⚠️ Precautions]:
For docker installation, please refer to:Build and deploy docker container based on centos7 in Linux Environment
2. Maximum number of open files
[2.1] parameter optimization: ulimit – a | grep open
[[email protected] ~]# ulimit -a |grep open
open files (-n) 65535
[[email protected] ~]#[2.2] modify the configuration file / etc / security/ limits.conf
#Add the following two lines at the bottom
* soft nofile 65536
* hard nofile 65536
* soft nproc 2048
* hard nproc 4096[2.3] confirm whether to modify: ulimit – a | grep open
[[email protected] ~]# ulimit -a |grep open
open files (-n) 65535
[[email protected] ~]#3. Adjust the number of system processes
Modification method: VIM / etc / security / limits.d/20- nproc.conf
#Adjust to the following configuration
* soft nproc 4096
root soft nproc unlimited4. Adjust the virtual memory and the maximum number of concurrent connections
Virtual memory – > vm.max_ map_ count
Elasticsearch uses the hybrid mmapfs / niofs directory by default to store indexes. The default operating system has too low a limit on the MMAP count, which may cause an out of memory exception
[1] Temporary solution:
sysctl –w vm.max_map_count=262144
sysctl –w vm.max_map_count=262144If it is not optimized, the following prompt will appear when elasticsearch is started:
max virtual memory areas vm.max_map_count [65535] is too low, increase to at least [262144][2] Permanent effective scheme: update system file / etc/ sysctl.conf Of vm.max_ map_ Count field
Execute command: VIM / etc/ sysctl.conf
[[email protected] ~]# vim /etc/sysctl.conf
[[email protected] ~]#
#Settings Elasticsearch Virtual Memory—>655360
vm.max_map_count=655360
fs.file-max=655360
vm.overcommit_memory=1Load to system: sysctl – P — load / etc/ sysctl.conf
[[email protected] elasticsearch]# sysctl -p --load /etc/sysctl.conf
vm.swappiness = 0
net.ipv4.neigh.default.gc_stale_time = 120
net.ipv4.conf.all.rp_filter = 0
net.ipv4.conf.default.rp_filter = 0
net.ipv4.conf.default.arp_announce = 2
net.ipv4.conf.lo.arp_announce = 2
net.ipv4.conf.all.arp_announce = 2
net.ipv4.tcp_max_tw_buckets = 5000
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_syn_backlog = 1024
net.ipv4.tcp_synack_retries = 2
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1
kernel.sysrq = 1
vm.max_map_count = 655360
fs.file-max = 655360
vm.overcommit_memory = 1
[[email protected] elasticsearch]#5. Turn off elasticsearch self detection
stay elasticsearch.yml Add configuration item in:
bootstrap.memory_lock: false
bootstrap.system_call_filter: falsePs[ ⚠️ Precautions]:
It is mainly to avoid the following problems:
[1] Max file descriptors [1024] for elasticsearch process is too low, increase to at least [65536]
[2] Adjust virtual memory and maximum number of concurrent connections: Max virtual memory areas vm.max_ map_ count [65530] is too low, increase to at least [262144]
[3] Turn off elasticsearch self detection: system call filters failed to install; check the logs and fix your configuration or disable system call filters at your own risk
[4] Adjust the number of system processes: max number of threads [1024] for user [lish] like too low, increase to at least [2048]
Actual deployment of elasticsearch service
Basic steps for docker to deploy application services: Search Image > pull Image > run
1. Query elasticsearch image:
docker search elasticsearch
PS [precautions]:
- Generally, image resources are pulled from the official docker warehouse[docker-hub]Pull or build your own docker cloud warehousealiyun-dockeretc.
- The elk images selected in this tutorial are all based on the official elk docker warehouseelastic-io
2. Pull elasticsearch image:
docker pull docker.elastic.co/elasticsearch/elasticsearch:7.3.1
PS [precautions]:
1. This tutorial adopts version 7.3. X, and the latest version is 7.4. X [mainly built in Alibaba cloud with version 7.3. X, to avoid pit entry PROBLEMS]
2. The [net / http: TLS handshake timeout] problem may occur during the pull process. Try it several times more, mainly due to the limitation of network bandwidth
3. Modify the image name:
docker tag docker.elastic.co/elasticsearch/elasticsearch:7.3.1 elasticsearch:latest
PS [precautions]:
1. The name is too long, which makes it inconvenient to view. Through docker tag source image [source image] target image, [target image] is recommended- image:target-version ]Format definition, and does not take up space, equivalent to renaming the image
2. For pull kibana[ docker.elastic.co/kibana/kibana : 7.3.1] and logstash[ docker.elastic.co/logstash/logstash It is suggested to modify.
4. Deploy image service:
Deployment command:
docker run -itd -p 9200:9200 -p 9300:9300 --restart=always --privileged=true --name elasticsearch-server -e "discovery.type=single-node" -e ES_JAVA_OPTS="-Xms=512m -Xms=512m" elasticsearch:latestDefault configuration directory:
/usr/share/elasticsearch/config
/usr/share/elasticsearch/logsTo view a list of containers:
docker ps --format "table {{.ID}}\t{{.Names}}\t{{.Ports}}"
PS [precautions]:
1. Open ports [9200 and 9300] – > 9200 are used as HTTP protocol, mainly used for external communication, 9300 as TCP protocol, communication between jars is through TCP protocol, and cluster deployment is usually through 9300. Recommended [host custom port: 9200]
2. — restart = always: configure the container restart policy. When the host machine restarts, it is not necessary to start it manually because it is configured to start automatically
3. — privileged: configure container operation authority [true root operation authority, false current container user operation permission]
4. For the deployment network mode, the default bridging mode is recommended, and the host mode can also be customized
5. Modify the configuration:
Enter container: docker exec – it container ID [container ID] or container name [container name] / bin / Bash
For example: docker exec – it f2d2e97da375 / bin / bash ᦇ f2d2e97da375 – > container ID
Modify the configuration file:
[[email protected] elasticsearch]# ls
LICENSE.txt NOTICE.txt README.textile bin config data jdk lib logs modules plugins
[[email protected] elasticsearch]#
[[email protected] elasticsearch]# cd config
[[email protected] config]# ls
elasticsearch.keystore elasticsearch.yml jvm.options log4j2.properties role_mapping.yml roles.yml users users_roles
[[email protected] config]# vi elasticsearch.ymlAdd cross domain configuration:http.cors.enabled: true && http.cors.allow-origin: "*"
cluster.name: "docker-cluster"
network.host: 0.0.0.0
http.cors.enabled: true
http.cors.allow-origin: "*"Then exit the exit container and restart the container on the host: docker restart container ID [container ID] or container name [container name]
docker restart f2d2e97da375
[[email protected] config]# exit
exit
[[email protected] ~]# docker restart f2d2e97da375
f2d2e97da375
[[email protected] ~]#PS [precautions]:
1. Enter container mode: including using docker attach command or docker exec command,
The docker exec command is recommended. reason:
- Docker attach: using exit to exit the container will cause the container to stop
- Docker exec: using exit to exit the container does not cause the container to stop
- Refer to several methods for docker to enter the container-Several methods of docker entering container
2. If docker has installed the visual interface portainer, it is recommended to enter the container in this way
Build and deploy elasticsearch head service
Elasticsearch head: the web front-end interface of elastic search cluster, built with nodjs, is mainly used to view elasticsearch related information
1. Pull elasticsearch head image: docker pull mobz / elasticsearch- head:5
[[email protected] ~]# docker pull mobz/elasticsearch-head:5
5: Pulling from mobz/elasticsearch-head
75a822cd7888: Pull complete
57de64c72267: Pull complete
4306be1e8943: Pull complete
871436ab7225: Pull complete
0110c26a367a: Pull complete
1f04fe713f1b: Pull complete
723bac39028e: Pull complete
7d8cb47f1c60: Pull complete
7328dcf65c42: Pull complete
b451f2ccfb9a: Pull complete
304d5c28a4cf: Pull complete
4cf804850db1: Pull complete
Digest: sha256:55a3c82dd4ba776e304b09308411edd85de0dc9719f9d97a2f33baa320223f34
Status: Downloaded newer image for mobz/elasticsearch-head:5
docker.io/mobz/elasticsearch-head:5
[[email protected] ~]#2. Modify elasticsearch head image name: docker tag mobz / elasticsearch- head:5 elasticsearch- head:latest
[[email protected] ~]# docker tag mobz/elasticsearch-head:5 elasticsearch-head:latest
[[email protected] ~]# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
grafana/grafana latest 05d1bcf30d16 7 days ago 207MB
nginx latest 540a289bab6c 3 weeks ago 126MB
prom/prometheus latest 2c8e464e47f4 3 weeks ago 129MB
moxm/sentinel-dashboard latest 0ccaac81584e 4 weeks ago 167MB
portainer latest 4cda95efb0e4 4 weeks ago 80.6MB
portainer/portainer latest 4cda95efb0e4 4 weeks ago 80.6MB
apache/skywalking-ui latest fa66ca9c9862 2 months ago 123MB
apache/skywalking-oap-server latest 376a37cdf65c 2 months ago 190MB
docker.elastic.co/kibana/kibana 7.3.1 b54865ba6b0b 2 months ago 1.01GB
docker.elastic.co/elasticsearch/elasticsearch 7.3.1 3d3aa92f641f 2 months ago 807MB
elasticsearch latest 3d3aa92f641f 2 months ago 807MB
prom/node-exporter latest e5a616e4b9cf 5 months ago 22.9MB
google/cadvisor latest eb1210707573 12 months ago 69.6MB
elasticsearch-head latest b19a5c98e43b 2 years ago 824MB
mobz/elasticsearch-head 5 b19a5c98e43b 2 years ago 824MB
tutum/influxdb latest c061e5808198 3 years ago 290MB
[[email protected] ~]#3. Deploy elasticsearch head container
docker run -itd --restart=always --privileged=true -p 9100:9100 --name elasticsearch-head-server elasticsearch-head:latestTo view container services:
docker ps --format "table {{.ID}}\t{{.Names}}\t{{.Ports}}"
4. Browser access:http://remote-ip:9100/
Actual test elasticsearch head service
1. Prepare data:
{
"companyId": "ebcb4f99e0cb4ad781278ae636c1031f",
"Classifyname": hormone detection,
"Devicetype": hormone detection,
"data": {
"deviceId": "20000060100000002",
"deviceNo": "QC00020000060100000002",
"appid": "00000000",
"sim": "89860404191792655118",
"csq": "30",
"electric": "98",
"voltage": "13279",
"softVer": "5143000500010601-01020006|5143000500010601-01010003",
"hardVer": "5143000500010601-00000002|5143000500010601-00000002",
"status": "0",
"date": "2020-09-13 11:23:52",
"elements": [
{
"key": "20",
"value": "10",
"time": "2020-09-13 11:23:52"
},
{
"key": "21",
"value": "11",
"time": "2020-09-13 11:23:52"
},
{
"key": "22",
"value": "12",
"time": "2020-09-13 11:23:52"
},
{
"key": "23",
"value": "13",
"time": "2020-09-13 11:23:52"
}
]
}
}2. Set request header: application / JSON; charset = UTF-8
3. Send put request:http://remote-ip/device/deviceData/QC00020000060100000001
{
"_index": "device",
"_type": "deviceData",
"_id": "QC00020000060100000002",
"_version": 1,
"result": "created",
"_shards": {
"total": 2,
"successful": 1,
"failed": 0
},
"created": true
}4. Open the browser to view the results:
Summary of building elasticsearch
1. Deployment basic command:
docker run -itd -p 9200:9200 -p 9300:9300 --restart=always --privileged=true --name elasticsearch-server --network-alias elasticsearch-server --hostname elasticsearch-server -v /docker/elasticsearch/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml -v /docker/elasticsearch/data:/usr/share/elasticsearch/data -v /docker/elasticsearch/logs:/usr/share/elasticsearch/logs -e "discovery.type=single-node" -e ES_JAVA_OPTS="-server -Xms512m -Xmx512m -Xmn256m -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=320m -XX:+AlwaysPreTouch -XX:-UseBiasedLocking " -e TZ="Asia/Shanghai" elasticsearch:latest2. Configuration file[ elasticsearch.yml ]:
cluster.name: elasticsearch-cluster
node.name: elasticsearch-server
network.host: 0.0.0.0
network.bind_host: 0.0.0.0
network.publish_host: 0.0.0.0
http.cors.enabled: true
http.cors.allow-origin: "*"
bootstrap.memory_lock: false
bootstrap.system_call_filter: false
transport.tcp.port: 9300
transport.tcp.compress: true
http.max_content_length: 128mb3. installation of participle plug-in:
./bin/elasticsearch-plugin install https://github.com/medcl/elasticsearch-analysis-ik/releases/download/v7.5.0/elasticsearch-analysis-ik-7.5.0.zip4. Docker deployment elasticsearch head:
docker run -itd --restart=always --privileged=true -p 9100:9100 --network-alias elasticsearch-head-server --name elasticsearch-head-server --hostname elasticsearch-head-server -v /docker/elasticsearch-head/app:/usr/src/app elasticsearch-head:latest5. Elasticsearch head 406 error:
Ajax settings part of the JSON configuration:
ajaxSettings: {
url: ajaxLocation,
isLocal: rlocalProtocol.test(ajaxLocParts[1]),
global: true,
type: "GET",
//contentType: "application/x-www-form-urlencoded",
contentType: "application/json;charset=UTF-8",
processData: true,
async: true,
/*
timeout: 0,
data: null,
dataType: null,
username: null,
password: null,
cache: null,
traditional: false,
headers: {},
*/
accepts: {
xml: "application/xml, text/xml",
html: "text/html",
text: "text/plain",
json: "application/json, text/javascript",
"*": "*/*",
},JSON configuration of inspectdata part:
// Detect, normalize options and install callbacks for jsonp requests
jQuery.ajaxPrefilter("json jsonp", function (s, originalSettings, jqXHR) {
//var inspectData = s.contentType === "application/x-www-form-urlencoded" &&
var inspectData =
(s.contentType === "application/x-www-form-urlencoded" &&
typeof s.data === "string") ||
(s.contentType === "application/json;charset=UTF-8" &&
typeof s.data === "string");Copyright notice: This article is the original article of the blogger. It is subject to the relevant copyright agreement. If you reprint or share it, please attach the original source link and link source.