Elasticsearch 6.4.0 cluster building

Time:2020-3-25

Recently, in order to facilitate their own use, a 3-node es cluster has been built on the local virtual machine. In the process of building, there are many holes, and the information on the Internet is relatively scattered, so please sort out the building process in detail for reference. In the process of building, because there are two virtual machines, there is a very fast way to build one, and then clone the virtual machine and modify it quickly. But in order to be more familiar with the whole process, this process is to operate each virtual machine one by one.

1、 Software and environment preparation

  • Server environment

CentOS 6.9-64 bit two sets
es-master:192.168.0.11
es-slave:192.168.0.12
Note: if memory allows, slightly increase the memory, otherwise startup will fail. 1.5-2g memory can be used for local test. Two nodes are ready to be built on ES master, so it is better to give more than 2G memory

  • ES software

Latest version of elastic search 6.4.0
Download address: https://artifacts.elastic.co/

  • JDK software

jdk-8u161-linux-x64.tar.gz
Download address: http://download.oracle.com/ot

  • Environmental planning

Es master consists of two nodes: elasticsearch-node1 and elasticsearch-node3
Es slave includes a node: elasticsearch-node2

  • Modify the host name of the server for subsequent operation

(1) Operation on server 192.168.0.11:

[[email protected] ~]# hostname es-master
[[email protected] ~]# vim /etc/sysconfig/network
#Modify the following:
HOSTNAME=es-master
#Sign out and sign in again
[[email protected] ~]# logout

(2) Operation on server 192.168.0.12:

[[email protected] ~]# hostname es-slave
[[email protected] ~]# vim /etc/sysconfig/network
#Modify the following:
HOSTNAME=es-slave
#Sign out and sign in again
[[email protected] ~]# logout

(3) For the convenience of subsequent operation, the firewalls of ES master and ES slave servers are closed respectively

[[email protected] ~]# service iptables stop
[[email protected] ~]# service iptables stop

Note: the firewall cannot be closed directly in the production environment. Firewall rules can be added to make the TCP port of ES open to the remote designated host.

2、 JDK installation configuration

Operation on the host es master

  • Download and extract to the specified directory
[[email protected] ~]# mkdir -pv /home/software
 [[email protected] ~]# cd /home/software/
 [[email protected] software]# wget http://download.oracle.com/otn-pub/java/jdk/8u181-b13/96a7b8442fe848ef90c96a2fad6ed6d1/jdk-8u181-linux-x64.tar.gz
 Note: if there is no network, you can download it manually and upload it.
 [[email protected] software]# mkdir -pv /usr/local/java
 [[email protected] software]# tar xf jdk-8u181-linux-x64.tar.gz -C /usr/local/java

The operation on the host es slave is the same as above, omitted

  • The process of configuring environment variables is the same on ES master and ES slave, as follows
[[email protected] software]# vim /etc/profile
#Add the following at the end:
JAVA_HOME=/usr/local/java/jdk1.8.0_181
export PATH=$PATH:$JAVA_HOME/bin
#Make environment variables effective
[[email protected] ~]# source /etc/profile
  • Test whether the installation is completed
[[email protected] ~]# java -version
java version "1.8.0_161"
#Output version number, indicating successful configuration

3、 Elasticsearch installation and configuration

The operations on ES master are as follows:

  • Download and unzip elasticsearch 6.4.0
[[email protected] ~]# cd /home/software/
[[email protected] ~]# wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.4.0.tar.gz
[[email protected] software]# tar xf elasticsearch-6.2.4.tar.gz -C /usr/local/
[[email protected] local]# mv elasticsearch-6.4.0 elasticsearch-node1/
#Copy one as the second es node
[[email protected] local]# cp -r elasticsearch-node1/ elasticsearch-node3
  • Create the elk user and authorize the elasticsearch-node1 and elasticsearch-node3 to elk
Note: since the elastic search cannot be started directly with root, you need to create a normal user
[[email protected] local]# useradd elk
[[email protected] local]# chown -R elk:elk elasticsearch-node1/ elasticsearch-node3/
  • Create data directory and log directory of two elastic search nodes respectively
[[email protected] local]# mkdir -pv /data/{elasticsearch1,elasticsearch3}/{data,logs}
[[email protected] local]# chown -R elk:elk /data/{elasticsearch1,elasticsearch3}
[[email protected] local]# vim /etc/sysctl.conf
#Add the following at the end of the configuration file
vm.max_map_count=655360

Explanation: the max map count file contains a limit on the number of Vmas a process can have. The virtual memory area is a continuous virtual address space area.
In the life cycle of a process, these areas are created whenever a program attempts to map files in memory, link to shared memory segments, or allocate heap space.
When the process reaches the VMA online but can only release a small amount of memory to other kernel processes, the operating system will throw a memory shortage error.

#Make the modified profile effective
[[email protected] local]# sysctl -p
  • Modify the limits.conf file
#Limits.conf: used to protect system resource access, similar to sysctl.conf, but limits.conf is for users and sysctl.conf is for operating system
[[email protected] local]# vim /etc/security/limits.conf
#Add the following at the end of the file:
elk soft nofile 65536
elk hard nofile 131072
elk soft nproc 4096
elk hard nproc 4096
  • Modify the configuration file jvm.options of elasticsearch-node1 node
[[email protected] local]# cd elasticsearch-node1/config/
[[email protected] config]# vim jvm.options
Modify the following two options:
-Xms512m - the initial heap memory size allocated by the JVM when elasticsearch starts
-The maximum heap memory size allowed to be allocated by the JVM after xmx512m ᦇ elastic search is started, which may need to be increased in the production environment
Note: if the memory is large enough, it can not be modified. The default is 1g
  • Modify the configuration file elasticsearch.yml of the elasticsearch-node1 node
[[email protected] local]# vim elasticsearch-node1/config/elasticsearch.yml
#Modify the following
#Represents the cluster ID. multiple nodes in the same cluster use the same ID
cluster.name: elasticsearch
#Node name
node.name: "es-node1"
#Data storage directory
path.data: /data/elasticsearch1/data
#Log directory
path.logs: /data/elasticsearch1/logs
#The IP address to which the node is bound, and the node is notified to other nodes in the cluster
network.host: 192.168.0.11
#Bind the listening network interface to listen for incoming requests. It can be set to IP address or host name
network.bind_host: 192.168.0.11
#The publishing address is used to notify other nodes in the cluster and communicate with other nodes. If it is not set, it can be set automatically by default. Must be an existing IP address
network.publish_host: 192.168.0.11
#The HTTP port of the external service, the default is 9200
http.port: 9200
#The initial list of primary nodes in the cluster. When the primary node is started, this list will be used for non primary node monitoring
discovery.zen.ping.unicast.hosts: ["192.168.0.11","192.168.0.12"]
#The following parameter controls the minimum number of master nodes that a node needs to see before it can operate in the cluster. The official recommended value is (n / 2) + 1;
#Where n is the number of nodes with master qualification (our case is 3, so this parameter is set to 2)
#However, when there are only two nodes, setting it to 2 will cause some problems. After a node is down, it will definitely not connect to two servers, which should be noted
discovery.zen.minimum_master_nodes: 2
#Es turns on memory address locking by default, in order to avoid memory exchange and improve performance. However, centos6 does not support seccomp function, and an error will be reported when starting, so it needs to be set to false
bootstrap.memory_lock: false
bootstrap.system_call_filter: false
  • Modify the configuration file jvm.options of elasticsearch-node3 node
[[email protected] local]# cd elasticsearch-node3/config/
[[email protected] config]# vim jvm.options
Modify the following two options:
-Xms512m
-Xmx512m
  • Modify the configuration file elasticsearch.yml of the elasticsearch-node3 node
[[email protected] local]# vim elasticsearch-node3/config/elasticsearch.yml
#Modify the following
cluster.name: elasticsearch
node.name: "es-node3"
path.data: /data/elasticsearch3/data
path.logs: /data/elasticsearch3/logs
network.host: 192.168.0.11
network.bind_host: 192.168.0.11
network.publish_host: 192.168.0.11
http.port: 9201
discovery.zen.ping.unicast.hosts: ["192.168.0.11","192.168.0.12"]
discovery.zen.minimum_master_nodes: 2
bootstrap.memory_lock: false
bootstrap.system_call_filter: false

At this point, the configuration operation on ES master is completed.


The operations on ES slave are as follows:

  • Upload the installation package of elastic search, extract and rename it. You can pull it from ES master remotely
[[email protected] ~]# cd /home/software/
[[email protected] software]# scp 192.168.0.11:/home/software/elasticsearch-6.4.0.tar.gz ./
[[email protected] software]# tar xf elasticsearch-6.4.0.tar.gz -C /usr/local/
  • Create the elk user and authorize the elasticsearch-node2 to elk
[[email protected] local]# useradd elk
[[email protected] local]# chown -R elk:elk elasticsearch-node2
  • Create a data directory and change the owner and group of the directory to elk user and elk group
[[email protected] local]# mkdir -pv /data/elasticsearch2/{data,logs}
[[email protected] local]# chown -R elk:elk /data/elasticsearch2
  • Modify the kernel configuration file sysctl.conf of the operating system
[[email protected] local]# vim /etc/sysctl.conf
#Add the following at the end of the configuration file
vm.max_map_count=655360
[[email protected] local]# sysctl -p
  • Modify the limits.conf file, which can be copied directly from the ES master configuration
[[email protected] local]# vim /etc/security/limits.conf
#Add the following at the end of the file:
elk soft nofile 65536
elk hard nofile 131072
elk soft nproc 4096
elk hard nproc 4096
  • Modify the configuration file jvm.options of elasticsearch-node2 node
[[email protected] local]# cd elasticsearch-node2/config/
[[email protected] config]# vim jvm.options
Modify the following two options:
-Xms512m
-Xmx512m
  • Modify the configuration file elasticsearch.yml of the elasticsearch-node2 node
[[email protected] local]# vim elasticsearch-node2/config/elasticsearch.yml
#Modify the following
cluster.name: elasticsearch
node.name: "es-node2"
path.data: /data/elasticsearch2/data
path.logs: /data/elasticsearch2/logs
network.host: 192.168.0.12
network.bind_host: 192.168.0.12
network.publish_host: 192.168.0.12
discovery.zen.ping.unicast.hosts: ["192.168.0.11","192.168.0.12"]
discovery.zen.minimum_master_nodes: 2
bootstrap.memory_lock: false
bootstrap.system_call_filter: false

At this point, the elastic search node on ES slave is configured.

4、 Start es cluster

  • First, switch to elk user and start the elasticsearch node2 node on the ES slave server
[[email protected] local]# su - elk
[[email protected] ~]$ cd /usr/local/elasticsearch-node2/bin/
[[email protected] bin]$ ./elasticsearch -d
Note: the - D parameter indicates that the subsequent process starts, and the log will be output in the console by default.
  • To check whether the startup is successful, you can check the process and port to determine whether the startup is successful. If the startup fails, you can check the log to find out the reason
[[email protected] bin]$ ps -ef| grep elasticsearch
[[email protected] bin]$ lsof -i:9200
Note: the log file is in the directory specified in the elasticsearch.yml configuration file
  • Test if it can be accessed
[[email protected] ~]$ curl 192.168.0.12:9200
If the following results are returned, the startup is normal:
{
  "name" : "es-node2",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "gczyFNKrTe2YbEQmlg7-nw",
  "version" : {
    ...
  },
  "tagline" : "You Know, for Search"
}
  • Switch to the ES master server, then switch to elk identity, and start the elasticsearch-node1 node and elasticsearch-node3 node respectively
[[email protected] ~]# su - elk
[[email protected] ~]$ cd /usr/local/elasticsearch-node1/bin/
[[email protected] bin]$ ./elasticsearch -d
[[email protected] bin]$ cd /usr/local/elasticsearch-node3/bin/
[[email protected] bin]$ ./elasticsearch -d
  • Check the process and port to determine whether it is started successfully
#View the process. You can see the process with two nodes
[[email protected] bin]$ ps -ef| grep elasticsearch
[[email protected] bin]$ lsof -i:9200
[[email protected] bin]$ lsof -i:9300
  • Use curl to test whether the node can be accessed normally. The method is the same as above. You need to verify the elasticsearch-node1 and elasticsearch-node3 nodes
[[email protected] ~]$ curl 192.168.0.11:9200
[[email protected] ~]$ curl 192.168.0.11:9300
  • After verification, the ES cluster is started.

5、 Installation and configuration of ES front-end graphical operation tool

  • Download kibana and upload it to es master server
Download address: https://artifacts.elastic.co/downloads/kibana/kibana-6.4.0-linux-x86_.tar.gz
  • Unzip, rename
[[email protected] software]# wget https://artifacts.elastic.co/downloads/kibana/kibana-6.4.0-linux-x86_64.tar.gz
[[email protected] software]# tar xf kibana-6.4.0-linux-x86_64.tar.gz -C /usr/local
[[email protected] software]# cd /usr/local
[[email protected] local]# mv kibana-6.4.0-linux-x86_64 kibana-6.4.0
  • Modify kibana’s configuration file kibana.yml
[[email protected] local]# cd kibana-6.4.0/config
[[email protected] config]# vim kibana.yml
#Amend the following:
#External service listening port
server.port: 5601
#The binding can access the IP address of 5601 port service. 0.0.0.0 means that any address can be accessed without firewall restrictions. The production environment is not set in this way and is not secure.
server.host: "0.0.0.0"
#The default value is host name, which means the host bound by kibana instance, which can be IP address or host name
server.name: "es-master"
#Service URL used to process es requests
elasticsearch.url: "http://192.168.0.11:9200"
#Used to control the certificate authentication. The optional values are full, none and certificate. There is no certificate here, so it is set to null, otherwise the startup will prompt an error
elasticsearch.ssl.verificationMode: none
  • Start kibana
[[email protected] config]# cd /usr/local/kibana-6.4.0/bin
#After the process starts, kibana will start in console mode by default, and Ctrl + C will exit
[[email protected] bin]# nohup ./kibana &
#Check whether the log starts normally
[[email protected] bin]# tail -f nohup.out
  • If the startup is normal, visit http://192.168.0.11:5601 on the browser side to see the graphical operation tool

At this point, we can start the journey of ES grammar learning happily.
For the grammar learning of ES, please refer to: https://www.elastic.co/guide /.

6、 Stop of elasticsearch and kibana

At present, there is no script to stop es service and kibana service in bin directory of ES and kibana. You can use kill-9 PID to stop the service

  • Stop es service
[[email protected] bin]# ps -ef| grep elasticsearch | grep -v grep | awk '{print $2}'
[root @ es master bin] ා kill - 9 PID [PID output in the previous step]
  • Stop kibana service
[[email protected] bin]# ps -ef| grep node | grep -v grep | grep -v elasticsearch | awk '{print $2}'
[root @ es master bin] ා kill - 9 PID [PID output in the previous step]

So far, all the contents are sorted out. Welcome to forward the comments!

Further articles will be updated inPersonal stationOn, welcome to view.

In addition, we provide some excellent it video materials, which can be downloaded for free! Please check https://www.592xuexi.com if necessary

Recommended Today

Understanding and deepening of relative path and absolute path

What is relative path and absolute path Last week’s report solved some problems, but also exposed many problems, one of which is the relative path and absolute path. For PHP using xampp to build a server, the relative path refers to the current file relative to the user’s access, and the absolute path refers to […]