Elk Development Journal (1) – elastic 7.2.0 auto start, MySQL batch import, user authentication, SSL communication, Chinese Pinyin segmentation

Time:2020-1-21

Recently, I did search system service for my platform and found elk, a treasure. I took a hole first and wrote some experience while learning

1、 Development path

1. Deploy elasticsearch to serve the system

  • Download the v7.2.0 Debian installation package, which can’t be used for free in the version less than 7.2.0
  • Download address: https://www.elastic.co/downlo
  • Copy to server, executedpkg -i elasticsearch-7.2.0-amd64.debinstall
  • For Internet access, configure/etc/elasticsearch.ymlThe documents are as follows:
Cluster.name: your cluster name
Node.name: your node name. Multiple nodes can use suffixes such as ﹣ 1, ﹣ 2, ﹣ 3 to facilitate management
Network.host: 127.0.0.1 ා es working IP address, default localhost shield remote
Http.port: 9200 ා es work port, default 9200
Discovery.seed ᦇ hosts: ["127.0.0.1"] ᦇ node discovery IP, initial recommendation is the same as working IP
Http.cors.enabled: true enable cross domain request policy
http.cors.allow-origin: "*"
http.cors.allow-headers: X-Requested-With, X-Auth-Token, Content-Type, Content-Length, Authorization
http.cors.allow-credentials: true
  • Call service elasticsearch start or systemctl start elasticsearch.service to start
  • You can also log in to a non root account and call. / bin / elasticsearch – D – P PID
  • If startup fails, check/etc/elasticsearch,/var/log/elasticsearch,/var/lib/elasticsearchWhether the three folders belong to elasticsearch users
  • If you run with a custom user, change the owner of the above three folders to the user you are using
  • If start still fails, modify/usr/lib/systemd/system/elasticsearch.serviceDocument:
Change user = Cosoli to the current user
Group = Cosoli change to the current user group
  • Test:GET http://es.host.name:9200
  • If the returned result is as follows, the configuration is successful:

Be careful!Remember to configure port 92009100560113585044 in the server security policy to allow access

2. Enable HTTP basic authorization

  • Enter es installation directory
  • callbin/elasticsearch-setup-passwords interactiveInitialize the password of each preset account
  • After successful initialization, configure/etc/elasticsearch/elasticsearch.ymlAs follows:
xpack.security.enabled: true
  • Test access:GET http://es.host.name:9200, configuration success will return401 (Unauthorized)error
  • After authentication is enabled, if script communication is used, it needs to be configured in the request headerAuthorizationField:
#Authorization field value is generated by Base64 encoding, JS can call: btoa ('username: password ')
request.header[‘Authorization’] = ‘Basic Y0EtqzNzX129zTpQA2FbG9sa’
  • If you want to use the browser, just enter the user name and password in the pop-up dialog box

3. Enable SSL communication

  • Apply for SSL certificate, try to avoid using self signed certificate, which is easy to be blocked by browser
  • Copy certificate to/etc/elasticsearch/certs/lower
  • To configure/etc/elasticsearch/elasticsearch.ymlAs follows:
#Enable SSL transfer between nodes
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
#SSL key of working domain name
xpack.security.transport.ssl.key: certs/your.ssl.cert.key
#SSL certificate of working domain name, certificate chain may be incomplete, please refer to the following prompt!!!!!
xpack.security.transport.ssl.certificate: certs/your.ssl.cert_chain.crt

#Enable server HTTPS communication
xpack.security.http.ssl.enabled: true
#SSL key of working domain name
xpack.security.http.ssl.key: certs/your.ssl.cert.key
#SSL certificate of working domain name, certificate chain may be incomplete, please refer to the following prompt!!!!!
xpack.security.http.ssl.certificate: certs/your.ssl.cert_chain.crt
  • adoptopenssl s_client -connect host:9200 -showcertsTo view certificate chain information, you should return0(ok)

Be careful!Incomplete certificate chain may cause the file recognized as unsafe by the browser! Try not to use self signed certificates! You can apply for a free SSL certificate online,During deployment, the public key certificate and certificate chain are combined into a CRT file as a certification fileIt doesn’t matter if you don’t understand the principle. You can use Notepad + + to put two pieces of keys together, first certificate and then certificate chain.

4. Access es through postman, dejavu, ES head (skip 5)

5. Deploy kibana to access the visual es query + analysis + management interface

  • Download the v7.2.0 Debian installation package (must be the same version as elasticsearch)
  • Download address: https://www.elastic.co/downlo

6. IK Chinese word segmentation device

  • Install automatically. Use the following command to install the plug-in directly from GitHub (slightly slow network speed):
./bin/elasticsearch-plugin install http://github.com/medcl/elasticsearch-analysis-ik/releases/download/v7.2.0/elasticsearch-analysis-ik-7.2.0.zip
  • Install manually, download the zip file to local, copy to./elasticsearch/plugins/ik/Catalog
  • Enter the IK directory and callunzip elasticsearch-analysis-ik-7.2.0.zipdecompression

7. Pinyin

8. Import data from Mysql to es

  • Note: set mapping on import

9. Deploy filebeat to collect logs

  • Use after downloaddpkg -i filebeat-7.2.0-amd64.debInstallation, configurationfilebeat.ymlIf the programs that enable listening are installed in the default location, you can start each required module directly through filebeat module enable without manually configuring the input
  • Usefilebeat setup -eInitialize environment
  • Useservice filebeat startStart collecting log files
  • Please configure carefullyelasticsearch.gcandelasticsearch.deprecateThere are many modules and data explosion. Please consider whether your hard disk is enough

10. Deploy logstash

2、 Manual command aggregation:

  • Enter es Directory:cd /usr/share/elasticsearch
  • The foreground starts es:./bin/elasticsearch
  • Start es in the background:./bin/elasticsearch -d -p pid
  • Restart es manually:
pkill -F pid
bin/elasticsearch -d -p pid
  • Start filebeat manually:
./bin/filebeat -e \
-c /etc/filebeat/filebeat.yml \
-path.home /usr/share/filebeat \
-path.config /etc/filebeat \
-path.data /var/lib/filebeat \
-path.logs /var/log/filebeat