windows2016_ X64 build elk (datasource > filebeat > logstash > elastic search > kibana)

Time:2021-5-6

windows2016_ X64 build elk (datasource > filebeat > logstash > elastic search > kibana)

  1. The example log program is based on asp.net core 3.0 + NLog
  2. Elk related procedures are as follows:

  3. Log component recommendation:
java: logback、lo4j
asp.net: log4net、nlog

Client installation and operation

  1. Download the filebeat installation packagefilebeat7.5.1
  2. Edit filebeat.yml. The configuration of the simplified version is as follows:
filebeat.inputs:
- type: log
  enabled: true
  paths:
    #- /var/log/*.log
    - F:\logs\xxx\*.log

filebeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml

  # Set to true to enable config reloading
  reload.enabled: false

output.logstash:
  #Logstash address and port
  hosts: ["192.168.3.43:5044"]

processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~
  1. Run the program and generate the log (Windows 7)_ X64|192.168.3.42:8090), log file name: log-2020-01-08.log
2012-01-08 10:21:41.5296|0|INFO|logfile|windows service start... 
2012-01-08 11:03:03.5926|0|INFO|logfile|windows service end... 
2012-01-08 11:04:18.2756|0|INFO|logfile|windows service start... 
2012-01-08 11:04:26.7556 | 0 | info | logfile | test 0
  1. Filebeat package file root directory executor: filebeat – E – C filebeat.yml
D:\devops\filebeat-7.5.1-windows-x86_64>filebeat -e -c filebeat.yml
...
2020-01-08T14:06:16.038+0800    INFO    [monitoring]...
  1. For more information, seeFilebeat official document

Server installation and operation

  1. Install JDK and configure environment variables. Jdk11 + is required for elasticsearch version 7.5.1.JDK11
JAVA_HOME: C:\Program Files\jdk-11.0.5
CLASSPATH: .;%JAVA_HOME%\lib;%JAVA_HOME%\lib\tools.jar;%JAVA_HOME%\lib\dt.jar
Path: Java_ HOME%\bin;% JAVA_ HOME%\jre\bin;
  1. Detecting JDK with CMD > java – version
java version "11.0.5" 2019-10-15 LTS
Java(TM) SE Runtime Environment 18.9 (build 11.0.5+10-LTS)
Java HotSpot(TM) 64-Bit Server VM 18.9 (build 11.0.5+10-LTS, mixed mode)
  1. Configure elasticsearch environment variable
ES_HOME:C:\Program Files\elasticsearch-7.5.1
Path append% es_ HOME%\bin;
  1. Edit / config / elasticsearch.yml. The configuration of the simplified version is as follows:
cluster.name: elasticsearch-application
node.name: node-1
network.host: 192.168.3.41
http.port: 9200
cluster.initial_master_nodes: ["node-1"]
#Configure head CORS
http.cors.enabled: true
http.cors.allow-origin: "*"
node.master: true
node.data: true

5. Run elasticsearch 7.5.1 / bin / elasticsearch.bat and visit 192.168.3.41:9200. The following JSON string will be displayed to indicate successful deployment.

{
  "name" : "node-1",
  "cluster_name" : "elasticsearch-application",
  "cluster_uuid" : "MUOnu8JbTTe7bT1ru3Vb7Q",
  "version" : {
    "number" : "7.5.1",
    "build_flavor" : "default",
    "build_type" : "zip",
    "build_hash" : "3ae9ac9a93c95bd0cdc054951cf95d88e1e18d96",
    "build_date" : "2019-12-16T22:57:37.835892Z",
    "build_snapshot" : false,
    "lucene_version" : "8.3.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}
  • Install elasticsearch head plug-in
  1. Node and grunt are required to install head in Es5 or above.downloadnodejsAnd install.
  2. Add nodejs environment variable
Path is appended: C: // program files \ \ nodejs \;
  1. After the installation, execute node – V with CMD to view the version number – > v10.15.0
  2. Run NPM install – G grunt cli to install grunt. After the installation, run grunt – version to check whether the installation is successful. The version number of the installation will be displayed
  3. stayhttps://github.com/mobz/elasticsearch-headTo download the head plug-in in, select download zip
  4. Unzip the file to the specified folder, and then enter the folder G: (elasticsearch-7.5.1) elasticsearch head master. Edit gruntfile.js and add the corresponding host name: ‘*’
connect: {
	server: {
		options: {
			hostname:"*",
			port: 9100,
			base: '.',
			keepalive: true
		}
	}
}
  1. After the NPM install is completed (receiving 100%) in D: elasticsearch-7.5.1: elasticsearch head master.
  2. Run the grunt server or NPM run start to run the head plug-in. If it is not successful, re install grunt.
  3. Rerun elasticsearch 7.5.1 / bin / elasticsearch.bat, visit 192.168.3.41:9100, and the elasticsearch head page appears, indicating that the installation is successful, as shown in the following figure:
  1. Edit / kibana-7.5.1-windows-x86_ 64 / config / kibana.yml. The configuration of the simplified version is as follows:
server.port: 9101
server.host: "0.0.0.0"
elasticsearch.hosts: ["http://192.168.3.41:9200"]
  1. Run kibana-7.5.1-windows-x86_ If you visit 192.168.3.40:9101, the kibana page will appear successfully, which means the deployment is successful.
  1. In windows, logstash execution path cannot have spaces
  2. Edit the file D / logstash-7.5.1/config/logstash-sample.conf. The simplified version is as follows:
input {
  beats {
    port => 5044
  }
}

output {
  elasticsearch {
    hosts => ["http://192.168.3.41:9200"]
    index => "log-%{+YYYY.MM.dd}"
  }
}
  1. Copy logstash-sample.conf to the bin root directory and execute the command: logstash – F logstash-sample.conf — config.reload.automatic

function

  1. Run elasticsearch, kibana, logstash, filebeat and log in turn
  2. Creating log index in kibana > disconver

Other construction methods

  • datasource->logstash->elasticsearch->kibana
  • datasource->filebeat->kafka->logstash->elasticsearch->kibana
  • datasource->filebeat->logstash->redis/kafka->logstash-> elasticsearch->kibana

Recommended Today

Solve the problem of configuring nginx in Ubuntu

Various errors in configuring nginx in Ubuntu 18.04 lackPCRE Library Compiling nginx An error occurred Error installing PCRE Library Manual compilation and installation of PCRE Library (1) Download and unzip the PCRE library wget https://ftp.pcre.org/pub/pcre/pcre-8.43.tar.gz tar -xvf pcre-8.43.tar.gz (2) Compile and install PCRE Library cd pcre-8.43 sudo ./configure sudo make sudo make install Recompile nginx […]