Logstash 7. X installation and data import (Alibaba cloud Ubuntu)

Time:2020-1-22

Continued: installation and interface preview of kibana 7. X (Alibaba cloud Ubuntu)

System environment

  • Operating system: Ubuntu 18.04 lts (Alibaba cloud)
  • System IP
#Intranet, private address
172. Inside. Inside. Inside
#Public address
112. Outer. Outer. Outer
  • Elasticsearch version: 7.2
  • Kibana version: 7.2
  • Logstash version: 7.2

Installation and configuration

  • Official document: https://www.elastic.co/guide/
  • Logstash is different from elasticsearch and kibana and requires a separate Java environment
~$ sudo apt install default-jdk
~$ java -version
openjdk version "11.0.3" 2019-04-16
OpenJDK Runtime Environment (build 11.0.3+7-Ubuntu-1ubuntu218.04.1)
OpenJDK 64-Bit Server VM (build 11.0.3+7-Ubuntu-1ubuntu218.04.1, mixed mode, sharing)
  • Download logstash
wget https://artifacts.elastic.co/downloads/logstash/logstash-7.2.0.tar.gz
  • Unzip logstash
tar -zxf logstash-7.2.0.tar.gz
  • Follow up operations are carried out in the directory of logstash-7.2.0
cd logstash-7.2.0/
  • Download the movielens test data set to GroupLens
wget http://files.grouplens.org/datasets/movielens/ml-latest-small.zip
  • Decompress test data set
unzip ml-latest-small.zip
  • Create and edit logstash.conf file, add the following (Ruby syntax)
input {
  file {
    path => "/home/walker/es/ml-latest-small/movies.csv"
    start_position => "beginning"
    sincedb_path => "/dev/null"
  }
}
filter {
  csv {
    separator => ","
    columns => ["id","content","genre"]
  }
  mutate {
    split => { "genre" => "|" }
    remove_field => ["path", "host","@timestamp","message"]
  }
  mutate {

    split => ["content", "("]
    add_field => { "title" => "%{[content][0]}"}
    add_field => { "year" => "%{[content][2]}"}
  }
  mutate {
    convert => {
      "year" => "integer"
    }
    strip => ["title"]
    remove_field => ["path", "host","@timestamp","message","content"]
  }
}
output {
   elasticsearch {
     hosts => "http://172.18.193.52:9200"
     index => "movies"
     document_id => "%{id}"
   }
  stdout {}
}
  • Import data. Note that it will not exit automatically after import. Use ctrl-c to exit manually
#Personal path is different
./bin/logstash -f /home/walker/es/ml-latest-small/logstash.conf
  • Management view data (index is the table of relational database)

Logstash 7. X installation and data import (Alibaba cloud Ubuntu)

  • Abstraction and analogy between relational database and elastic search

Logstash 7. X installation and data import (Alibaba cloud Ubuntu)

  • Dev tools view the total number of documents (that is, the total number of data)
#View total data
GET /movies/_count

#Console output
{
  "count" : 9743,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  }
}

Logstash 7. X installation and data import (Alibaba cloud Ubuntu)

  • View data with ID 1
#View data with ID 1
GET /movies/_doc/1

#Console output
{
  "_index" : "movies",
  "_type" : "_doc",
  "_id" : "1",
  "_version" : 1,
  "_seq_no" : 121,
  "_primary_term" : 1,
  "found" : true,
  "_source" : {
    "@version" : "1",
    "genre" : [
      "Adventure",
      "Animation",
      "Children",
      "Comedy",
      "Fantasy"
    ],
    "year" : 1995,
    "id" : "1",
    "title" : "Toy Story"
  }
}
  • Delete movies data (table of elasticsearch’s index benchmarkable relational database)
#Delete command
DELETE /movies

#Console output
{
  "acknowledged" : true
}

Walker’s directory structure

$ tree /home/walker/es/ -L 2
/home/walker/es/
├── elasticsearch-7.2.0
│   ├── bin
│   ├── config
│   ├── data
│   ├── jdk
│   ├── lib
│   ├── LICENSE.txt
│   ├── logs
│   ├── modules
│   ├── NOTICE.txt
│   ├── plugins
│   └── README.textile
├── elasticsearch-7.2.0-linux-x86_64.tar.gz
├── kibana-7.2.0-linux-x86_64
│   ├── bin
│   ├── built_assets
│   ├── config
│   ├── data
│   ├── LICENSE.txt
│   ├── node
│   ├── node_modules
│   ├── NOTICE.txt
│   ├── optimize
│   ├── package.json
│   ├── plugins
│   ├── README.txt
│   ├── src
│   ├── target
│   ├── webpackShims
│   └── x-pack
├── kibana-7.2.0-linux-x86_64.tar.gz
├── logstash-7.2.0
│   ├── bin
│   ├── config
│   ├── CONTRIBUTORS
│   ├── data
│   ├── Gemfile
│   ├── Gemfile.lock
│   ├── lib
│   ├── LICENSE.txt
│   ├── logs
│   ├── logstash-core
│   ├── logstash-core-plugin-api
│   ├── modules
│   ├── NOTICE.TXT
│   ├── tools
│   ├── vendor
│   └── x-pack
├── logstash-7.2.0.tar.gz
├── ml-latest-small
│   ├── links.csv
│   ├── logstash.conf
│   ├── movies.csv
│   ├── ratings.csv
│   ├── README.txt
│   └── tags.csv
└── ml-latest-small.zip

This paper is the learning notes of Ruan Yiming’s “elastic search core technology and actual combat”.

Recommended Today

[Redis5 source code learning] analysis of the randomkey part of redis command

baiyan Command syntax Command meaning: randomly return a key from the currently selected databaseCommand format: RANDOMKEY Command actual combat: 127.0.0.1:6379> keys * 1) “kkk” 2) “key1” 127.0.0.1:6379> randomkey “key1” 127.0.0.1:6379> randomkey “kkk” Return value: random key; nil if database is empty Source code analysis Main process The processing function corresponding to the keys command is […]