How to back up the snapshot of elasticsearch to OSS

Time:2020-1-4

Preface

Elasticsearch is an open source distributed restful search and analysis engine. It can store, query and analyze massive data in near real time. It also supports the backup of snapshots to HDFS / S3, while Alibaba cloud OSS is compatible with S3 API. This article will introduce how to use es repository-s3 plug-in to backup snapshots to OSS.

Deployment and configuration

First, we need to install repository-s3. Please refer to the official documents:
https://www.elastic.co/guide/en/elasticsearch/plugins/7.2/repository-s3.html

Starting es, we can see from the log that ES has loaded the plugin:

[2019-07-15T14:12:09,225][INFO ][o.e.p.PluginsService     ] [master] loaded module [aggs-matrix-stats]
[2019-07-15T14:12:09,225][INFO ][o.e.p.PluginsService     ] [master] loaded module [analysis-common]
[2019-07-15T14:12:09,225][INFO ][o.e.p.PluginsService     ] [master] loaded module [ingest-common]
[2019-07-15T14:12:09,226][INFO ][o.e.p.PluginsService     ] [master] loaded module [ingest-geoip]
[2019-07-15T14:12:09,226][INFO ][o.e.p.PluginsService     ] [master] loaded module [ingest-user-agent]
[2019-07-15T14:12:09,226][INFO ][o.e.p.PluginsService     ] [master] loaded module [lang-expression]
[2019-07-15T14:12:09,226][INFO ][o.e.p.PluginsService     ] [master] loaded module [lang-mustache]
[2019-07-15T14:12:09,227][INFO ][o.e.p.PluginsService     ] [master] loaded module [lang-painless]
[2019-07-15T14:12:09,227][INFO ][o.e.p.PluginsService     ] [master] loaded module [mapper-extras]
[2019-07-15T14:12:09,227][INFO ][o.e.p.PluginsService     ] [master] loaded module [parent-join]
[2019-07-15T14:12:09,227][INFO ][o.e.p.PluginsService     ] [master] loaded module [percolator]
[2019-07-15T14:12:09,227][INFO ][o.e.p.PluginsService     ] [master] loaded module [rank-eval]
[2019-07-15T14:12:09,228][INFO ][o.e.p.PluginsService     ] [master] loaded module [reindex]
[2019-07-15T14:12:09,228][INFO ][o.e.p.PluginsService     ] [master] loaded module [repository-url]
[2019-07-15T14:12:09,228][INFO ][o.e.p.PluginsService     ] [master] loaded module [transport-netty4]
[2019-07-15T14:12:09,228][INFO ][o.e.p.PluginsService     ] [master] loaded plugin [repository-s3]
[2019-07-15T14:12:12,375][INFO ][o.e.d.DiscoveryModule    ] [master] using discovery type [zen] and seed hosts providers [settings]
[2019-07-15T14:12:12,801][INFO ][o.e.n.Node               ] [master] initialized
[2019-07-15T14:12:12,802][INFO ][o.e.n.Node               ] [master] starting ...

Then, we need to configure the access key and secret key used by OSS to es, and execute the following commands respectively:

bin/elasticsearch-keystore add s3.client.default.access_key
bin/elasticsearch-keystore add s3.client.default.secret_key

Function

First, we create a backup:

[[email protected] ~]# curl -XPUT 'http://localhost:9200/_snapshot/test' -H 'Content-Type: application/json' -d '{ "type": "s3", "settings": { "bucket": "hadoop-oss-test", "endpoint": "oss-cn-zhangjiakou-internal.aliyuncs.com"} }'
{"acknowledged":true}

NOTE:The above command uses HTTPS protocol to transfer data by default. If you want to use HTTP protocol, you need to"protocol": "http", "disable_chunked_encoding": trueAdd tosettingsThis feature will be available after the new version is released.

You can use the following command to make sure the creation is successful:

[[email protected] ~]# curl -XGET localhost:9200/_snapshot/test?pretty
{
  "test" : {
    "type" : "s3",
    "settings" : {
      "bucket" : "hadoop-oss-test",
      "endpoint" : "oss-cn-zhangjiakou-internal.aliyuncs.com"
    }
  }
}

We can write some test data to es, and then look at the index information of the current cluster:

[[email protected] ~]# curl -X GET "localhost:9200/_cat/indices?v"
health status index    uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   sales    89ouBy6RQsuT34QRbn_jeQ  10   0     271786            0       15mb           15mb
green  open   customer fQCMEvXsQOu0UgMm1SAJlA   5   0      10000            0      717kb          717kb

Let’s just back up the sales index:

[[email protected] ~]# curl -XPUT 'http://localhost:9200/_snapshot/test/sales' -H 'Content-Type: application/json' -d '{ "indices": "sales" }'
{"accepted":true}

Then we can see the backup results from the OSS console:

How to back up the snapshot of elasticsearch to OSS

Now let’s write some data into the sales index:

[[email protected] ~]# curl -X GET "localhost:9200/_cat/indices?v"
health status index    uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   sales    89ouBy6RQsuT34QRbn_jeQ  10   0     281502            0     15.6mb         15.6mb
green  open   customer fQCMEvXsQOu0UgMm1SAJlA   5   0      10000            0      717kb          717kb

We use the snapshot just backed up to OSS to recover the sales index, and execute the following commands respectively:

[[email protected] ~]# curl -XPOST localhost:9200/sales/_close
{"acknowledged":true,"shards_acknowledged":true,"indices":{"sales":{"closed":true}}}
[[email protected] ~]# curl -XPOST 'http://localhost:9200/_snapshot/test/sales/_restore?pretty'
{
  "accepted" : true
}
[[email protected] ~]# curl -X GET "localhost:9200/_cat/indices?v"
health status index    uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   sales    89ouBy6RQsuT34QRbn_jeQ  10   0     271786            0       15mb           15mb
green  open   customer fQCMEvXsQOu0UgMm1SAJlA   5   0      10000            0      717kb          717kb

As we can see, the sales index is the same as before.

Reference material

https://www.elastic.co/guide/en/elasticsearch/plugins/7.2/repository-s3.html

https://www.elastic.co/cn/products/elasticsearch



Author: Cold Moon

Read the original text

This is the original content of yunqi community, which can not be reproduced without permission.

Recommended Today

Analysis of new data type instances of ES6 learning notes

An example of this paper describes the new data types of ES6 learning notes. To share with you for your reference, as follows: 1. Data deconstruction assignment 1. Deconstruction and assignment of arrays Basic usage: let [key1, key2…] = [value1, Value2…] Let [name, age, sex] = [‘xiaoming ‘, 24,’ male ‘]; console.log(name); console.log(age); console.log(sex); Note […]