Use elasticsearch to periodically delete index data

1. Sometimes we use es

Due to limited resources or business needs, we only want to save the data of the latest period of time, so it is necessary to delete the data regularly.

2. Script

vim del_es_by_day.sh
#!/bin/bash
#Periodically delete elasticsearch index
#author menard 2019-3-25
date=`date -d "-7 days" "+%Y.%m.%d"`
/usr/bin/curl -v --user elastic:password -XDELETE "http://192.168.10.201:9200/*-$date"

Add executable permissions Chmod + X del_ es_ by_ day. sh

3. Create an index for testing

put test-2019.03.18
put index-2019.03.18

4. After executing the script test results, you can see that the deletion is successful

5. Do scheduled tasks

crontab -e
00 01 * * * /workspace/script/del_es_by_day.sh

Supplement: elasticsearch regularly backs up and restores index data

Scheduled backup script

The Linux scheduled task function uses the cron service

Write cron expressions for timed tasks

Crontab – e # enter cron scheduled task editing

Scheduled task

*/1 * * * * /opt/scheduler/es_bk.sh >> /opt/scheduler/bk_log.txt 2>&1

Execute es in / opt / scheduler / directory every 1 minute_ The bk.sh script writes the data contents to bk.log.in the / opt / scheduler directory Txt file

Check es_ Bk.sh script content

#! /bin/bash
echo '=================================start======================================'
#Delete backup snapshot
curl -i -X DELETE localhost:9200/_snapshot/es_backup/snapshot01
#Back up again
curl -i -X PUT localhost:9200/_snapshot/es_backup/snapshot01
echo '==================================end======================================='

Es backup data requires a snapshot of the index data to be backed up. A snapshot name needs to be specified, and the same snapshot cannot be used. Therefore, the old snapshot needs to be deleted before each backup.

Es backup and recovery

Create a backup warehouse (directory)

mkdir -p /bk/es/data
#Modify permissions
chmod -R 777 bk

Modify elasticsearch YML file, specify warehouse location

Send initialization warehouse request

curl -i -H ""'Content-Type:application/json;charset=UTF-8'"" -X PUT --data '{"type": "fs","settings": {"location": "/bk/es/data"}}' localhost:9200/_snapshot/es_backup

es_ Backup is the backup namespace, which can be specified at will

Create first snapshot

curl -i -X PUT localhost:9200/_snapshot/es_backup/snapshot01

Will use es_ Backup backs up all index data of ES to snapshot 01. Of course, you can only back up the specified index

curl -i -H ""'Content-Type:application/json;charset=UTF-8'"" -X PUT --data '{"indices": "'bk_user_index_server'"}' localhost:9200/_snapshot/es_backup/snapshot01

Restart scheduled tasks

systemctl restart cron

Restore index

Restore specified

curl -i -H ""'Content-Type:application/json;charset=UTF-8'"" -X POST --data '{"indices": "'bk_user_index_server'"}' localhost:9200/_snapshot/es_backup/snapshot01/_restore

Restore all

curl -i -X POST localhost:9200/_snapshot/es_backup/snapshot01/_restore

The above is my personal experience. I hope I can give you a reference, and I hope you can support developpaer. If there are mistakes or not fully considered, please don’t hesitate to comment.