Alibaba Cloud Elasticsearch: Lifecycle Management for Index Data

Scenario Introduction

Procedure

Configure ILM in Heartbeat Configuration

heartbeat.monitors:
- type: icmp
schedule: '*/5 * * * * * *'
hosts: ["47.111.169.233"]
setup.template.settings:
index.number_of_shards: 3
index.codec: best_compression
index.routing.allocation.require.box_type: "hot"
setup.kibana:
# Kibana Host
# Scheme and port can be left out and will be set to the default (http and 5601)
# In case you specify and additional path, the scheme is required: http://localhost:5601/path
# IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
host: "https://es-cn-4591jumei000u1zp5.kibana.elasticsearch.aliyuncs.com:5601"
output.elasticsearch:
# Array of hosts to connect to.
hosts: ["es-cn-4591jumei000u1zp5.elasticsearch.aliyuncs.com:9200"]
ilm.enabled: true
setup.template.overwrite: true
ilm.rollover_alias: "heartbeat"
ilm.pattern: "{now/d}-000001"
# Enabled ilm (beta) to use index lifecycle management instead daily indices.
#ilm.enabled: false
# Optional protocol and basic auth credentials.
#protocol: "https"
username: "elastic"
password: "Elastic@363"

ILM and Settings

Customize ILM Policies

PUT /_ilm/policy/heartbeat-policy
{
"policy": {
"phases": {
"hot": {
"actions": {
"rollover": {
"max_size": "5mb",
"max_age": "1d",
"max_docs": 100
}
}
},
"warm": {
"min_age": "60s",
"actions": {
"forcemerge": {
"max_num_segments":1
},
"shrink": {
"number_of_shards":1
}
}
},
"cold": {
"min_age": "3m",
"actions": {
"allocate": {
"include": {
"box_type": "warm"
}
}
}
},
"delete": {
"min_age": "1h",
"actions": {
"delete": {}
}
}
}
}
}
  • Hot: When the written data reaches 5 MB, the usage time exceeds one day, or the number of documents exceeds 100, Rollover is triggered as long as any of the conditions are met, and the system creates a new index. The new index restarts the policy, and the current index enters the Warm stage 60 seconds after a rolling update.
  • Warm: After the index enters the Warm stage, ILM scales down the index to one shard and forces the index to be merged into one segment. After this operation is completed, the index will wait for three minutes (calculated from the rolling update) before entering the Cold stage.
  • Cold: After the index enters the Cold stage, ILM moves the index from the Hot node to the Cold node or the Warm node. After the operation is completed, it will wait one hour to enter the Delete stage.
  • Delete: The index enters the Delete stage and will be deleted in one hour.

Rolling Index Management in Kibana

# sudo ./heartbeat -e

Associate an Index with ILM

ILM Policy Cycle

PUT _cluster/settings
{
"transient": {
"indices.lifecycle.poll_interval":"1m"
}
}

Summary

Original Source:

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Alibaba Cloud

Alibaba Cloud

Follow me to keep abreast with the latest technology news, industry insights, and developer trends. Alibaba Cloud website:https://www.alibabacloud.com