Alibaba Cloud Elasticsearch Performance Optimization

1) Suggestions for Elasticsearch Deployment

1.1) Use an SSD

1.2) Configure the JVM with Half of the Machine Memory, Up to 32 GB

1.3) Configure Dedicated Master Nodes for Large-scale Clusters to Avoid Split-brain

# Configure dedicated master nodes (conf/elasticsearch.yml):
node.master:true
node.data: false
node.ingest:false
# Configure data nodes (conf/elasticsearch.yml):
node.master:false
node.data:true
node.ingest:true
# Configure the minimum number of master nodes (conf/elasticsearch.yml):
discovery.zen.minimum_master_nodes: 2

1.4) Optimize the Linux Operating System

sed -i '/swap/s/^/#/' /etc/fstab
swapoff -a
echo "* - nofile 655360" >> /etc/security/limits.conf
echo "* - nproc 131072" >> /etc/security/limits.conf
echo "vm.max_map_count = 655360" >> /etc/sysctl.conf
sysctl -p

2) Suggestions for Index Performance Optimization

2.1) Set Appropriate Numbers of Index Shards and Replicas

curl -XPUT http://localhost:9200/fulltext001?pretty -H 'Content-Type: application/json'   -d '
{
"settings" : {
"refresh_interval": "30s",
"merge.policy.max_merged_segment": "1000mb",
"translog.durability": "async",
"translog.flush_threshold_size": "2gb",
"translog.sync_interval": "100s",
"index" : {
"number_of_shards" : "21",
"number_of_replicas" : "0"
}
}
}
'
curl -XPOST http://localhost:9200/fulltext001/doc/_mapping?pretty  -H 'Content-Type: application/json' -d '
{
"doc" : {
"_all" : {
"enabled" : false
},
"properties" : {
"content" : {
"type" : "text",
"analyzer":"ik_max_word"
},
"id" : {
"type" : "keyword"
}
}
}
}
'
curl -XPUT 'http://localhost:9200/fulltext001/doc/1?pretty' -H 'Content-Type: application/json' -d '
{
"id": "https://www.huxiu.com/article/215169.html",
"content": "“娃娃机,迷你KTV,VR体验馆,堪称商场三大标配‘神器’。”一家地处商业中心的大型综合体负责人告诉懂懂笔记,在过去的这几个月里,几乎所有的综合体都“标配”了这三种“设备”…"
}'
curl -XPUT "http://localhost:9200/fulltext001/_settings" -H 'Content-Type: application/json' -d'
{
"number_of_replicas": 1
}'

2.2) Use Batch Requests

curl -XPOST "http://localhost:9200/_bulk" -H 'Content-Type: application/json' -d'
{ "index" : { "_index" : "test", "_type" : "_doc", "_id" : "1" } }
{ "field1" : "value1" }
{ "delete" : { "_index" : "test", "_type" : "_doc", "_id" : "2" } }
{ "create" : { "_index" : "test", "_type" : "_doc", "_id" : "3" } }
{ "field1" : "value3" }
{ "update" : {"_id" : "1", "_type" : "_doc", "_index" : "test"} }
{ "doc" : {"field2" : "value2"} }
'

2.3) Send Data through Multiple Processes or Threads

2.4) Increase the Refresh Interval

curl -XPUT "http://localhost:9200/index" -H 'Content-Type: application/json' -d'
{
"settings" : {
"refresh_interval": "30s"
}
}'
curl -XPUT "http://localhost:9200/index/_settings" -H 'Content-Type: application/json' -d'
{ "refresh_interval": -1 }'
curl -XPUT "http://localhost:9200/index/_settings" -H 'Content-Type: application/json' -d'
{ "refresh_interval": "1s" }'

2.5) Design Reasonable Field Types for Mapping Configuration

curl -XPUT "http://localhost:9200/twitter/doc/1?pretty" -H 'Content-Type: application/json' -d'
{
"user": "kimchy",
"post_date": "2009-11-15T13:12:00",
"message": "Trying out Elasticsearch, so far so good?"
}'
curl -XGET "http://localhost:9200/twitter"
{
"twitter": {
"mappings": {
"doc": {
"properties": {
"message": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"post_date": {
"type": "date"
},
"user": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
}
},
"settings": {
"index": {
"number_of_shards": "5",
"number_of_replicas": "1",
}
}
}
}
curl -XPUT "http://localhost:9200/twitter?pretty" -H 'Content-Type: application/json' -d'
{
"settings" : {
"index" : {
"number_of_shards" : "20",
"number_of_replicas" : "0"
}
}
}'
curl -XPOST "http://localhost:9200/twitter/doc/_mapping?pretty" -H 'Content-Type: application/json' -d'
{
"doc" : {
"_all" : {
"enabled" : false
},
"properties" : {
"user" : {
"type" : "keyword"
},
"post_date" : {
"type" : "date"
},
"message" : {
"type" : "text",
"analyzer" : "cjk"
}
}
}
}'

3) Suggestions for Query Performance Optimization

3.1) Cache Filter-based Query Results and Shard-based Query Results

  • Normal Query
curl -XGET "http://localhost:9200/twitter/_search" -H 'Content-Type: application/json' -d'
{
"query": {
"match": {
"user": "kimchy"
}
}
}'
  • Filter-based Query
curl -XGET "http://localhost:9200/twitter/_search" -H 'Content-Type: application/json' -d'
{
"query": {
"bool": {
"filter": {
"match": {
"user": "kimchy"
}
}
}
}
}'
indices.requests.cache.size: 1%
curl -XGET "http://localhost:9200/_cat/nodes?h=name,query_cache.memory_size,request_cache.memory_size,fielddata.memory_size,segments.memory&v"

3.2) Use the _routing Field

shard_num = hash(_routing) % num_primary_shards
curl -XPUT "http://localhost:9200/my_index/my_type/1?routing=user1" -H 'Content-Type: application/json' -d'
{
"title": "This is a document",
"author": "user1"
}'
curl -XGET "http://localhost:9200/my_index/_search" -H 'Content-Type: application/json' -d'
{
"query": {
"match": {
"title": "document"
}
}
}'
{
"took": 2,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"skipped": 0,
"failed": 0
}
......
}
curl -XGET "http://localhost:9200/my_index/_search?routing=user1" -H 'Content-Type: application/json' -d'
{
"query": {
"match": {
"title": "document"
}
}
}'
{
"took": 1,
"timed_out": false,
"_shards": {
"total": 1,
"successful": 1,
"skipped": 0,
"failed": 0
}
......
}

3.3) Merge Read-only Indexes and Disable Historical Data Indexes

  • API for Merging Indexes
curl -XPOST "http://localhost:9200/abc20180923/_forcemerge"
  • API for Disabling Indexes:
curl -XPOST "http://localhost:9200/abc2017*/_close"

3.4) Configure Query Aggregation Nodes

# Configure query aggregation nodes (conf/elasticsearch.yml):
node.master: false
node.data: false
node.ingest:false

3.5) Set Fields and the Number of Records to Be Read

  • Sample Query Request
curl -XGET http://localhost:9200/fulltext001/_search?pretty  -H 'Content-Type: application/json' -d ' 
{
"from": 0,
"size": 10,
"_source": "id",
"query": {
"bool": {
"must": [
{"match": {"content":"虎嗅"}}
]
}
},
"sort": [
{
"id": {
"order": "asc"
}
}
]
}
'

3.6) Prevent Prefix Fuzzy Matching

3.7) Avoid Index Sparsity

3.8) Add and Upgrade Cluster Nodes

Original Source:

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Alibaba Cloud

Alibaba Cloud

Follow me to keep abreast with the latest technology news, industry insights, and developer trends. Alibaba Cloud website:https://www.alibabacloud.com