How to Migrate from ELK to Log Service

Overview

Introduction to Data Migration

Elasticsearch Data Migration Solution

Mechanism

  • Log Service uses Scroll API to pull data from Elasticsearch. Scroll API is able to efficiently pull large amounts of data from Elasticsearch without the cost of deep pagination.
  • Log Service creates a data migration task for each shard of each index in Elasticsearch, and submits these tasks to an internal process pool for execution. This improves the parallelism and throughput of data migration.

Features

  • Allows you to migrate all or some indexed documents from Elasticsearch to a specified Log Service project (CLI initializes logstores with the same names as those of the Elasticsearch indexes).
  • Allows you to customize filter conditions to migrate only the qualified documents to Log Service.
  • Allows you to customize the mapping relationships between indexes in Elasticsearch and logstores in Log Service.
  • Allows you to control the parallelism of data migration tasks by using the pool_size parameter.
  • Allows you to customize values of some log fields: time, __source__, and __topic__.
  • Allows you to use HTTP basic access authentication to migrate data from Elasticsearch.

Mapping Relationship

Example

Migration Command

aliyun log es_migration --hosts=<your_es> --project_name=<your_project> --indexes=filebeat-* --logstore_index_mappings='{"nginx-access": "filebeat-*"}' --time_reference=@timestamp

Query and Analysis

* | SELECT  date_trunc('day' ,  __time__)  as t, "nginx.access.response_code" AS status, COUNT(1) AS count GROUP BY status, t ORDER BY t ASC
* | SELECT ip_to_country("nginx.access.remote_ip") as country, count(1) as count GROUP BY country

Performance Tuning

The Speed of Reading Data from Elasticsearch

The Speed of Writing Data to Log Service

Performance Data

References

Original Source

--

--

--

Follow me to keep abreast with the latest technology news, industry insights, and developer trends. Alibaba Cloud website:https://www.alibabacloud.com

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Table often person meeting other.

Generate PDF with WkhtmlToPdf at Spring boot project in Ubuntu.

Software Engineering is much more than just building APIs

HarperDB Cloud: What Else Could I Ask For?!

WeChat for Dummies | Official Account BASIC Functions Part 1

The Big Red Book : Database Systems— don’t reinvent the wheel in database systems

Setting up Sublime Text 3 to use virtual environment packages for autocompletion using Anaconda

Is Learning a Programming Language like Learning a Foreign Language?

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Alibaba Cloud

Alibaba Cloud

Follow me to keep abreast with the latest technology news, industry insights, and developer trends. Alibaba Cloud website:https://www.alibabacloud.com

More from Medium

How to Connect Elastic Sink Connector with Kafka

Using Search Template — ElasticSearch

How to deploy Scalar DB Server on Kubernetes

Deploying Open Source Observability Stack in AWS — Part — II