How to Migrate from ELK to Log Service

Overview

Introduction to Data Migration

Elasticsearch Data Migration Solution

Mechanism

  • Log Service uses Scroll API to pull data from Elasticsearch. Scroll API is able to efficiently pull large amounts of data from Elasticsearch without the cost of deep pagination.
  • Log Service creates a data migration task for each shard of each index in Elasticsearch, and submits these tasks to an internal process pool for execution. This improves the parallelism and throughput of data migration.

Features

  • Allows you to migrate all or some indexed documents from Elasticsearch to a specified Log Service project (CLI initializes logstores with the same names as those of the Elasticsearch indexes).
  • Allows you to customize filter conditions to migrate only the qualified documents to Log Service.
  • Allows you to customize the mapping relationships between indexes in Elasticsearch and logstores in Log Service.
  • Allows you to control the parallelism of data migration tasks by using the pool_size parameter.
  • Allows you to customize values of some log fields: time, __source__, and __topic__.
  • Allows you to use HTTP basic access authentication to migrate data from Elasticsearch.

Mapping Relationship

Example

Migration Command

aliyun log es_migration --hosts=<your_es> --project_name=<your_project> --indexes=filebeat-* --logstore_index_mappings='{"nginx-access": "filebeat-*"}' --time_reference=@timestamp

Query and Analysis

* | SELECT  date_trunc('day' ,  __time__)  as t, "nginx.access.response_code" AS status, COUNT(1) AS count GROUP BY status, t ORDER BY t ASC
* | SELECT ip_to_country("nginx.access.remote_ip") as country, count(1) as count GROUP BY country

Performance Tuning

The Speed of Reading Data from Elasticsearch

The Speed of Writing Data to Log Service

Performance Data

References

Original Source

--

--

--

Follow me to keep abreast with the latest technology news, industry insights, and developer trends. Alibaba Cloud website:https://www.alibabacloud.com

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Recording a Scenario in Mobile Applications

Building Event-driven Architectures with WSO2 Integration Platform — Part 1

Why can Agile be used in every industry- The universal truth explained

Google Slides API — a deep dive on its powerful image manipulation methods

Open CV :- Basic Image Processing Functions and Detection

iOS Universal Links Tips and Tricks

Using S3 Intermediate Storage for Efficient Horizontal Scalability

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Alibaba Cloud

Alibaba Cloud

Follow me to keep abreast with the latest technology news, industry insights, and developer trends. Alibaba Cloud website:https://www.alibabacloud.com

More from Medium

Produce and consume messages from a Kafka topic using docker

Schedule PostgreSQL Backups with IBM Cloud Code Engine

Historize elastic APM server data

Service Mesh, Istio and Why Do We Need It