How to Migrate Data from an Amazon ES Domain to an Alibaba Cloud Elasticsearch Cluster

Introduction to Migration

Terms

Migration Plan

Prerequisites

Elasticsearch Service

  • Create an Amazon ES 5.5.2 domain in the Singapore region.
  • Create an Alibaba Cloud Elasticsearch v5.5.3 cluster in the China (Hangzhou) region.
  • The sample index named “movies”.

Prerequisites for Creating Manual Snapshots in an Amazon ES Domain

Create an S3 Bucket

arn:aws:s3:::eric-es-index-backups

Create an IAM Role

{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "",
"Effect": "Allow",
"Principal": {
"Service": "es.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
}

Create an IAM Policy

{
"Version": "2012-10-17",
"Statement": [
{
"Action": [
"s3:ListBucket"
],
"Effect": "Allow",
"Resource": [
"arn:aws:s3:::eric-es-index-backups"
]
},
{
"Action": [
"s3:GetObject",
"s3:PutObject",
"s3:DeleteObject"
],
"Effect": "Allow",
"Resource": [
"arn:aws:s3:::eric-es-index-backups/*"
]
}
]
}

Register a Manual Snapshot Repository

1) Modify the Sample Python Client File

2) Install Amazon Web Services Library boto-2.48.0

# wget https://pypi.python.org/packages/66/e7/fe1db6a5ed53831b53b8a6695a8f134a58833cadb5f2740802bc3730ac15/boto-2.48.0.tar.gz#md5=ce4589dd9c1d7f5d347363223ae1b970 
# tar zxvf boto-2.48.0.tar.gz
# cd boto-2.48.0
# python setup.py install

3) Run the Python Client to Register the Snapshot Repository

# pyth
on snapshot.py
GET _snapshot

Create the First Snapshot and Restore Data from the Snapshot

1) Create a Snapshot in the Amazon ES Domain

PUT _snapshot/eric-snapshot-repository/snapshot_movies_1
{
"indexes": "movies"
}
GET _snapshot/ eric-snapshot-repository/snapshot_movies_1

2) Transfer the Created Snapshot From S3 Bucket to OSS Bucket

3) Restore Data From the Snapshot to Alibaba Cloud Elasticsearch Cluster

PUT _snapshot/eric-snapshot-repository
{
"type": "oss",
"settings": {
"endpoint": "http://oss-cn-hangzhou-internal.aliyuncs.com",
"access_key_id": "Put your AccessKey id here.",
"secret_access_key": "Put your secret AccessKey here.",
"bucket": "eric-oss-aws-es-snapshot-s3",
"compress": true
}
}
GET _snapshot/eric-snapshot-repository/snapshot_movies_1
"start_time_in_millis": 1519786844591
"end_time_in_millis": 1519786846236

4) Restore Data From the Snapshot

POST _snapshot/eric-snapshot-repository/snapshot_movies_1/_restore
{
"indexes": "movies"
}
GET movies/_recovery

Create the Final Snapshot and Restore Data from the Snapshot

1) Insert Data to the Movies Index in Amazon ES Domain

2) Create a Snapshot

3) Transfer the Snapshot From S3 Bucket to OSS Bucket

4) Restore Data from the Snapshot

POST /movies/_close
GET movies/_stats
POST _snapshot/eric-snapshot-repository/snapshot_movies_2/_restore
{
"indexes": "movies"
}
POST /movies/_open

Summary

References

Original Source:

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Alibaba Cloud

Alibaba Cloud

Follow me to keep abreast with the latest technology news, industry insights, and developer trends. Alibaba Cloud website:https://www.alibabacloud.com