Alibaba Cloud Elasticsearch: What’s New and Latest Features

Image for post
Image for post

By Qian Yuxin, Product Manager in the Search and Recommendation Division of the Alibaba Group

Released by ELK Geek

What Is Elasticsearch?

Image for post
Image for post

Alibaba Cloud Elasticsearch provides a fully managed Elasticsearch service and is compatible with the open-source version. It optimizes kernel performance and provides commercial features (formerly X-Pack) that are out-of-the-box (OFTB), highly available, elastically scalable, and billed in pay-as-you-go mode. In the following figure, we compare Alibaba Cloud Elasticsearch and other vendors’ Elasticsearch products in terms of reliability, security, and system hosting.

In terms of reliability, Alibaba Cloud Elasticsearch has data durability of 99.9% and regularly backs up data to Object Storage Service (OSS) to facilitate data recovery. In addition, the active zone-redundancy solution provides a powerful disaster recovery capability. Alibaba Cloud Elasticsearch has also significantly improved upon the open-source versions. For kernel performance optimization, Alibaba Cloud Elasticsearch not only separates storage from computing but also optimizes Elastic Compute Service (ECS) instances. For the index build service, Alibaba Cloud Elasticsearch accelerates high-concurrency data writing, which allows mutual influence between data write and query. Alibaba Cloud Elasticsearch uses the index build service to build offline indexes, splits native indexes into smaller shards, and merges them with online indexes. This avoids I/O overhead of the online cluster and ensures query stability during high-concurrency data writing. In terms of smart operation and maintenance (O&M), Alibaba Cloud Elasticsearch provides the EU smart O&M system for O&M, monitoring, and intelligent analysis on clusters, helping users see the health status of clusters. It also provides warnings and suggestions for improvement. In addition, Alibaba Cloud Elasticsearch has been integrated with the Natural Language Processing (NLP) analyzer provided by Alibaba DAMO Academy for better business analysis and retrieval. The X-Pack service is integrated into Elasticsearch and Kibana to provide commercial plug-ins. Previously, users needed to pay for these commercial plug-in packages. However, Alibaba Cloud Elasticsearch provides many services through X-Pack, such as authentication, permission management, report visualization, and machine learning. In general, compared with other vendors’ Elasticsearch solutions and user-created Elasticsearch solutions, Alibaba Cloud Elasticsearch offers more powerful product capabilities at better cost performance.

Image for post
Image for post

Due to these capabilities, Alibaba Cloud Elasticsearch is suitable for various scenarios such as IT O&M, information retrieval, and log analysis. In terms of IT O&M, Alibaba Cloud Elasticsearch supports metric monitoring and network log analysis. In terms of information retrieval, it supports app retrieval, database acceleration, and aggregate searches. In terms of log analysis, it is applicable to web log analysis, risk control, risk auditing, risk analysis, user behavior analysis, user profiling, business intelligence (BI) analysis, and ad hoc data analysis. Alibaba Cloud Elasticsearch is available as a subscription and in pay-as-you-go mode.

Output Modes

Image for post
Image for post

Architecture

Image for post
Image for post

Alibaba Cloud Elasticsearch is deployed in the CIDR blocks of Elastic Compute Service (ECS) instances, which is equivalent to purchasing a large number of ECS instances. You may purchase many Elasticsearch clusters, each cluster contains many nodes, and each node is an ECS instance. All ECS instances are deployed in the Virtual Private Cloud (VPC) network of the system and support zone-disaster recovery across zones. This means services are easily deployed in different zones in a region. By configuring IP address mappings between Alibaba Cloud VPCs and your VPCs, you can deploy nodes of each cluster in different zones.

For disaster recovery, nodes regularly backup snapshots to OSS. If a data fault occurs, it’s easy and quick to restore data from OSS. Ultra-disks, solid-state drives (SSDs), and on-premises disks are used for overall data storage. Alibaba Cloud Elasticsearch has recently improved its kernel to support storage and computing separation. An Elasticsearch index needs to be sharded for convenient storage. In order to improve query efficiency, each shard has multiple replicas that improve the speed by expanding the storage space. However, this causes a large amount of redundant data which results in high storage costs. In addition, to improve query efficiency, more memory overhead is incurred when you write data which results in slow write speed. In this situation, Alibaba Cloud Elasticsearch optimizes the kernel by separating storage from computing. This allows it to shard and map multiple replicas of data to the same physical media. Compared with native Elasticsearch, Alibaba Cloud Elasticsearch reduces storage costs by at least 50%, improves real-time data writing performance by 70%, and improves replica and shard change performance by 99%.

Regions

Image for post
Image for post

Audit Solution for Persistent Financial Databases

Image for post
Image for post

Example — Log Analysis

Image for post
Image for post

How Does Elasticsearch Process Logs?

Image for post
Image for post
  • In terms of centralized collection and storage of log data, Alibaba Cloud Elasticsearch collects regular log data, including log files, log system data, and network congestion logs. By collecting data and migrating offline Hadoop data, Alibaba Cloud Elasticsearch quickly gathers log data, stores the data in Elasticsearch, and builds indexes.
  • In terms of log search capabilities, Alibaba Cloud Elasticsearch supports full-text search, metadata search, metric and tag search, and location-based search.
  • In terms of aggregate analysis and visualization, after data is aggregated in Alibaba Cloud Elasticsearch, you can implement an aggregate analysis by using aggregate functions such as sum, average, min, and max. Also, implement machine learning analysis through X-Pack and visualize online data through Kibana. In Alibaba Cloud Elasticsearch, you may directly implement configurations and create a visualization panel in the Kibana console.
  • In terms of security and role management, Alibaba Cloud Elasticsearch provides role-based access control (RBAC) user permissions and Transport Layer Security (TLS) and Secure Socket Layer (SSL) protocols. It also implements real-time monitoring and alerting. In addition, the X-Pack features of Alibaba Cloud Elasticsearch provide services such as automatic data reports and triggered reports, helping you better manage and query data.
  • In terms of scalability, Alibaba Cloud Elasticsearch supports elastic scaling. Nodes in an Elasticsearch cluster use peer-to-peer connections, supporting quick replication and elastic scaling to manage data at different scales.

Architecture of Alibaba Cloud Elasticsearch Ecosystem

Image for post
Image for post

Summary

Original Source:

Follow me to keep abreast with the latest technology news, industry insights, and developer trends.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store