Open, Universal, and High-Performance: Time-series Data Storage for Log Service Empowers Comprehensive Enterprise-level Monitoring Solutions

Time-series Data Is Everywhere

  • The Apple Watch monitors the wearer’s heart rate information to help detect serious heart diseases early.
  • The State Grid analyzes the electricity consumption curve of each community and household to detect electricity leakage and theft.
  • E-commerce companies quickly detect various abnormalities by monitoring changing trends in key processes such as order placements, transactions, returns, and reviews.
  • Gaming platforms analyze user behavior patterns, such as their actions and locations, to determine whether cheating tools are being used.

What Kind of Time-series Storage Is Needed?

Recent years have seen the development of a number of time-series storage engines that support time-series analysis and monitoring in various scenarios. These engines include TimescaleDB, CrateDB, InfluxDB, OpenTSDB, and Prometheus. Each of these engines has its own ecosystem and application scenarios. For example, TimescaleDB is based on PostgreSQL, and therefore users familiar with PostgreSQL have an advantage in learning and beginning to use TimescaleDB. InfluxDB has a rich ecosystem, known as the TICK Stack, that includes Telegraf, InfluxDB, Chronograf, and Kapacitor. Prometheus has become the de facto standard for monitoring in Kubernetes thanks to its ease of use in cloud-native scenarios and its convenient, flexible query language, PromQL.

  1. Openness: Generally, multiple departments in a company perform different types of analysis and monitoring on the time-series data in different systems. Time-series storage must be open enough to support various methods of data access and downstream consumption.
  2. Low Cost: Time-series storage requires low resource and manual O&M costs. In accordance with Moore’s law, the cost per unit of resources is constantly decreasing, but personnel cost per unit is increasing every year. Controlling the labor cost of O&M for time-series storage is key to reducing overall cost.
  3. Intelligence: Static rules alone are not always sufficient to find abnormalities in monitored objects, in particular when a large number of objects are being monitored. Intelligent algorithms are required on the upper layer of time-series storage systems to improve monitoring accuracy.

Release of Time-series Data Storage for Log Service (SLS)

The Alibaba Cloud Log Service (SLS) log storage engine was released in 2016. At present, dozens of petabytes of log data from Alibaba and other enterprises are added to SLS each day. Time-series data and data used to compute time-series metrics make up a large part of the log data in SLS. The new time-series storage feature for SLS provides users with comprehensive data access, cleansing, processing, extraction, storage, visualization, monitoring, and problem analysis throughout the entire DevOps lifecycle. Together with SLS, this feature can solve data storage-related problems on all kinds of systems.

Features

  • High performance: The separation of computing and storage in SLS ensures optimal use of cluster capabilities. The end-to-end speed increases significantly when a large amount of data is processed.
  • Zero O&M: Time-series storage for SLS is provided as a service. Users do not need to operate and maintain instances themselves, and three replicas of all data are stored, making it unnecessary to worry about data reliability.
  • Open-source-friendliness: Time-series storage for SLS has native support for writing and querying data in Prometheus. It supports SQL-92 analysis methods and can natively connect to visualization solutions such as Grafana.
  • **Intelligence**: SLS provides a variety of AIOps algorithms with which you can build an intelligent alerting and diagnosis platform suited to your company. These time-series algorithms include multi-period estimation, prediction, error detection, and classification.

Typical Scenarios

Application and Service Monitoring

Cloud-Native Monitoring

Access Log Analysis

Original Source:

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store