Troubleshooting MaxCompute Jobs with Logview

Logview is a tool for checking and debugging a job after it is submitted to MaxCompute. With Logview, you can check the running status, details, and result of a job, and the progress of each phase. After the job is submitted to MaxCompute, a Logview link is generated. You can open the link in a browser to view the job information. However, how can you locate issues with so many parameters in Logview? How can you view the running status and resource usage of each instance or task through Logview? How can you analyze the execution plan and issues in query execution and locate long-tail tasks for the purpose of making data analysis efficient and cost-effective? Yun Hua, a product expert at Alibaba Cloud Computing Platform, provides answers to these questions.

The following content is based on Yun Hua’s presentation and PPT file.

This article consists of the following parts:

  1. What is Logview?
  2. Concepts and principles
  3. Descriptions of Logview parameters
  4. Troubleshooting with Logview

Many users may encounter problems when using MaxCompute but are unable to locate the problems and optimize their systems. This article describes Logview parameters and troubleshooting methods for you to address these problems.

What Is Logview?

Image for post
Image for post

Concepts and Principles

MaxCompute System Architecture

Image for post
Image for post

The MaxCompute computing cluster is built based on the Apsara system. The core modules of the Apsara system include Apsara Distributed File System, Job Scheduler, Apsara Name Service and Distributed Lock Synchronization System, Distributed System Performance Monitor, Kuafu, and Zhongkui. In the MaxCompute architecture, the scheduling and management of tasks between the management layer and Job Scheduler are the most important and complex.

MaxCompute Metadata Model

Image for post
Image for post

MaxCompute Job Processing Flow

Image for post
Image for post

Descriptions of Logview Parameters

MaxCompute Instance

  1. URL indicates the endpoint.
  2. Project specifies the name of the project.
  3. InstanceID consists of the time stamp and a random string. The time stamp is accurate to milliseconds and uses UTC time. That is, the time is inconsistent with StartTime, which specifies the time when the task was submitted on the PC.
  4. StartTime and EndTime indicate the task start time and end time, respectively.
  5. Latency indicates the amount of time consumed while the task was running.
  6. A task has the following four status options:
  7. Waiting: indicates that the task is being processed in MaxCompute and has not been submitted to Job Scheduler.
  8. Waiting List: n: indicates that the task has been submitted to Job Scheduler and is queued. n indicates the position of the task in the queue.
  9. Running: indicates that the task is running in Job Scheduler.
  10. Terminated: indicates that the task has completed.
    In the table, if Status of a task is not Terminated, you can double-click the status of the task to view detailed information in the Queue Detail and SubStatus History areas.
Image for post
Image for post

Queue Detail and SubStatus History

Image for post
Image for post

Two Types of Job Scheduler Jobs

Image for post
Image for post

As shown in the preceding figure, a Job Scheduler job is named in the format of Odps/______.

MaxCompute Task

Image for post
Image for post

Details of a Job Scheduler Job

Image for post
Image for post

Details of Job Scheduler Tasks

Image for post
Image for post

Details of Job Scheduler Instances

Image for post
Image for post

Summary of Job Scheduler Jobs

Image for post
Image for post

Tips: Metering and Billing Based on the Summary Information

Computing fee of a MapReduce task of the day = Total computing time of the day x RMB 0.46

Task computing time = Task running time (hours) x Number of cores called by the task

On the Summary tab page, you can directly obtain the CPU computing time without the need to calculate it using the formula. Use the following formula to calculate the fee for a SQL job:

Computing fee for a SQL job = Amount of input data for computing x SQL complexity x SQL price

Run the cost sql command to calculate the amount of input data and the SQL complexity. For more information about metering and billing, visit the Alibaba Cloud website.

Image for post
Image for post

Diagnosis

Image for post
Image for post

Import and Export of Logview Information

Image for post
Image for post

Troubleshooting with Logview

Image for post
Image for post

Task Errors

Image for post
Image for post

Slow Tasks

A symptom of a slow task is that the task is always queuing or the Job Scheduler job is always in the Waiting status on the console. Open Logview and check whether Status of the task is Waiting or Waiting List to locate where the task is queued. If Status is Waiting List, further check the queue length and the position of the task in the queue. You can also check the sub-status of the task on the SubStatus History tab page.

Image for post
Image for post

If you are unable to locate slow tasks, run show p to check the information about all instances or run top instance to check the running jobs. The job with the longest running time may be the task that is blocking the queue and causing other tasks to be waiting. To resolve the resource preemption issue, optimize the tasks as follows:

  1. If the billing method of your service is Pay-As-You-Go, you can run periodic and regular tasks in the Subscription resource group so that the resources are not preempted.
  2. If the billing method of your service is Subscription, we recommend that you tune the task execution time when running multiple tasks in parallel and run temporary tasks in the Pay-As-You-Go resource group. For more information about the reasons for job queuing, see the related articles in the Yunqi Community.
Image for post
Image for post

Large number of small files

A task may run slowly when a large number of small files exist. For example, the execution plan when a job is initiated is shown on the left of the following figure. The job contains two Map tasks, a Join task, and a Reduce task. After the Reduce task is completed, a Merge task is automatically added for small file merging.

Image for post
Image for post

Data files in Apsara Distributed File system are stored by block. The size of each block is 64 MB. A file whose size is smaller than 64 MB is a small file. Small files may be generated for the following reasons:

  1. A large number of small files are generated during the Reduce computing process.
  2. Small files are generated when Tunnel collects data.
  3. Temporary files are generated during job running and expired files are retained in the recycle bin.

Because of the excessive small files, data read in the Map stage is unevenly distributed, causing the long tail issue. In addition, excessive small files waste resources, exhaust disk space, and affect the overall execution performance. Consequently, small files generated in the computing process must be merged to improve both the performance and storage. MaxCompute has been optimized to automatically allocate a Job Scheduler merge task for merging small files. However, small files generated in many scenarios may not be merged effectively. Therefore, MaxCompute provides parameters that allow you to merge small files.

Image for post
Image for post

Run desc extended TableName to obtain the number of small files and determine whether your table contains excessive small files. If excessive small files exist, run the SQL statement shown in the following figure to merge the small files.

Image for post
Image for post

We recommend that you take the following measures to avoid generation of small files: To avoid small files from being generated in the Reduce process, run INSERT OVERWRITE to write data to the original table or delete the original table after writing data to a new table. To avoid small files from being generated when Tunnel collects data, call the Tunnel SDK. That is, when uploading data, upload the buffered data after its size reaches 64 MB. Do not upload data frequently. When uploading a partition table, set a lifecycle for the table so that expired data is automatically cleared. When there are a lot of temporary tables, set a lifecycle for all temporary tables so that expired temporary tables are automatically moved to the recycle bin. For more information about handling small files, visit the Alibaba Cloud website.

Long-tail Tasks Caused by Data Skew

Image for post
Image for post

Long-tail tasks caused by data skew may run slowly. The reason for data skew is uneven data distribution. The amount of data processed by some Job Scheduler instances is much greater than that processed by other instances, causing long-tail tasks. In MaxCompute Logview, hover your mouse cursor over the Long-Tails tab. The message “Latency is more than twice the average” is displayed, indicating that the long tail issue exists.

Image for post
Image for post

You can check whether a task is a long-tail task in Logview in two ways. One is to check the maximum latency of Job Scheduler instances. If the number in the parentheses after Long Tails is greater than 0, the long tail issue exists. Click the Long-Tails tab to view all long-tail instances and their details. The other way is to check the summary and diagnosis information of the Job Scheduler job. Analyze the summary to check the position of the long-tail task. If the value of max differs a lot from that of avg in the instance time field, the long tail issue exists. If the value of max differs a lot from that of avg in the input records field, data skew has occurred. In the diagnostic window, click Diagnose long tail and data skew to check the data skew and long tail issues, as well as the suggestions for improvement.

This section describes how to resolve different types of data skew issues.

  1. Data skew in the Join stage: The Join keys are unevenly distributed, resulting in a key containing a large amount of data. Because the data in the key is allocated to the same instance for processing, the instance requires a long period of time to process the data, causing the long tail issue. For example, data is skewed in the Join stage when a large table and a small table are joined or a large number of null values exist in the key. In this case, run MapJoin for optimization as it ensures better performance than Join. The principle is to perform the Join operation ahead of the schedule in the Map stage. Data in the small table is loaded into the memory of the program that performs the Join operation. In this way, the Join operation is accelerated. If null values exist, we recommend that you filter out the null values and add a random number to each key before the Join operation. This is equivalent to redistributing the keys.
  2. Data skew in the Group By stage: The Group By keys are unevenly distributed. To resolve this issue, set the anti-skew parameter or add a random number to each key to redistribute the keys.
  3. Data skew caused by Distinct: The Distinct clause is used to remove duplicates from fields. When Distinct is used, the Group By clause cannot be used in the Shuffle stage of the Map task to reduce the amount of transmitted data. All data is shuffled to the Reduce stage, causing the long tail issue when the key data is unevenly distributed. In this case, use the Group By clause with the Count function instead of the Distinct clause.
  4. Data skew caused by dynamic partitions: A large number of dynamic partitions may cause excessive small files. To sort small files, the system starts a Reduce task. If data is skewed when being written into dynamic partitions, the long tail issue occurs. In this case, we recommend that you do not use dynamic partitions and specify the corresponding partition when running Insert.
Image for post
Image for post

To learn more about Alibaba Cloud MaxCompute, visit https://www.alibabacloud.com/product/maxcompute

Follow me to keep abreast with the latest technology news, industry insights, and developer trends.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store