How Can We Monitor “No Incoming Messages” Data Exceptions?

By Digoal

Stream computing can be used in a variety of scenarios, but, in e-commerce, often it can struggle given delivery and refund timeouts. But one question is how can streaming warnings made to be suited even in these scenarios? The answer that I will present in this blog is a solution that is done through a combination of timeout and scheduling operations that uses PostgreSQL.

PostgreSQL is advantageous to our needs in this tutorial because it uses flexible common table expression (CTE) syntax common to SQL languages and partial indexes that do not record unnecessary data. Also, it uses data manipulation language (DML) statements that can be returned and used together with CTE syntax to minimize interactive computing, and it takes multi-index bitmap scans, which combine multiple indexes for scanning, which allows us to combine the indexes of multiple fields for scanning when using an OR condition.

To sum things up, in this tutorial, I will show you how you can create an example solution for these problems through a combination of timeout and scheduling operations using PostgreSQL.

Create a Demo Design

To create a demo solution, first you need to design the structure of the monitored table. This table will record the timeout processing time of orders and refund events, as well as the number of timeout notifications, next notification time, and completion statuses. To do this, you can use the following command in PostgreSQL:

Next, you’ll want to create a partial index. This index is so that users are notified of only the unfinished tickets, which are what they mostly likely to be mainly concerned about. By using a partial index, we can reduce the index size and increase the speed. To do this, enter this code:

Next, as part of this, we need to obtain the related data from users, which would including the notification data and update the number of notifications and the next notification time. You can do this by running this code:

Then, execute the plan, as so:

Create Performance Indicators

Assuming that 0.1 billion data records are written, one question you may ask is how long does it take to process one million data records concurrently? Well, consider the following:

As you can see from the above code, the notification performance, for example, is generating a notification each time the processing of 10,000 records times out.

Following this, a small batch of data is retrieved, and the timeout is updated so that AutoVacuum can be introduced to collect garbage in real time. Consider the following:

So as you can see, the data is processed smoothly.


So to sum up, in this tutorial, I have introduced method by which you can solve the problem of timeout data monitoring in e-commerce scenarios that not only solves this issue but also delivers good performance.

Original Source

Follow me to keep abreast with the latest technology news, industry insights, and developer trends.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store