Optimizing Random Key Value Queries for Large Data Sets

Case Study: esProc

  1. Single-field key
  2. Multi-field key
  3. Multi-thread query
  4. Data appending

Single-Field Key

Create a Combination Table

Create an Index

Query Data

Multi-Field Key

Method 1 (Common Method)

Method 2 (merged primary key)

Multi-Thread Query

Create a Combination Table

Create an Index

Query Data

Data Appending

Append Ordered Key Values

Append Unordered Key Values

Append a Large Amount of Data

  1. Create the new-portion and old-portion combination table files. In this example, N is equal to 4, indicating that four combination tables are created:
  1. N is equal to 4. The new-portion and old-portion combination table files generated for the preceding four files are as follows:
  1. Append new data to the new-portion combination table files.
  1. After merging new data with the new-portion combination table files, clear the original new-portion combination tables and indexes and then re-create the indexes for the new-portion combination tables.
  1. Check the size of new data. If the size exceeds the value of B (unit: bytes), merge the new data with data in the old-portion combination tables.
  1. After merging the old-portion combination tables with the new-portion combination tables, clear the original old-portion combination tables and indexes and then re-create the indexes for the old-portion combination tables. Clear the new-portion combination tables that have been merged, and create empty combination tables.
  1. Use multiple threads to query the new-portion and old-portion combination table files.

--

--

--

Follow me to keep abreast with the latest technology news, industry insights, and developer trends. Alibaba Cloud website:https://www.alibabacloud.com

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

AURA Staking Stats — Period 1

Activate/Deactivate Game Objects with Activation Track in Unity

Internet Basics using Metaphors [Networks for Noobs-1]

Moon Mission Contest

Iorad Analytics provide insights about your learners.

Cubemap GLSL shader debug functions

Build a Simple Tracker Program Using Python and SQL — Video

Experiences in the low-code/no-code landscape

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Alibaba Cloud

Alibaba Cloud

Follow me to keep abreast with the latest technology news, industry insights, and developer trends. Alibaba Cloud website:https://www.alibabacloud.com

More from Medium

Kafka Producer/Consumer with Message key and offset

Big Data : Know everything here

Building A Structured Streaming Data Pipeline

How to Build Apache Ranger 2.1 in CentOS/Rocky/RHEL