Optimizations with Full-Text Search in PostgreSQL

By Digoal.

With built-in GIN indexes, PostgreSQL supports full-text search and searching multi-value data types including arrays. GIN in the GIN index stands for Generalized Inverted Index. PostgreSQL can also conduct queries that do not contain any keyword and that simply scan, skipping the token above the main tree.

All of these kinds of queries can be cost-effective by being optimized to the data they are querying. The later type is very cost-effective, in particular, because through skipping a token that contains a relatively large amount of data, these queries can go much faster than queries than use tokens. Moreover, for this particular kind of function and query, PostgreSQL also uses a cost-based optimizer (CBO), which is a type of execution plan optimizer, which automatically choose the best index for the particular job.

In this article we will look at four different query examples to show how PostgreSQL, equipped with both GIN indexes and the cost-based optimizer (CBO), can use the most optimal plan to choose the best index. Last, we will summarize what we discovered about from our example queries.

Example 1. Full-Text Search: “Does Not Contain” Query

For this type of query, you’ll need to set up some things. First, create a test table:

Next, create a function to generate random strings.

Insert 1 million entries of test data:

After you do all of this, now create full-text index, which is a type of GIN index.

Now, it’s time for the query. The query this time does not contain a keyword. So you’re want to query one of the records:

Next, you’ll want to do a test. For the text, the database automatically chooses Full Table Scan, and does not use the GIN index. You may ask why wasn’t an index used? The reason for this is because the number of data records here containing the keyword amounts to relatively small number. Therefore, it is not cost-effective to use indexes for filtering in a query that does not contain a keyword, as a result the optimizer doesn’t choose this as an option. However, note that, if the query contains a keyword, it will be very cost-effective to use the GIN indexes. “Contains” and “Does Not Contain” are inversions of each other, and so are the associated costs.

Next, you can force disable the Full Table Scan, and allow the database to choose the indexes. From this, we can see that searches that use index are really slow. Most of the time we can trust the database to find the most efficient method to complete our queries. All of these can be calibrated as long as cost factors and environmental performance measuring are sufficiently accurate.

Example 2. Full-Text Search: “Does Not Contain” Query

For this example query, you will create some unevenly distributed data. The token for this data will contain a large number of repeated content, and you will filter them out with a “Does Not Contain” operator. So the question remains, will the index be used?

For this example, first generate some test data:

Given that the test search entries that do not contain ABC, the database automatically chooses indexed scanning and skips data blocks that do not need to be searched.

Next, you’ll want to force enable Full Table Scan. From this, we can see that the performance is truly no match for indexed scanning, which proves that PostgreSQL is a cost-based optimizer and can automatically choose the most optimal execution plan.

Example 3. Ordinary B-Tree Indexes: “Not Equal To” Searches

This example will be an ordinary search that uses a B-tree index. So the question for us is whether PostgreSQL supports the “Not Equal To” indexed search. The test method which will be used for this example will be similar to that of GIN test. In this test, I you will use both unevenly and evenly distributed data.

For a query that uses the “Does Not Contain” operator on unevenly distributed data, the number of records filtered out with indexes is a relatively small number. So far, as we have seen in this tutorial, “Does Not Contain” searches that use B-tree indexes are not supported at the kernel level. Granted, though, this can be achieved by skipping Branch nodes that do not need to be scanned with INDEX SKIP SCAN.

Next, you’re going to want to change the SQL write method to achieve an indexed search. As INDEX SKIP SCAN is not used in this case, we need a JOIN process. Otherwise, the resulting performance will not be very optimal.

In fact, in the SQL statement, we can also use <1 or >1 or null.

Next, PostgreSQL uses multi-core parallel computing to allow incredible performance improvements to Full Table Scan. Parallel scanning can remarkably improve the performance in case with a significantly large number of records.

Performance saw a marked improvement when we used a parallel query.

Example 4. Ordinary Partial B-Tree Indexes: “Not Equal To” Searches

For this last example query, we will look at “Not Equal To” queries. Generally, for fixed “Not Equal To” queries, we can use the partial index function of PostgreSQL.

Interestingly, it takes only 0.03 ms to run a “Not Equal To” search over 10 million entries of data using partial index.

Summary

From the examples above, we can learn several things:

  1. With built-in GIN indexes, PostgreSQL supports full-text search and searching multi-value data types including arrays.
  2. PostgreSQL uses a cost-based execution plan optimizer. It automatically chooses the best execution plan. When running a “Does Not Contain” search, PostgreSQL automatically chooses whether to use indexed scanning.
  3. With regards to B-tree indexes, technically it can be used to implement the “Not Equal To” searches (INDEX SKIP SCAN), but it is currently not supported at the kernel level. We can use indexed scanning by adjusting the current SQL write methods.
  4. PostgreSQL uses multi-core parallel computing to allow incredible performance improvements to Full Table Scan. Parallel scanning can remarkably improve the performance in case with a significantly large number of records.
  5. PostgreSQL supports partial index, which supports partitioned or partial indexes. Performance is outstanding for “Not Equal To” queries with fixed conditions.

Original Source

Follow me to keep abreast with the latest technology news, industry insights, and developer trends.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store