Search refactoring to better utilize index and improve capabilities

[psrok1]

Your checklist for this pull request

• I've read the contributing guideline.
• I've tested my changes by building and running the project, and testing changed functionality (if applicable)
• I've added fixed automated tests for my change (if applicable, optional)
• I've updated documentation to reflect my change (if applicable)

What is the current behaviour?

Current search engine was modified incrementally and few things, especially related with escaping are horrible mess. In addition I can't easily optimize things in array/jsonb columns as I need to use specific operators in query to utilize a GIN index: https://www.postgresql.org/docs/current/gin-builtin-opclasses.html#GIN-BUILTIN-OPCLASSES-TABLE

What is the new behaviour?

I have heavily reworked search engine and here is a list of functional changes:

• Exact queries against JSON columns (without wildcards) are using @? operator, so they can utilize GIN index making them really fast:

(https://www.postgresql.org/docs/current/functions-json.html)

This type of operator is able to utilize GIN index, but we need to build our predicate using jsonpath grammar.

For example query cfg.cncs*.host:"example.com" is converted to the following SQL query:

SELECT * FROM object WHERE (
    cfg @? '$.cfg.cncs[*] ? (@ == "example.com")'
)

Unfortunately queries with wildcards against JSON columns can't be optimized easily so work still in progress.

• Exact queries against file names are also faster:

alt_names column is queried using @> operator, so it can utilize GIN index. After adding collection of alternative upload names (#482), both file_name column and alt_names array are checked by the query making it unexpectedly slow.

• Types of objects can be mixed within the same query

#661 changed inheritance model from join-based to single-table-based, so we no longer need to join on all types of objects while making queries involving multiple types. This might be useful for parent/child queries with OR operator, but main reason was to remove code that was checking that.

• Both inclusive and exclusive ranges are allowed for date-time columns

That one was really annoying for me as a user, so I just treat exclusive ranges as inclusive. It doesn't make any huge difference if we query for upload_time:<5d or upload_time:<=5d

• Range boundaries are automatically sorted

Fixed another annoying thing, especially in dates:

upload_time:[1d TO 5d] will return you nothing because it means FROM NOW-1 day TO NOW-5 day and the left value is greater than the right side...

• More fixed corner cases of escaping

I feel we're finally doing it right...

• Query values are tokenized using tokenize_string which is a heart of a new parser.
• All string operations that are transforming value from Lucene pattern syntax to SQL LIKE-specifc or jsonpath-specific patterns are contained in mwdb.core.search.parse_helpers module. Most important methods are:
  • transform_for_eq_statement: trivia, just unescaping characters for __eq__ operator
  • transform_for_like_statement converting unescaped Lucene wildcards to SQL wildcards and then escaping all backslashes and SQL wildcard characters.
  • transform_for_quoted_like_statement made for LIKE statement against JSON typecasted to String. String inside JSON objects are quoted and additionally escaped which needs to be considered while making a pattern
  • transform_for_config_* which is additionally transforming value using encode("unicode-escape"). PostgreSQL is not accepting null-bytes in strings, so if we have high probability of lazily-encoded binary data as a string, we're using additional encoding which needs to be included in pattern
• I bumped luqum and I'm using new TreeVisitor. I decided to use visitor only for building condition. Values are parsed depending on what is expected by specific type of a search field.

Breaking changes

Some of them needs discussion because they can be avoided with some additional code

• size:">=5kB" no longer represents a range. The correct forms are:

  size:>="5kB"
  size:"5kB"
  

  That's because latest luqum added support for OpenRange operators, so now >=,>,<,<= is a thing that is no longer a part of Term: Add support for unbounded ranges jurismarches/luqum#91. Of course we can fix that and leave our parsing of >= in place, but do we really need to?

• I tried to unify exception messages, so some of them been changed (https://github.com/CERT-Polska/mwdb-core/pull/906/files#diff-03ca869f84201fd8bfc70b98b0e4fc0cb4c0bb1f91c375f8dfbf0af31fa8782c). I don't think anyone relied on that.

• jsonb @? jsonpath is querying values a bit differently than our previous code:

  • We're not casting exact queries to strings, so if value looks like a number or boolean, we're querying it both in quoted and unquoted way: https://github.com/CERT-Polska/mwdb-core/pull/906/files#diff-defa34ff47121dc90fa86b14135c87d7e621187fe5222d1bf6ae33d4be7e5c9cR327
  • Arrays nested in arrays are also considered while querying an array for a value... which was somehow checked in one of the tests. I fixed that test because I don't believe someone is relying on that behavior.

Test plan

Already included tests are really good at testing corner cases

[msm-cert]

I've managed to review ~25% of the changes for now.