[FEATURE] Improve filtering condition push down in Flint data source

[dai-chen]

Is your feature request related to a problem?

Currently there are only limited support for push down optimization in Flint data source. For example, value set column in skipping index is actually array and should use ARRAY_CONTAINS in filtering condition. Because of no push down support, index query has to use = instead.

What solution would you like?

1. Support push down for more operators in filtering condition
2. [TBD] Support array field by field metadata

[dai-chen]

This requirement is part of integrating OpenSearch as a data source in Spark, which includes:

1. OpenSearch data types
2. OpenSearch scalar and aggregate functions
3. OpenSearch DSL pushdown capabilities

One approach is to leverage the OpenSearch SQL plugin by integrating OpenSearch as a JDBC connector via the OpenSearch SQL JDBC driver. If this proves to be a viable long-term solution, future enhancements, such as high performance communication via Apache Arrow, can be encapsulated and managed within the JDBC driver, ensuring a centralized and efficient implementation.