[BUG] Value set cannot handle high cardinality column

[dai-chen]

What is the bug?

In scenarios with high cardinality columns like clientIp in HTTP logs, the resulting value set can become exceedingly large. This leads to the creation of huge array fields within a single OpenSearch document.

How can one reproduce the bug?

CREATE SKIPPING INDEX ON http_logs
(status VALUE_SET)
WITH (
  auto_refresh=true
);

What is the expected behavior?

Either raise exception beforehand or provide other data structure like BloomFilter in #193

[dai-chen]

A workaround is to provide a max size for value set. For larger value set, an empty set is stored. Ref: https://clickhouse.com/docs/en/optimize/skipping-indexes#set

CREATE SKIPPING INDEX ON ...
(clientIp VALUE_SET(100))

Need to figure out:

1. How to express this with Spark COLLECT_SET() function (currently used by skipping index build)
2. How to ignore empty value set in OpenSearch query