Add skipping index benchmark test

[dai-chen]

Description

This PR introduces a benchmark test focusing on the skipping index data structure. It is positioned between end-to-end benchmarking and low-level microbenchmarking. It aims to provide insights into the read and write performance of different skipping data structure using OpenSearch as index store.

The reasons behind this addition instead of E2E or microbenchmarking are:

• End-to-End Benchmarking:

  • This type of benchmarking necessitates splitting test data into files and persisting them on disk. While feasible within the current test framework, this process is exceptionally slow for local testing. Future plans might include the addition of another benchmark test class that executes by submitting to a Spark cluster.
  • Additionally, it requires the presence of SQL plugin and UI in OpenSearch Dashboards, alongside the EMR-S Spark environment.

• Microbenchmarking:

  • The implementation of Flint BloomFilter resides in flint-core, making it straightforward to incorporate into microbenchmarking.
  • However, other skipping data structures are implemented within Spark aggregate functions, posing challenges in isolating and testing the algorithms independently.

TODO

1. Check if any cache or other factors in Spark and OpenSearch may impact the test result
2. Include more metrics such as resource consumption, OS index size etc
3. Analyze test results and update user manual with recommendation

Test Cases

Please find details in the Javadoc on FlintSkippingIndexBenchmark. The test is based on Spark benchmark test framework and will be triggered manually.

Test Results

https://github.com/dai-chen/opensearch-spark/blob/add-skipping-index-benchmark-rebased/docs/benchmark-skipping-index.txt

Test Data (Generated)

The schema and size of skipping index written in the test:

curl "localhost:64541/_cat/indices?v"
health status index                                                                                 uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   flint_benchmark_bloom_filter_cardinality_64_adaptive_false_fpp_0.03_num_items_1000000 krhbKOALQB2Ezxn-9PRCZQ   1   1          1            0    902.3kb        902.3kb
yellow open   flint_benchmark_bloom_filter_cardinality_64_adaptive_true_fpp_0.03_num_candidates_10  qyYpKbY7R0C5plNn16QIXQ   1   1          1            0      4.8kb          4.8kb
yellow open   flint_benchmark_value_set_cardinality_64_max_size_100                                 wdhxAEv4QNSep0YXn3d2Uw   1   1          2            0      7.2kb          7.2kb
yellow open   flint_benchmark_value_set_cardinality_64_max_size_2147483647                          FcbPKJa2TLSKVmjSIOWlMA   1   1          2            0      7.2kb          7.2kb
yellow open   flint_benchmark_min_max_cardinality_64_default                                        WU8cLd6SSSKuppOSRvAREQ   1   1          2            0      6.7kb          6.7kb
yellow open   flint_benchmark_partition_cardinality_1_default                                       55R97W4zR3Kzt55BpgTzAA   1   1          1            0      3.1kb          3.1kb
yellow open   flint_benchmark_bloom_filter_cardinality_64_adaptive_true_fpp_0.03_num_candidates_5   e4sLSBtcROCJ4yIsd406gw   1   1          1            0      4.8kb          4.8kb
yellow open   flint_benchmark_bloom_filter_cardinality_64_adaptive_true_fpp_0.03_num_candidates_15  f9aqpip1TdeIjhT3imJQww   1   1          1            0      4.8kb          4.8kb
yellow open   flint_benchmark_bloom_filter_cardinality_64_adaptive_false_fpp_0.03_num_items_64      ATGUO7OnRi-f570mvWOhEQ   1   1          1            0      3.2kb          3.2kb

curl "localhost:64541/flint_benchmark_*/_mapping?pretty"
{
  "flint_benchmark_bloom_filter_cardinality_64_adaptive_true_fpp_0.03_num_candidates_15" : {
    "mappings" : {
      "properties" : {
        "value" : {
          "type" : "binary",
          "doc_values" : true
        }
      }
    }
  },
  "flint_benchmark_bloom_filter_cardinality_64_adaptive_false_fpp_0.03_num_items_1000000" : {
    "mappings" : {
      "properties" : {
        "value" : {
          "type" : "binary",
          "doc_values" : true
        }
      }
    }
  },
  "flint_benchmark_bloom_filter_cardinality_64_adaptive_true_fpp_0.03_num_candidates_5" : {
    "mappings" : {
      "properties" : {
        "value" : {
          "type" : "binary",
          "doc_values" : true
        }
      }
    }
  },
  "flint_benchmark_value_set_cardinality_64_max_size_2147483647" : {
    "mappings" : {
      "properties" : {
        "value" : {
          "type" : "long"
        }
      }
    }
  },
  "flint_benchmark_bloom_filter_cardinality_64_adaptive_false_fpp_0.03_num_items_64" : {
    "mappings" : {
      "properties" : {
        "value" : {
          "type" : "binary",
          "doc_values" : true
        }
      }
    }
  },
  "flint_benchmark_value_set_cardinality_64_max_size_100" : {
    "mappings" : {
      "properties" : {
        "value" : {
          "type" : "long"
        }
      }
    }
  },
  "flint_benchmark_partition_cardinality_1_default" : {
    "mappings" : {
      "properties" : {
        "value" : {
          "type" : "long"
        }
      }
    }
  },
  "flint_benchmark_min_max_cardinality_64_default" : {
    "mappings" : {
      "properties" : {
        "MinMax_value_0" : {
          "type" : "long"
        },
        "MinMax_value_1" : {
          "type" : "long"
        }
      }
    }
  },
  "flint_benchmark_bloom_filter_cardinality_64_adaptive_true_fpp_0.03_num_candidates_10" : {
    "mappings" : {
      "properties" : {
        "value" : {
          "type" : "binary",
          "doc_values" : true
        }
      }
    }
  }
}

Skipping index data written:

curl "localhost:64541/flint_benchmark_*/_search?pretty"
{
  "took" : 105,
  "timed_out" : false,
  "_shards" : {
    "total" : 9,
    "successful" : 9,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 12,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "flint_benchmark_bloom_filter_cardinality_64_adaptive_false_fpp_0.03_num_items_1000000",
        "_id" : "zItIYo4BM5f06XztvDKX",
        "_score" : 1.0,
        "_source" : {
          "value" : "AAAAA..."
        }
      },
      {
        "_index" : "flint_benchmark_bloom_filter_cardinality_64_adaptive_false_fpp_0.03_num_items_64",
        "_id" : "zYtJYo4BM5f06Xzt4TL2",
        "_score" : 1.0,
        "_source" : {
          "value" : "AAAAAQAAAAUAAAAIXfxLWNX6W3GBF/TxtjFPE7pdpOfrUCWgDKuuovFceYEu3SPLCdFRGprA9cOk07kApVD2oIHCron49EOAANoFZw=="
        }
      },
      {
        "_index" : "flint_benchmark_bloom_filter_cardinality_64_adaptive_true_fpp_0.03_num_candidates_10",
        "_id" : "zotJYo4BM5f06Xzt-zKD",
        "_score" : 1.0,
        "_source" : {
          "value" : "AAAAAQAAAAUAAAB1AAAAAAAAIAAAAAAAAAAAAAAAAAAAABAAAAAAQAAAAAAIAAAAARAAAAAAAAAEQQEAAADEBAAAAAASAgAAAEAMAAAgAAAAAAAAAQBAAAABgAACAAIAAAAAAAAAAAAAAAAAAAEAAAAAABAAAAAgAIAAAACABIBAAAAAEEACAAAAAAAAUCEAIIAAAAAAEAAIAAQBAAAEAAAAAAACAABAADAAAAAoAABAAAAAAJAAAAAAAAAAAAACAAAAAAAAAAAAAAACAAAAAAAAAAAAAACAgBgAAEEAAIAAAigAAAACEAAAAAAAAAIAAAAQAAABAAAAAAAAAAAAACAAAACACAAAAAAAAAAAIAABAAAAAAAABAAUAAgIABAAAAAAAQAAAkBAAAAAABAgBIDAAACgQAgAAAABAQAAAAAQAAACAACAAAAAAAAIQAAAAAAAAAAAAQAAAAAEAAAFwAAICgAAECQABAAEAKAEAAAAAAADAAAAAAECABAAGAAjgEAAAAACAAAAQAQAAAQgQAAgAAAAAAAAAIAAABAAAAIAAAAAAABAAIAAAgAAAACAAAAAAAAQAAKAAAAAAAAAAAAAQBAAAAAIIAAAYAAAAAAAABACAAAAAAAAAAAAUAAAAAAgQQABAABAAAABCACAAAAAAEIAAEAIAAAAgCAABAAAAAgAAACAAAAAEAAIgACAgAAAAAAAAAABAAAAAAAAAAACCAAABABAAABACAAAQAAAAAAAAIAAABAABQAAACAAAgAAAAAAAAAAmCAAoAAAAAAAAAAIEEAAAAAAAABAABAAQACAAAAAACAAAAAAIAAAAgAAIAAAICABAgAAAAABAIAAAABAAAAAAAAAAAwQAA4AQAAAAgAgAAAAAAAAgACAAAAAIIAQAAAAAAAAAQAAAAAAAAAQAAAABAAJACQQAIAQAAAAAAAAAAAAAAAAAAAgAAAAgAgACAAAAAIBAEAAQAAAAAAgAAAAABAAQAAAAAAAAEAAAACAAAAAAAAAAAAAAAAAAAEAEAEAAQkAAAAAAAAAIAAAAAAAAAAAAAgAAAAAAAAAAAAAAAACAAAAAAAAIgAAAAEEAAAAAQAAAAQgAAAAAAAAAASQAAABAAAAAAAAAAAAAAAgABAAAQABAAAAIAIAAAIIAAAAQAAAAAAAQAgAAAAIAIAAgABAAAQBAAAACAAAAAACIQAAAAAAAAAAAAAAAEQAAAAAEIAAAAQAAAAQEACAIAAAAEAAAAAAIEABAAEA"
        }
      },
      {
        "_index" : "flint_benchmark_bloom_filter_cardinality_64_adaptive_true_fpp_0.03_num_candidates_15",
        "_id" : "0ItKYo4BM5f06XztJTJf",
        "_score" : 1.0,
        "_source" : {
          "value" : "AAAAAQAAAAUAAAB1AAAAAAAAIAAAAAAAAAAAAAAAAAAAABAAAAAAQAAAAAAIAAAAARAAAAAAAAAEQQEAAADEBAAAAAASAgAAAEAMAAAgAAAAAAAAAQBAAAABgAACAAIAAAAAAAAAAAAAAAAAAAEAAAAAABAAAAAgAIAAAACABIBAAAAAEEACAAAAAAAAUCEAIIAAAAAAEAAIAAQBAAAEAAAAAAACAABAADAAAAAoAABAAAAAAJAAAAAAAAAAAAACAAAAAAAAAAAAAAACAAAAAAAAAAAAAACAgBgAAEEAAIAAAigAAAACEAAAAAAAAAIAAAAQAAABAAAAAAAAAAAAACAAAACACAAAAAAAAAAAIAABAAAAAAAABAAUAAgIABAAAAAAAQAAAkBAAAAAABAgBIDAAACgQAgAAAABAQAAAAAQAAACAACAAAAAAAAIQAAAAAAAAAAAAQAAAAAEAAAFwAAICgAAECQABAAEAKAEAAAAAAADAAAAAAECABAAGAAjgEAAAAACAAAAQAQAAAQgQAAgAAAAAAAAAIAAABAAAAIAAAAAAABAAIAAAgAAAACAAAAAAAAQAAKAAAAAAAAAAAAAQBAAAAAIIAAAYAAAAAAAABACAAAAAAAAAAAAUAAAAAAgQQABAABAAAABCACAAAAAAEIAAEAIAAAAgCAABAAAAAgAAACAAAAAEAAIgACAgAAAAAAAAAABAAAAAAAAAAACCAAABABAAABACAAAQAAAAAAAAIAAABAABQAAACAAAgAAAAAAAAAAmCAAoAAAAAAAAAAIEEAAAAAAAABAABAAQACAAAAAACAAAAAAIAAAAgAAIAAAICABAgAAAAABAIAAAABAAAAAAAAAAAwQAA4AQAAAAgAgAAAAAAAAgACAAAAAIIAQAAAAAAAAAQAAAAAAAAAQAAAABAAJACQQAIAQAAAAAAAAAAAAAAAAAAAgAAAAgAgACAAAAAIBAEAAQAAAAAAgAAAAABAAQAAAAAAAAEAAAACAAAAAAAAAAAAAAAAAAAEAEAEAAQkAAAAAAAAAIAAAAAAAAAAAAAgAAAAAAAAAAAAAAAACAAAAAAAAIgAAAAEEAAAAAQAAAAQgAAAAAAAAAASQAAABAAAAAAAAAAAAAAAgABAAAQABAAAAIAIAAAIIAAAAQAAAAAAAQAgAAAAIAIAAgABAAAQBAAAACAAAAAACIQAAAAAAAAAAAAAAAEQAAAAAEIAAAAQAAAAQEACAIAAAAEAAAAAAIEABAAEA"
        }
      },
      {
        "_index" : "flint_benchmark_bloom_filter_cardinality_64_adaptive_true_fpp_0.03_num_candidates_5",
        "_id" : "z4tKYo4BM5f06XztFDKb",
        "_score" : 1.0,
        "_source" : {
          "value" : "AAAAAQAAAAUAAAB1AAAAAAAAIAAAAAAAAAAAAAAAAAAAABAAAAAAQAAAAAAIAAAAARAAAAAAAAAEQQEAAADEBAAAAAASAgAAAEAMAAAgAAAAAAAAAQBAAAABgAACAAIAAAAAAAAAAAAAAAAAAAEAAAAAABAAAAAgAIAAAACABIBAAAAAEEACAAAAAAAAUCEAIIAAAAAAEAAIAAQBAAAEAAAAAAACAABAADAAAAAoAABAAAAAAJAAAAAAAAAAAAACAAAAAAAAAAAAAAACAAAAAAAAAAAAAACAgBgAAEEAAIAAAigAAAACEAAAAAAAAAIAAAAQAAABAAAAAAAAAAAAACAAAACACAAAAAAAAAAAIAABAAAAAAAABAAUAAgIABAAAAAAAQAAAkBAAAAAABAgBIDAAACgQAgAAAABAQAAAAAQAAACAACAAAAAAAAIQAAAAAAAAAAAAQAAAAAEAAAFwAAICgAAECQABAAEAKAEAAAAAAADAAAAAAECABAAGAAjgEAAAAACAAAAQAQAAAQgQAAgAAAAAAAAAIAAABAAAAIAAAAAAABAAIAAAgAAAACAAAAAAAAQAAKAAAAAAAAAAAAAQBAAAAAIIAAAYAAAAAAAABACAAAAAAAAAAAAUAAAAAAgQQABAABAAAABCACAAAAAAEIAAEAIAAAAgCAABAAAAAgAAACAAAAAEAAIgACAgAAAAAAAAAABAAAAAAAAAAACCAAABABAAABACAAAQAAAAAAAAIAAABAABQAAACAAAgAAAAAAAAAAmCAAoAAAAAAAAAAIEEAAAAAAAABAABAAQACAAAAAACAAAAAAIAAAAgAAIAAAICABAgAAAAABAIAAAABAAAAAAAAAAAwQAA4AQAAAAgAgAAAAAAAAgACAAAAAIIAQAAAAAAAAAQAAAAAAAAAQAAAABAAJACQQAIAQAAAAAAAAAAAAAAAAAAAgAAAAgAgACAAAAAIBAEAAQAAAAAAgAAAAABAAQAAAAAAAAEAAAACAAAAAAAAAAAAAAAAAAAEAEAEAAQkAAAAAAAAAIAAAAAAAAAAAAAgAAAAAAAAAAAAAAAACAAAAAAAAIgAAAAEEAAAAAQAAAAQgAAAAAAAAAASQAAABAAAAAAAAAAAAAAAgABAAAQABAAAAIAIAAAIIAAAAQAAAAAAAQAgAAAAIAIAAgABAAAQBAAAACAAAAAACIQAAAAAAAAAAAAAAAEQAAAAAEIAAAAQAAAAQEACAIAAAAEAAAAAAIEABAAEA"
        }
      },
      {
        "_index" : "flint_benchmark_min_max_cardinality_64_default",
        "_id" : "xotIYo4BM5f06XztZzKc",
        "_score" : 1.0,
        "_source" : {
          "MinMax_value_0" : 1,
          "MinMax_value_1" : 64
        }
      },
      {
        "_index" : "flint_benchmark_min_max_cardinality_64_default",
        "_id" : "x4tIYo4BM5f06XztbDII",
        "_score" : 1.0,
        "_source" : {
          "MinMax_value_0" : 1,
          "MinMax_value_1" : 64
        }
      },
      {
        "_index" : "flint_benchmark_partition_cardinality_1_default",
        "_id" : "xYtIYo4BM5f06XztYjIx",
        "_score" : 1.0,
        "_source" : {
          "value" : 1
        }
      },
      {
        "_index" : "flint_benchmark_value_set_cardinality_64_max_size_100",
        "_id" : "yItIYo4BM5f06XztcTK3",
        "_score" : 1.0,
        "_source" : {
          "value" : [
            60,
            52,
            31,
            2,
            54,
            25,
            4,
            48,
            27,
            19,
            50,
            42,
            21,
            44,
            23,
            15,
            46,
            38,
            17,
            61,
            40,
            11,
            63,
            34,
            13,
            5,
            57,
            36,
            28,
            7,
            59,
            51,
            30,
            9,
            1,
            53,
            32,
            24,
            3,
            55,
            47,
            26,
            49,
            20,
            43,
            22,
            14,
            45,
            37,
            16,
            39,
            18,
            10,
            62,
            41,
            33,
            12,
            64,
            56,
            35,
            6,
            58,
            29,
            8
          ]
        }
      }
      ...
    ]
  }
}

Issues Resolved

opensearch-project/sql#1399

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[noCharger]

Is this the intended result? Can we include some explanations for why the BF read takes so long?

[dai-chen]

I think it's because BF is translated to Painless script filtering which deserializes BF out of bytes and then do the membership check. Reading other data structures is simply translated to OpenSearch match query.

[penghuo]

why BF latency is lower than MinMax?

[noCharger]

@dai-chen do we want to move this to 0.4?

[dai-chen]

@dai-chen do we want to move this to 0.4?

Moved to 0.4. Will try to address the comments this week. Thanks!

[dai-chen]

Will reopen and address the comments.