Add dataset parser for vector search

[VijayanB]

Description

Vector Search Workloads usually uses dataset of type hdf5 and bigann. Hence, added support
to parse, partition those file formats. This will be later used by VectorSearch Param source in next pr.

Issues Resolved

part of #103

Testing

• New functionality includes testing

[Describe how this change was tested]

make test
tests/utils/parse_test.py::ParseParamsFunctionalTests::test_parse_float_parameter_default PASSED [ 55%]
tests/utils/parse_test.py::ParseParamsFunctionalTests::test_parse_float_parameter_from_params PASSED [ 55%]
tests/utils/parse_test.py::ParseParamsFunctionalTests::test_parse_int_parameter_default PASSED [ 56%]
tests/utils/parse_test.py::ParseParamsFunctionalTests::test_parse_int_parameter_from_params PASSED [ 56%]
tests/utils/parse_test.py::ParseParamsFunctionalTests::test_parse_string_parameter_default PASSED [ 56%]
tests/utils/parse_test.py::ParseParamsFunctionalTests::test_parse_string_parameter_from_params PASSED [ 56%]

tests/utils/dataset_test.py::DataSetTestCase::testBigANNAsAcceptableDataSetFormatWithFloatExtension PASSED [ 47%]
tests/utils/dataset_test.py::DataSetTestCase::testHDF5AsAcceptableDataSetFormat PASSED [ 47%]
tests/utils/dataset_test.py::DataSetTestCase::testUnSupportedDataSetFormat PASSED [ 47%]

================= 1200 passed, 5 skipped, 3 warnings in 15.22s =================

---

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[VijayanB]

I tested this with my next change where i actually perform vector search using new param source that accepts hdf5 and u8bin dataset.

[IanHoang]

Nit: what does v represent? Is there a more descriptive term that you can be used instead?

[IanHoang]

Update: v represents vector. Can we update the variable to be vector?

[VijayanB]

Ack

[IanHoang]

There's some redundancy in these three functions and I think we can simplify it. Assuming OSB will need to check the type before calling one of the three functions, maybe we can condense these three functions into a single generic function where default parameter is dynamically typed and treated as Any, such as the following:

def parse_parameter(key: str, params:dict, default=None):
    valid_parameter_types = [str, int, float]

    if key not in params:
        if default:
            return default
        else:
            raise ConfigurationError(
                f"Value cannot be None for param {key}"
            )

    for parameter_type in valid_parameter_types:
        if isinstance(params[key], parameter_type):
            return params[key]

    formatted_parameter_types = ", ".join([ parameter_type.__name__ for parameter_type in valid_parameter_types])

    raise ConfigurationError(f"Value must be one of the followings for param {formatted_parameter_types}")

I've tested this against your test cases and they all pass as well. Let me know your thoughts @VijayanB ?

[VijayanB]

What happens if we pass integer value to string parameter? The above looks complicated for simple function. I don't see any advantage. However, if you have strong opinion, i can refactor.

[IanHoang]

What happens if we pass integer value to string parameter?

Are you referring to passing an integer value to the string parameter key? The code above is based off of the three functions (parse_str_parameter(), parse_float_parameter(), and parse_int_parameter()) and how they were used in the unittests in parse_test.py. In parse_test.py, it doesn't seem like there's a scenario where an integer value is passed to key since only values such as "int-value", "string-value", "default-value", "float-value" are provided. Based off these unittests, I was under the assumption that OSB must first identify the parameter's data-type before calling any of these functions. Let me know if I'm misunderstanding this.

That being said, I don't have a strong preference and if it's more simple to use these three functions, we can keep them as is 👍🏻

[IanHoang]

Left some comments

[IanHoang]

Thanks for doing this!

[IanHoang]

LGTM!