Setup integration test with minio

[xushiyan]

No description provided.

[abyssnlp]

I'll take this one if noone's assigned to it yet.

[xushiyan]

@abyssnlp before you start, can you please elaborate on the design?

[abyssnlp]

Sure, so at a high level:

• Spin up minio via testcontainers before running the integration tests
• Setup mino object storage with s3, gs and create the test hudi table
• Run marked integration tests

I'll add more details today after work. Please feel free to add things I should keep in mind while I work on this.

[xushiyan]

@abyssnlp high-level looks good. a heads-up about testing data - since hudi-rs not yet supports hudi writer, we are using fixed pre-generated tables as the testing tables: see https://github.com/apache/hudi-rs/tree/main/crates/tests/data/tables Would like to see some detailed design around provisioning test tables through minio volumes

[abyssnlp]

Sorry about the delay.
Thanks for pointing to that. So we can mount the existing tables under here into the container before running the tests.

Something like:

.with_mount(Mount::bind_mount(canonicalize(Path::new("tests/data"))?.into_os_string().into_string().unwrap(),

However something that I found out about testcontainers in Rust is that it doesn't support reusing the container for multiple tests so the integration tests would be for ex. in a single test function. More about it here. There are also workarounds. Alternative would be to use docker-compose to spin up minio before running integration tests and spin it down after.

So this is how I'm thinking about approaching it:

• (optional) ./docker-compose.yaml for spinning up required containers for MinIO
• crates/tests/src/common.rs - some utility code for ex. to create an s3 bucket, put the pre-generated tables
• crates/tests/src/integration_test.rs - spin up minio using testcontainers and run integration tests
• Integration tests are marked as a separate feature (integration_test) so they can be run separately

I had some questions as well.

• Does this approach make sense? Would love to hear what you think
• Do we plan on also running integration tests for Azure and GCP?

[xushiyan]

@abyssnlp sounds good to make use of docker-compose - it'll be convenient to evolve the tests as we probably need to add more components in future. to answer the questions

• crates/tests is a crate to provide all kinds of common hudi test utilities, but we don't want to host actual tests in it.
• we may want to install the hudi crates locally, to mimic how users install hudi from crates.io
• we can organize the integration tests in a separate folder, not necessarily as a crate, since it's docker based, and we don't publish it either
• we surely want to integ test for azure and gcp as well - just picked minio for easy start with s3.

[abyssnlp]

Thanks for sharing your thoughts on it.
Having them separate from the crates sounds good. I've started some initial work on a local branch and managed to get Minio up with the pre-generated tables.

I'm currently running into some issues trying to read the tables via hudi-rs.

I've tried using both environment variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and local MinIO specific AWS_ENDPOINT and ALLOW_HTTP) and providing the config via the HudiDataSource::new_with_options as a Vec<&str, &str>.
I can confirm the object store config works for hudi::storage::Storage for ex. while trying to read the contents of .hoodie/hoodie.properties.

Might be some bad configuration on my end. I'll continue working on it this week and keep posting updates here.

[xushiyan]

@abyssnlp any plan to put this up in a PR?

[abyssnlp]

@xushiyan Yes i'll put it up in a PR soon (today or tomorrow).