S3 slow download compared to aws cli

[CAReyes]

I'm working on a custom file downloader using the aws rust sdk. I'm testing the performance of it versus the aws cli and noticed that I get capped at about 10-15 MiB/s download using the sdk. The aws cli is getting about up to 40 MiB/s download. I've scratched my head for a while trying figure out where performance was lost.

Here is the code snippet of what I am doing in the SDK:

let byte_stream = client
    .get_object()
    .bucket(bucket)
    .key(path)
    .send()
    .await?
    .body
    .into_async_read();

let mut reader = BufReader::new(byte_stream);
let mut writer = BufWriter::new(OpenOptions::new().create(true).write(true).open(&file_path).await?);

let mut buf = vec![0; 1024]; // 1KB buffer 
while let Ok(size) = reader.read(&mut buf).await {
    if size == 0 {
        break;
    }
    writer.write_all(&buf).await?;
}

versus running the aws s3 cp command to a local directory.

I've played around with buffer sizes and sync vs async but no luck.

[rcoh]

First, of course, is to double check that you're building in --release. At only 10-15MiB/s this is a possible culprit.

If you are using a new version of the CLI, the CLI is using the CRT, a C library to automatically split the download into multiple ranged GETs—the SDK does not do that automatically. You would need to write code to perform ranged GETs and use write_at to write the chunks to disk.

We have a code snippet in our S3 benchmark that shows how to use ranged GETs.

Note that build a library for customers to do this directly is high on our priority list for 2024.

We've benchmarked the Rust SDK achieving multiple Gib/S on large instance types so let me know if you're unable to match the performance.

[pavlospt]

@rcoh Hello, happened to stumble upon this today too. Why isn't this part of the SDK? 🤔

[Velfi]

The boring answer is that we have limited time and we have to make tough choices regarding what to focus on. We are tracking many tasks for things that people have asked for but we are a small team so there's always going to be more work to do than can be done.

[pavlospt]

That is understandable, it just feels a bit weird to be executing in Enhanced Network instances and S3 transfers from same VPC being the bottleneck :/ Is there something that we could help with?

[Velfi]

Work on a high level library to handle this will likely be an internal project until it's complete. However, we're going to be using code similar to the code snippet that @rcoh linked. Would you mind trying out that code and seeing if it works well for your use case?

[pavlospt]

I had given it a try but this specific code is downloading after creating a new File which in my use case, would not fit. Instead I had modified it to download in a HashMap<ID, u[8]> and sort it by its key, then collect it in ID order, aiming to "emulate" the .write_at that was being used on the sample code, since I was operating with bytes in order to avoid writing a new File for each S3 request I was making!