v0.3.0 - Multipart S3 Upload, Fs2AsyncResponseTransformer, and more!

Hello and I'm happy to announce a major feature upgrade for pure-aws in this new v0.3.0 release! There are several new features in this release to be aware of, and some breaking changes:

New Features

S3 Multipart Upload

S3Sink now supports streaming multipart uploads with adjustable part size! The default value of 5120 bytes is required by the AWS spec, so be sure to not set it any lower than that when uploading parts.

Fs2AsyncResponseTransformer

The AWS v2 SDK requires that streaming responses from the Async clients use some kind of AsyncResponseTransformer implementation. The problem is, all of the defaults involve either loading everything into memory or writing to a file, but not actually giving you the raw stream of bytes. In the core module shared by all submodules there is a new Fs2AsyncResponseTransformer available that gives you a tuple of (ResponseT, fs2.Stream[F, Byte]) where ResponseT is the response object of your operation and the stream is the raw bytestream of your requested object. This depends on the excellent work done as part of fs2-reactive-streams.

S3 ETag Support

Uploading objects to S3 will now return the object "ETag", a string that uniquely identifies your uploaded object.

S3 Testing Enhancements

The s3-testing module just got a bunch of small enhancements in line with the above changes, and then some:

• TestS3Sink now supports faking multipart uploads. Since it's a test backend, it works the same as all other uploads, but it's nice to note that the API got an enhancement.
• TestS3Sink also takes an optional Iterator[String] parameter now for emitting ETags after each upload. This way you can control what ETags your uploaded files have, if it matters to you. The default is just incrementing starting from 0, as a string.
• All S3 test interfaces now allow you to stub them with a Throwable response. Use this to test recovery from errors in code that uses these interfaces.

SimpleS3Client

The S3 module was originally written with the intent that you would only need certain sub-sections of the S3 API at any given time, so they are separated out into different traits. Unfortunately this had a side-effect of making it confusing to use multiple of these clients using the same backend. SimpleS3Client is a new one that combines all currently available traits (source, sink, object ops) into one package. This makes it easier to create one big client at the beginning of your application instead of having to manually create a PureS3Client and pass it in multiple times to each layered client.

Breaking Changes

Fs2AsyncResponseTransformer requires a ConcurrentEffect constraint, and streaming responses from the S3 async clients have been upgraded to use it. Currently this is just S3Source. Be sure that the scope you are constructing these in also has the ConcurrentEffect constraint, and you should be fine.