Async / streaming support

[martin-traverse]

Hello,

Please can we get a version of the storage API for Java that supports async and streaming operations. Both AWS and Azure have had this capability for some time, with separate async versions of their client classes.

While the blocking paradigm is good for small, simple applications there are many more advanced use caseas where this model is not a good fit. For example, middlewares and performance-critical platform code. I appreciate that the current SDK does do some buffering on read and write operations to minimize blocking, but this is not the same as providing a streaming data API, there is no solution for non-blocking read and flow-control cannot be integrated into the client code portion of the data pipeline. Unary operations also require thread-per-operation.

As a reference point, AWS uses Java's CompletableFuture for unary operations and provides implementations of org.reactivestreams Publisher / Subscriber classes for data transfer. Azure uses the full Project Reactor API, with Mono for unary operations and Flux for data transfer. Both of these approaches are quite sufficient for our use case, which requires each operation to run async on a single event loop with backpressure from the storage SDK up into our platform code an ultimately the client API. Also using e.g. Guava futures with the gRPC stream obvserver API would be sufficient for us, so long as the manual flow control interface was exposed, or another dedicated API that follows a simillar pattern.

As per our conversation in #2121, there is not currently a solution for an async storage API. The new gRPC API does support async, but (a) this is not ready for public use and (b) it is intended anyway as a low-level, internal API which is lacking a lot of functionality, e.g. error handling, retries etc.