Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for Google Cloud Storage #176

Open
wants to merge 14 commits into
base: master
Choose a base branch
from
21 changes: 21 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -406,6 +406,27 @@ single input file without looking for other .nd2 files. It is especially importa
working with ND2 plates for the first time or after any acquisition system updates, as there will not be an error in
the logs if the file name does not match `ND2PlateReader`'s expectations.

Object Storage
==============

Support for object storage is handled through Java NIO2.
Currently, only the output can be on cloud storage, and `--overwrite` is not supported.

AWS S3
------

S3 support is provided by [s3fs](https://github.com/lasersonlab/Amazon-S3-FileSystem-NIO2/).
Various parameters can be specified through `--output-options`.

Google Cloud Storage (GCS)
--------------------------

Credentials are handled through [Application Default Credentials](https://cloud.google.com/docs/authentication/application-default-credentials).
The credentials used require read/write access to the bucket. (Minimally, this can be `Storage Object Creator`,
`Storage Object Viewer` and `Storage Object Delete`).

`--output-options` are *not* currently supported with GCS.
=======
MCDReader
---------

Expand Down
2 changes: 2 additions & 0 deletions build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,8 @@ dependencies {
// implementation 'io.nextflow:nxf-s3fs:1.1.0'
implementation 'org.lasersonlab:s3fs:2.2.3'
implementation 'javax.xml.bind:jaxb-api:2.3.0'
implementation 'com.google.cloud:google-cloud-nio:0.127.5'

implementation group: 'ch.qos.logback', name: 'logback-classic', version: '1.3.14'
implementation group: 'ch.qos.logback', name: 'logback-core', version: '1.3.14'

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -128,7 +128,7 @@ public class Converter implements Callable<Integer> {
private volatile Path inputPath;
private volatile String outputLocation;

private Map<String, String> outputOptions;
private Map<String, String> outputOptions = new HashMap<String, String>();
private volatile Integer pyramidResolutions;
private volatile List<Integer> seriesList;

Expand Down