Opening from S3 with credentials only via official mechanisms

[tischi]

@martinschorb @constantinpape switching to the new N5 libraries I cannot easily support anymore that users would enter their S3 credentials into MoBIE, but they would follow the official AWS S3 mechanisms, which I think mean to store those credentials in some special file on their computer. Is that OK?

[constantinpape]

If somehow possible it would be quite helpful to still support the direct credential entry. I am using this in a couple of projects, and it makes it much easier to explain this to collaborators, especially since the location of the credential file differs between operating system.

If there's really no other way than I guess we can do without it, but maybe there is some way to add support for a credentials provider?

[martinschorb]

I guess you could make MoBIE write to the aws credentials file. It will be a bit hacky (figure out the OS, and storage location), but why not having something like credential manager inside MoBIE that would take the project name (github URL) as ID and manage the keys (create new ones in the file if not yet existing).

[tischi]

@martinschorb @constantinpape

I asked chatGPT about it. It seems we could add a new AddAWSCredentialsCommand which would appear under Plugins > MoBIE > AWS > Add Credentials and contain the below code.

What do you think?

You can save AWS credentials in a credentials file that the AWS SDK for Java
will automatically detect. Here's a simple Java code example to save AWS
credentials. This example saves the credentials in the default location (
~/.aws/credentials on Linux and macOS, or C:\Users\USERNAME.aws\credentials
on Windows).

    import com.amazonaws.auth.profile.ProfileCredentialsProvider;
    import com.amazonaws.auth.AWSCredentials;
    import com.amazonaws.auth.AWSStaticCredentialsProvider;
    import com.amazonaws.auth.BasicAWSCredentials;
    import com.amazonaws.services.s3.AmazonS3;
    import com.amazonaws.services.s3.AmazonS3ClientBuilder;

    import java.io.File;
    import java.io.FileWriter;
    import java.io.IOException;
    import java.util.Scanner;

    public class SaveAWSCredentials {

        public static void main(String[] args) {
            Scanner scanner = new Scanner(System.in);

            System.out.println("Enter AWS Access Key ID: ");
            String accessKeyId = scanner.nextLine();

            System.out.println("Enter AWS Secret Access Key: ");
            String secretAccessKey = scanner.nextLine();

            String profileName = "default"; // Use "default" or another profile
  name as needed
            saveCredentials(accessKeyId, secretAccessKey, profileName);
        }

        private static void saveCredentials(String accessKeyId, String
  secretAccessKey, String profileName) {
            String homeDirectory = System.getProperty("user.home");
            String awsCredentialsPath = homeDirectory + "/.aws/credentials"; //
  For Linux and macOS
            // For Windows, use: String awsCredentialsPath = homeDirectory +
  "\\.aws\\credentials";

            File credentialsFile = new File(awsCredentialsPath);
            credentialsFile.getParentFile().mkdirs(); // Create the .aws
  directory if it doesn't exist

            try (FileWriter writer = new FileWriter(credentialsFile, true)) {
                writer.append("\n[" + profileName + "]\n");
                writer.append("aws_access_key_id = " + accessKeyId + "\n");
                writer.append("aws_secret_access_key = " + secretAccessKey +
  "\n");
                System.out.println("Credentials saved successfully.");
            } catch (IOException e) {
                e.printStackTrace();
            }
        }
    }

This code prompts the user for their AWS Access Key ID and AWS Secret Access
Key, then appends them to the credentials file under the specified profile.
If the .aws/credentials file or directory doesn't exist, it creates them.

Important Security Note: Storing sensitive information like AWS credentials on
your computer should be done with caution. Ensure that the credentials file
permissions are set to be accessible only by your user account to prevent
unauthorized access.

[tischi]

In fact I am confused now, chatGPT says:

  AWS S3 does not support protecting buckets with a password in the traditional
  sense (like password-protected files or services). Access control in AWS S3 is
  managed through AWS Identity and Access Management (IAM) and bucket policies,
  rather than individual passwords.

  For your use case, where you want to distribute access to various users, you
  could consider the following approaches:

  ### 1. IAM Users and Policies

  • Create individual IAM users for each person who needs access.
  • Attach policies to these users or their group, granting them permissions to
  access the specific S3 bucket.
  • Distribute the AWS Access Key ID and Secret Access Key to each user, which
  they can use with an AWS SDK or CLI.

  ### 2. Pre-signed URLs

  • If users just need to download or upload specific files, you can generate
  pre-signed URLs.
  • Pre-signed URLs grant temporary access to a specific S3 object, and you can
  control the expiration time of these URLs.
  • Distribute these URLs to users for uploading or downloading without giving
  them full AWS credentials.

  ### 3. Cognito for Application Users

  • If you are managing access in an application, consider using Amazon Cognito.
  • Cognito can authenticate users and provide them with temporary AWS
  credentials limited to the permissions you define, such as access to a
  specific S3 bucket.

  ### 4. Third-party Solution

  • Use a third-party solution or custom application that manages user access and
  integrates with AWS S3, where the application layer controls access to S3
  resources based on user credentials (username/password).

  ### Summary

  For directly controlling access to an S3 bucket via a single password for
  multiple users, you would typically manage this outside of AWS (e.g., through
  a custom application layer that authenticates users and then provides access
  to S3 resources based on your logic). AWS's built-in mechanisms focus on IAM
  users, roles, and policies for access control, which do not align with the
  single-password model.

I am not sure what we are currently doing...do you?

[constantinpape]

1. IAM Users and Policies

• Create individual IAM users for each person who needs access.
• Attach policies to these users or their group, granting them permissions to
access the specific S3 bucket.
• Distribute the AWS Access Key ID and Secret Access Key to each user, which
they can use with an AWS SDK or CLI.

We are using this, but using generic users instead of creating an account per user. I.e. we usually have one user with read and one user with write access, and would distribute the credentials for the user with read access to whoever just needs to access the data.

I asked chatGPT about it. It seems we could add a new AddAWSCredentialsCommand which would appear under Plugins > MoBIE > AWS > Add Credentials and contain the below code.

What do you think?

This can be done, but it's not really a clean solution. I am not sure which library you're using now for reading n5/zarr, but not being able to pass on a credentials provider (or a client initialized with the credentials provider) seems to be a major lack of functionality.

For reference, here is a simple example how to do this according to ChatGPT:

[tischi]

We have to look whether we can use an n5 reader with credentials here:

            N5URI n5URI = new N5URI( uri );

            String containerPath = n5URI.getContainerPath();
            N5Reader n5 = new N5Factory().openReader( containerPath );
            // FIXME: somehow add the credentials if this is an AWS-S3 reader...
            String group = n5URI.getGroupPath() != null ? n5URI.getGroupPath() : "/";
            List< N5Metadata > metadata = Collections.singletonList( N5MetadataUtils.parseMetadata( n5, group ) );

            final DataSelection selection = new DataSelection( n5, metadata );
            converterSetups = new ArrayList<>();
            sourcesAndConverters = new ArrayList<>();
            bdvOptions = BdvOptions.options().frameTitle( "" );

[tischi]

OK, nice, in fact, upon closer examination the new n5 library provides functionality for this: N5Factory().s3UseCredentials( credentials ).

    private static N5Factory getN5Factory()
    {
        String[] s3AccessAndSecretKey = S3Utils.getS3AccessAndSecretKey();
        if( s3AccessAndSecretKey != null )
        {
            BasicAWSCredentials credentials = new BasicAWSCredentials( s3AccessAndSecretKey[ 0 ], s3AccessAndSecretKey[ 1 ] );
            return new N5Factory().s3UseCredentials( credentials );
        }
        else
        {
            return new N5Factory();
        }
    }

Is there something that I could easily test this with (you could send me the credentials in the EMBL chat)?

[tischi]

In fact, ideal would be a real @Test for this.
Could one of you create a mobie-credentials-test bucket containing one ome-zarr that can only be read with credentials?
I would then put those credentials inside the Java code, so it would be readable for everyone.

[constantinpape]

In fact, ideal would be a real @Test for this.
Could one of you create a mobie-credentials-test bucket containing one ome-zarr that can only be read with credentials?
I would then put those credentials inside the Java code, so it would be readable for everyone.

Yes, I think that's the best solution. It's probably best if you could get a bucket on the EMBL s3 for this.