Support for GCP Application Default Credentials strategy

[fahadshamiinsta]

Is your feature request related to a problem? Please describe.
Currently, the GoogleCloudStorageService does not support the Application Default Credential strategy of finding the credentials when the credential JSON file is not provided while fetching or creating snapshots in GCP (see the strategy search order here).
During the creation of StorageOptions (in createStorageOptions method) when ServiceAccountCredentials are not found we log a warning message saying "Application Default Credentials are not supported out of the box ....".
In addition to this, the snapshot api GET and PUT requests for snapshots will not work and access denied exception will be thrown when you do not have the credential JSON file.

GET _snapshot/my-opensearch-repo/my-first-snapshot

PUT /_snapshot//

Describe the solution you'd like
Support the use of Application Default Credentials by using the existent google auth library (google-auth-library-oauth2-http-1.7.0.jar). Update the GoogleCloudStorageService to support creating a GCS client using the Application Default Credentials, by fetching those default credentials in a doPrivileged block.

Describe alternatives you've considered
I do not see there will be an alternative solution within the existing Opensearch repository, unless the user has to implement the changes in its own client.

Additional context
We still get the access denied error if the google client is instantiated without being wrapped in a doPrivileged block i.e not using SocketAccess class

OpenSearch log output...

[2023-06-23T02:37:58,709][WARN ][o.o.r.g.GoogleCloudStorageService] [i6d169f91] "Application Default Credentials" are not supported out of the box. Additional file system permissions have to be granted to the plugin.
Jun 23 02:37:59 i6d169f91 opensearch[476672]: [2023-06-23T02:37:59,846][WARN ][r.suppressed ] [i6d169f91] path: /_snapshot/es-d8a9227a-e437-4bf9-bd70-652a60678f41-snapshot-repo/es-6d169f91-0063-4a28-9378-748bcabb09ab-snapshot-a8aec569-0b97-4a20-8390-95105a478470, params: {master_timeout=30s, repository=es-d8a9227a-e437-4bf9-bd70-652a60678f41-snapshot-repo, wait_for_completion=false, snapshot=es-6d169f91-0063-4a28-9378-748bcabb09ab-snapshot-a8aec569-0b97-4a20-8390-95105a478470}
Jun 23 02:37:59 i6d169f91 opensearch[476672]: org.opensearch.repositories.RepositoryException: [es-d8a9227a-e437-4bf9-bd70-652a60678f41-snapshot-repo] Unexpected exception when loading repository data
Jun 23 02:37:59 i6d169f91 opensearch[476672]: at org.opensearch.repositories.blobstore.BlobStoreRepository.doGetRepositoryData(BlobStoreRepository.java:1738) [opensearch-2.7.0.jar:2.7.0]
Jun 23 02:37:59 i6d169f91 opensearch[476672]: at org.opensearch.action.ActionRunnable$2.doRun(ActionRunnable.java:88) [opensearch-2.7.0.jar:2.7.0]
Jun 23 02:37:59 i6d169f91 opensearch[476672]: at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:806) [opensearch-2.7.0.jar:2.7.0]
Jun 23 02:37:59 i6d169f91 opensearch[476672]: at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-2.7.0.jar:2.7.0]
Jun 23 02:37:59 i6d169f91 opensearch[476672]: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
Jun 23 02:37:59 i6d169f91 opensearch[476672]: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
Jun 23 02:37:59 i6d169f91 opensearch[476672]: at java.lang.Thread.run(Thread.java:833) [?:?]
Jun 23 02:37:59 i6d169f91 opensearch[476672]: Caused by: com.google.cloud.storage.StorageException: 401 Unauthorized
Jun 23 02:37:59 i6d169f91 opensearch[476672]: GET https://storage.googleapis.com/download/storage/v1/b/d8a9227a-e437-4bf9-bd70-652a60678f41/o/5eece890-2b5d-4ac2-b214-c7d332db87c1%2Findex-2?alt=media
Jun 23 02:37:59 i6d169f91 opensearch[476672]: Anonymous caller does not have storage.objects.get access to the Google Cloud Storage object. Permission 'storage.objects.get' denied on resource (or it may not exist).
Jun 23 02:37:59 i6d169f91 opensearch[476672]: at com.google.cloud.storage.StorageException.translate(StorageException.java:97) ~[?:?]
Jun 23 02:37:59 i6d169f91 opensearch[476672]: at org.opensearch.repositories.gcs.GoogleCloudStorageRetryingInputStream.lambda$openStream$3(GoogleCloudStorageRetryingInputStream.java:146) ~[?:?]
Jun 23 02:37:59 i6d169f91 opensearch[476672]: at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:105) ~[?:?]
Jun 23 02:37:59 i6d169f91 opensearch[476672]: at com.google.cloud.RetryHelper.run(RetryHelper.java:76) ~[?:?]
Jun 23 02:37:59 i6d169f91 opensearch[476672]: at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50) ~[?:?]
Jun 23 02:37:59 i6d169f91 opensearch[476672]: at org.opensearch.repositories.gcs.GoogleCloudStorageRetryingInputStream.openStream(GoogleCloudStorageRetryingInputStream.java:128) ~[?:?]
Jun 23 02:37:59 i6d169f91 opensearch[476672]: at org.opensearch.repositories.gcs.GoogleCloudStorageRetryingInputStream.(GoogleCloudStorageRetryingInputStream.java:107) ~[?:?]
Jun 23 02:37:59 i6d169f91 opensearch[476672]: at org.opensearch.repositories.gcs.GoogleCloudStorageRetryingInputStream.(GoogleCloudStorageRetryingInputStream.java:89) ~[?:?]
Jun 23 02:37:59 i6d169f91 opensearch[476672]: at org.opensearch.repositories.gcs.GoogleCloudStorageBlobStore.readBlob(GoogleCloudStorageBlobStore.java:222) ~[?:?]
Jun 23 02:37:59 i6d169f91 opensearch[476672]: at org.opensearch.repositories.gcs.GoogleCloudStorageBlobContainer.readBlob(GoogleCloudStorageBlobContainer.java:85) ~[?:?]
Jun 23 02:37:59 i6d169f91 opensearch[476672]: at org.opensearch.repositories.blobstore.BlobStoreRepository.getRepositoryData(BlobStoreRepository.java:1889) ~[opensearch-2.7.0.jar:2.7.0]
Jun 23 02:37:59 i6d169f91 opensearch[476672]: at org.opensearch.repositories.blobstore.BlobStoreRepository.doGetRepositoryData(BlobStoreRepository.java:1700) ~[opensearch-2.7.0.jar:2.7.0]
Jun 23 02:37:59 i6d169f91 opensearch[476672]: ... 6 more
Jun 23 02:37:59 i6d169f91 opensearch[476672]: Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException: 401 Unauthorized

[robertdkmarshall]

@pajuric This is the GCP issue I mentioned.

[minalsha]

@davidlago , @peternied seeking your help with triaging this issue. Thanks

[peternied]

Thanks for filing @fahadshamiinsta this looks like a good feature request, I'd be happy to review a pull request.

Edit: Thanks for creating, I'll took a look #8394

[davidlago]

It seems like there is a PR for it by @fahadshamiinsta (#8394)