Support specifying GCP account credentials as a config option.

[teo-tsirpanis]

SC-44515

---

TYPE: CONFIG
DESC: Add vfs.gcs.service_account_credential config option that specifies a Google Cloud service account credential JSON string.

---

TYPE: CONFIG
DESC: Add vfs.gcs.external_account_credential config option that specifies a Google Cloud Workload Identity Federation credential JSON string.

[shortcut-integration]

This pull request has been linked to Shortcut Story #44515: Support specifying GCP account credentials as a config option..

[teo-tsirpanis]

I don't know why the tests are failing; they succeed on my local Windows machine in both Release and Debug mode.

[thetorpedodog]

I should be able to get this tested sometime today, so will get back to you with information about whether everything works end-to-end once I have done so.

In the meantime, I have some minor thoughts about naming, to be consistent with the way Google names and documents things. None of these are requirements, just ideas to consider.

• vfs.gcs.service_account_credentials might be better named vfs.gcs.[service_?]account_key, since Google Cloud describes these files as "account keys". (As for the inclusion or exclusion of service_, I think you might be able to provide a user account JSON string here and have it work, but I'm not positive. Even then, it's probably OK to include the service_ since that’s the intended use case, but it could be omitted for to keep the config name shorter.)
• vfs.gcs.external_account_credentials: maybe workload_identity_config[uration], federated_identity_config[uration], or identity_pool_config[uration] since those better match to the public naming and documentation of workload identity federation. It's not really a credential in and of itself; it's more like a configuration that tells you how to fetch a credential. (That said it is sometimes described as an "external account" in code. Naming is hard ¯\_(°_o)_/¯.)

Out of everything I think I'm most confident in that we should use _key for the service account key configuration field. Beyond that, it's just vibes.

[teo-tsirpanis]

I took the names from the Google Cloud functions MakeServiceAccountCredentials and MakeExternalAccountCredentials. For comparison Python calls the function from_service_account_info; I couldn't find an equivalent for external credentials. @KiterLuc what do you think?

you might be able to provide a user account JSON string here and have it work

I checked it and unfortunately this logic exists only for Application Default Credentials (the file in GOOGLE_APPLICATION_CREDENTIALS).

[thetorpedodog]

I took the names from the Google Cloud functions MakeServiceAccountCredentials and MakeExternalAccountCredentials. For comparison Python calls the function from_service_account_info; I couldn't find an equivalent for external credentials. @KiterLuc what do you think?

The way I look at it, it’s taking some input (the key, the federation info) to make the credentials; i.e., the credentials are the output, not the input. Also see the specific doc comment on MakeServiceAccountCredentials where the input is referred to specifically as a key.

[thetorpedodog]

I can confirm that everything seems to work here! I built this version of TileDB, pointed the Python library at it in a venv, and used this code:

#!/usr/bin/env python

import argparse
import shutil

import tiledb


def main() -> None:
    parser = argparse.ArgumentParser()
    parser.add_argument("--output", help="file to write to", required=True)
    parser.add_argument(
        "--impersonate", help="impersonate these accounts to open the file"
    )
    group = parser.add_mutually_exclusive_group()
    group.add_argument(
        "--federated-config",
        help="file with a federated workflow configuration",
        type=argparse.FileType(),
    )
    group.add_argument(
        "--service-account-key",
        help="file with a service account key",
        type=argparse.FileType(),
    )
    parser.add_argument("input_file", help="file to read from")

    parsed = parser.parse_args()

    tdb_cfg = {}
    if fedfile := parsed.federated_config:
        with fedfile:
            tdb_cfg["vfs.gcs.external_account_credentials"] = fedfile.read()
    if keyfile := parsed.service_account_key:
        with keyfile:
            tdb_cfg["vfs.gcs.service_account_credentials"] = keyfile.read()

    if parsed.impersonate:
        tdb_cfg["vfs.gcs.impersonate_service_account"] = parsed.impersonate

    vfs = tiledb.VFS(tdb_cfg)

    with vfs.open(parsed.input_file, "rb") as infile:
        with vfs.open(parsed.output, "wb") as outfile:
            shutil.copyfileobj(infile, outfile)


if __name__ == "__main__":
    main()

Cases I tested it with:

$ ./open-with-tiledb.py \
>   --output key-output.png \
>   --service-account-key ./gcp-account-key-impersonator.json \
>   --impersonate [email protected] \
>   gcs://some-bucket/some-file.png
$ ./open-with-tiledb.py \
>   --output federated-output.png \
>   --federated-config ./impersonate-from-url.json \
>   gcs://some-bucket/some-file.png

Weirdly, it seems like using application-default credentials didn’t work right (i.e., specifying neither service-account-key nor federated-config), though running gsutil cp gs://some-bucket/some-file.png gsutil-output.png did work. Could this be a me problem?

Here is the relevant output:

tiledb.cc.TileDBError: [TileDB::GCS] Error: Get object size failed on: some-file.png (Retry policy exhausted, with a last message of Could not create a OAuth2 access token to authenticate the request. The request was not sent, as such an access token is required to complete the request successfully. Learn more about Google Cloud authentication at https://cloud.google.com/docs/authentication. The underlying error message was: PerformWork() - CURL error [6]=Couldn't resolve host name)

[teo-tsirpanis]

I will rename them tomorrow to vfs.gcs.service_account_key and vfs.gcs.workload_identity_configuration, if there are no other objections.

[thetorpedodog]

I will rename them tomorrow to vfs.gcs.service_account_key and vfs.gcs.workload_identity_configuration, if there are no other objections.

I am happy with both of those names but feel free to use another if you come up with something you think is better. We have time to decide before this is finally merged.

[thetorpedodog]

Weirdly, it seems like using application-default credentials didn’t work right (i.e., specifying neither service-account-key nor federated-config), though running gsutil cp gs://some-bucket/some-file.png gsutil-output.png did work. Could this be a me problem?

It was a me problem. There is a difference between being logged in to your project within the Google Cloud CLI and actually having application-default credentials.

You are logged into Google Cloud services within the Google Cloud command-line utilities when you run gcloud init. After doing so, you need to run gcloud auth application-default login for that account to be made available as application-default credentials.

After doing so, my ADCs were created and all three test cases worked perfectly.

[teo-tsirpanis]

Test failures were fixed.

[thetorpedodog]

Looks great! There are a couple comments and strings that should probably be updated to use the new terminology but functionally everything is in order.

[thetorpedodog]

nit: credentials to configuration here.

[teo-tsirpanis]

Done, thanks!

[thetorpedodog]

another string to update with the new terminology (maybe the exact config key name?). I also might phrase this as something like "service account key set; ignoring workload identity configuration" to make it a little clearer (at least in my opinion; you are of course free to leave it as-is).

[teo-tsirpanis]

Done, thanks!

[robertbindar]

I'd usually prefer fixing the precedence of the 2 configs+env in a test, it means it's intentional and we support it that way, I'm also ok with merging this as is.