GCP Cloud Storage sink shouldn't include api_key

[nipsufn]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

Best I can tell based on a POC and documentation, GCP Cloud Storage will not work with authentication via api_key field (https://vector.dev/docs/reference/configuration/sinks/gcp_cloud_storage/#api_key).
Based on debug log output, I believe XML API is being used which has to be authenticated with Authorization header:
https://cloud.google.com/storage/docs/xml-api/put-object.
key query string parameter does not appear to be supported at all:
https://cloud.google.com/storage/docs/xml-api/reference-headers#headers

Configuration

test_sink:
    bucket:vector-***
    # this has to be base64
    api_key: dGhpcyBkb2VzIG5vdCBzZWVtIHRvIHdvcmsK
    type: gcp_cloud_storage
    inputs:
      - test_transform
    encoding:
      codec: json
    buffer:
      type: memory
      max_events: 10

Version

0.39.0

Debug Output

2024-07-25T09:52:47.278076Z TRACE sink{component_kind="sink" component_id=test_sink component_type=gcp_cloud_storage}: vector_common::internal_event::events_received: Events received. count=3 byte_size=8199
2024-07-25T09:52:47.488655Z TRACE sink{component_kind="sink" component_id=test_sink component_type=gcp_cloud_storage}: vector_common::internal_event::events_received: Events received. count=2 byte_size=5930
2024-07-25T09:52:47.750017Z TRACE sink{component_kind="sink" component_id=test_sink component_type=gcp_cloud_storage}: vector_common::internal_event::events_received: Events received. count=3 byte_size=8197
2024-07-25T09:52:48.074246Z TRACE sink{component_kind="sink" component_id=test_sink component_type=gcp_cloud_storage}: vector_common::internal_event::events_received: Events received. count=1 byte_size=2967
2024-07-25T09:52:48.271731Z TRACE sink{component_kind="sink" component_id=test_sink component_type=gcp_cloud_storage}: vector_common::internal_event::events_received: Events received. count=2 byte_size=5448
2024-07-25T09:52:48.489061Z TRACE sink{component_kind="sink" component_id=test_sink component_type=gcp_cloud_storage}: vector_common::internal_event::events_received: Events received. count=3 byte_size=7991
2024-07-25T09:52:48.688662Z TRACE sink{component_kind="sink" component_id=test_sink component_type=gcp_cloud_storage}: vector_common::internal_event::events_received: Events received. count=2 byte_size=5448
2024-07-25T09:52:48.904775Z TRACE sink{component_kind="sink" component_id=test_sink component_type=gcp_cloud_storage}: vector_common::internal_event::events_received: Events received. count=2 byte_size=5449
2024-07-25T09:52:49.163712Z TRACE sink{component_kind="sink" component_id=test_sink component_type=gcp_cloud_storage}: vector_common::internal_event::events_received: Events received. count=3 byte_size=8351
2024-07-25T09:52:49.333923Z TRACE sink{component_kind="sink" component_id=test_sink component_type=gcp_cloud_storage}: vector_common::internal_event::events_received: Events received. count=2 byte_size=5464
2024-07-25T09:52:49.595884Z TRACE sink{component_kind="sink" component_id=test_sink component_type=gcp_cloud_storage}: vector_common::internal_event::events_received: Events received. count=5 byte_size=13485
2024-07-25T09:52:49.727245Z TRACE sink{component_kind="sink" component_id=test_sink component_type=gcp_cloud_storage}: vector_common::internal_event::events_received: Events received. count=2 byte_size=5465
2024-07-25T09:52:51.057534Z TRACE sink{component_kind="sink" component_id=test_sink component_type=gcp_cloud_storage}: vector_common::internal_event::events_received: Events received. count=2 byte_size=5474
2024-07-25T09:52:51.058157Z TRACE sink{component_kind="sink" component_id=test_sink component_type=gcp_cloud_storage}: vector_common::internal_event::events_received: Events received. count=2 byte_size=5485
2024-07-25T09:52:51.161889Z TRACE sink{component_kind="sink" component_id=test_sink component_type=gcp_cloud_storage}: vector_common::internal_event::events_received: Events received. count=6 byte_size=16455
2024-07-25T09:52:51.234354Z TRACE sink{component_kind="sink" component_id=test_sink component_type=gcp_cloud_storage}: vector_common::internal_event::events_received: Events received. count=1 byte_size=2922
2024-07-25T09:52:51.359443Z TRACE sink{component_kind="sink" component_id=test_sink component_type=gcp_cloud_storage}: vector_common::internal_event::events_received: Events received. count=1 byte_size=2564
2024-07-25T09:52:51.518516Z DEBUG sink{component_kind="sink" component_id=test_sink component_type=gcp_cloud_storage}: vector::utilization: utilization=0.00028176866171518086
2024-07-25T09:52:51.568674Z TRACE sink{component_kind="sink" component_id=test_sink component_type=gcp_cloud_storage}: vector_common::internal_event::events_received: Events received. count=4 byte_size=10963
2024-07-25T09:52:51.788234Z TRACE sink{component_kind="sink" component_id=test_sink component_type=gcp_cloud_storage}: vector_common::internal_event::events_received: Events received. count=2 byte_size=5484
2024-07-25T09:52:52.001032Z TRACE sink{component_kind="sink" component_id=test_sink component_type=gcp_cloud_storage}: vector_common::internal_event::events_received: Events received. count=2 byte_size=5484
2024-07-25T09:52:52.272418Z TRACE sink{component_kind="sink" component_id=test_sink component_type=gcp_cloud_storage}: vector_common::internal_event::events_received: Events received. count=1 byte_size=2919
2024-07-25T09:52:56.577374Z DEBUG sink{component_kind="sink" component_id=test_sink component_type=gcp_cloud_storage}: vector::utilization: utilization=0.000050142568258952286
2024-07-25T09:53:01.519388Z DEBUG sink{component_kind="sink" component_id=test_sink component_type=gcp_cloud_storage}: vector::utilization: utilization=0.00000503720287155878
2024-07-25T09:53:06.520613Z DEBUG sink{component_kind="sink" component_id=test_sink component_type=gcp_cloud_storage}: vector::utilization: utilization=0.0000005258549215812304
2024-07-25T09:53:07.295540Z TRACE sink{component_kind="sink" component_id=test_sink component_type=gcp_cloud_storage}: vector_stream::driver: Submitting service request. in_flight_requests=0 request_id=1
2024-07-25T09:53:07.295651Z DEBUG sink{component_kind="sink" component_id=test_sink component_type=gcp_cloud_storage}:request{request_id=1}:http: vector::internal_events::http_client: Sending HTTP request. uri=https://storage.googleapis.com/vector-***/date=2024-07-25/1721901187-e6d93ac6-b4c9-4536-ad73-cf4280b9fd02.log?key=dGhpcyBkb2VzIG5vdCBzZWVtIHRvIHdvcmsK method=PUT version=HTTP/1.1 headers={"content-type": "application/json", "content-length": "691288", "x-goog-storage-class": "STANDARD", "user-agent": "Vector/0.39.0 (x86_64-unknown-linux-gnu 73da9bb 2024-06-17 16:00:23.791735272)", "accept-encoding": "identity"} body=[691288 bytes]
2024-07-25T09:53:07.295692Z TRACE sink{component_kind="sink" component_id=test_sink component_type=gcp_cloud_storage}:request{request_id=1}:http: hyper::client::pool: checkout waiting for idle connection: ("https", storage.googleapis.com)
2024-07-25T09:53:07.295725Z TRACE sink{component_kind="sink" component_id=test_sink component_type=gcp_cloud_storage}:request{request_id=1}:http: hyper::client::connect::http: Http::connect; scheme=Some("https"), host=Some("storage.googleapis.com"), port=None
2024-07-25T09:53:07.362162Z DEBUG sink{component_kind="sink" component_id=test_sink component_type=gcp_cloud_storage}:request{request_id=1}:http: hyper::client::connect::http: connecting to 142.251.168.207:443
2024-07-25T09:53:07.362652Z TRACE sink{component_kind="sink" component_id=test_sink component_type=gcp_cloud_storage}:request{request_id=1}:http: mio::poll: registering event source with poller: token=Token(139593849570432), interests=READABLE | WRITABLE
2024-07-25T09:53:07.371041Z DEBUG sink{component_kind="sink" component_id=test_sink component_type=gcp_cloud_storage}:request{request_id=1}:http: hyper::client::connect::http: connected to 142.251.168.207:443
2024-07-25T09:53:07.377581Z TRACE sink{component_kind="sink" component_id=test_sink component_type=gcp_cloud_storage}:request{request_id=1}:http: hyper::client::conn: client handshake Http1
2024-07-25T09:53:07.377629Z TRACE sink{component_kind="sink" component_id=test_sink component_type=gcp_cloud_storage}:request{request_id=1}:http: hyper::client::client: handshake complete, spawning background dispatcher task
2024-07-25T09:53:07.377704Z TRACE sink{component_kind="sink" component_id=test_sink component_type=gcp_cloud_storage}:request{request_id=1}:http: hyper::client::pool: checkout dropped for ("https", storage.googleapis.com)
2024-07-25T09:53:07.398882Z TRACE sink{component_kind="sink" component_id=test_sink component_type=gcp_cloud_storage}:request{request_id=1}:http: hyper::client::pool: put; add idle connection for ("https", storage.googleapis.com)
2024-07-25T09:53:07.398903Z DEBUG sink{component_kind="sink" component_id=test_sink component_type=gcp_cloud_storage}:request{request_id=1}:http: hyper::client::pool: pooling idle connection for ("https", storage.googleapis.com)
2024-07-25T09:53:07.398963Z DEBUG sink{component_kind="sink" component_id=test_sink component_type=gcp_cloud_storage}:request{request_id=1}:http: vector::internal_events::http_client: HTTP response. status=403 Forbidden version=HTTP/1.1 headers={"content-type": "application/xml; charset=UTF-8", "content-length": "304", "vary": "Origin", "x-guploader-uploadid": "AHxI1nNwx30Wyby0Pl62rNZXlW0g0T-3CNebXp9CcHpHYFjdrex6leHEkkYdBMYTjCMLYboTVDo", "date": "Thu, 25 Jul 2024 09:53:07 GMT", "server": "UploadServer"} body=[304 bytes]
2024-07-25T09:53:07.399046Z ERROR sink{component_kind="sink" component_id=test_sink component_type=gcp_cloud_storage}:request{request_id=1}: vector::sinks::util::retries: Not retriable; dropping the request. reason="response status: 403 Forbidden" internal_log_rate_limit=true
2024-07-25T09:53:07.399074Z TRACE sink{component_kind="sink" component_id=test_sink component_type=gcp_cloud_storage}:request{request_id=1}: vector_stream::driver: Service call succeeded. request_id=1
2024-07-25T09:53:07.399109Z ERROR sink{component_kind="sink" component_id=test_sink component_type=gcp_cloud_storage}:request{request_id=1}: vector_common::internal_event::service: Service call failed. No retries or retries exhausted. error=None request_id=1 error_type="request_failed" stage="sending" internal_log_rate_limit=true
2024-07-25T09:53:07.399139Z ERROR sink{component_kind="sink" component_id=test_sink component_type=gcp_cloud_storage}:request{request_id=1}: vector_common::internal_event::component_events_dropped: Events dropped intentional=false count=252 reason="Service call failed. No retries or retries exhausted." internal_log_rate_limit=true
2024-07-25T09:53:07.400816Z DEBUG sink{component_kind="sink" component_id=test_sink component_type=gcp_cloud_storage}: vector::topology::builder: Sink finished normally.

Example Data

No response

Additional Context

Running on Kubernetes cluster, installed via helm chart (https://github.com/vectordotdev/helm-charts/tree/develop/charts/vector)

References

Possibly related to #13221