Add option to provide Prefect API key to created jobs as a Kubernetes secret

[desertaxle]

Adds an option to have a worker store its Prefect API key as a Kubernetes secret and provide the secret to created jobs. Prevents the exposure of a worker's Prefect API key by inspecting the manifests of the jobs it creates. The worker will clean up any created secrets upon shutdown.

This feature is opt-in because it requires the worker to have the necessary permissions to create secrets. Once it is battle-tested, this feature will be updates to be the default.

Closes PrefectHQ/prefect#10716

Example

To activate, set the PREFECT_KUBERNETES_WORKER_STORE_PREFECT_API_IN_SECRET environment variable:

$ export PREFECT_KUBERNETES_WORKER_STORE_PREFECT_API_IN_SECRET="true"

After running a job, you'll see a new secret created:

$ kubectl get secrets
kubernetesworker-c7d5551b-e0b9-49b1-a592-65c7-api-key   Opaque   1      96s

If you inspect the job, you'll see the same secret used:

$ kubectl get job papaya-lion-cwl2j -o yaml
apiVersion: batch/v1
kind: Job
spec:
  template:
    spec:
      containers:
      - args:
        - prefect
        - flow-run
        - execute
        env:
        - name: PREFECT_API_URL
          value: https://api.prefect.cloud/api/accounts/a0f10033-f08a-42f8-9b8d-b7a7d4b4b3c1/workspaces/e4b1c666-a4ab-42c7-8172-48d43966e691
        - name: PREFECT_API_KEY
          valueFrom:
            secretKeyRef:
              key: value
              name: kubernetesworker-c7d5551b-e0b9-49b1-a592-65c7-api-key

After shutting down the worker, the created secret should be gone:

$ kubectl get secrets 
No resources found in default namespace.

Screenshots

Checklist

• References any related issue by including "Closes #" or "Closes ".
  • If no issue exists and your change is not a small fix, please create an issue first.
• Includes tests or only affects documentation.
• Passes pre-commit checks.
  • Run pre-commit install && pre-commit run --all locally for formatting and linting.
• Includes screenshots of documentation updates.
  • Run mkdocs serve view documentation locally.
• Summarizes PR's changes in CHANGELOG.md

[chrisguidry]

That's slick!

[taylor-curran]

Comment from user -- medium priority:

PrefectHQ/prefect#10716 (comment)