secret() function seems inconsistent with sourcing secrets

[pypeaday]

Describe the issue

I can sucesfully run 1 flow but another very similar one fails. It looks to me to be an issue with passing a secret down because if I hardcode the values in my flows/scripts then they work fine.

Successful flow

I can create a bucket just fine on my minio instance

I have a .env_encoded like the docs say, it looks like this

SECRET_AWS_ACCESS_KEY_ID=base64encoded_key
SECRET_AWS_SECRET_ACCESS_KEY=base64encoded_access_key
SECRET_AWS_ENDPOINT_URL_S3=base64encoded_minio_url

I docker compose up -d and make a flow

---
id: s3_compatible_bucket
namespace: homelab.dev
tasks:
- id: create_bucket
  type: io.kestra.plugin.minio.CreateBucket
  accessKeyId: "{{ secret('AWS_ACCESS_KEY_ID') }}"
  secretKeyId: "{{ secret('AWS_SECRET_ACCESS_KEY') }}"
  endpoint: "{{ secret('AWS_ENDPOINT_URL_S3') }}"
  bucket: "kestra-test-bucket"

this runs great

Failure

In the same encoded env file I have

SECRET_OPENAI_API_KEY=base64encoded_apikey

Then the flow that failes

id: openai
namespace: homelab.dev
tasks:
  - id: prompt
    type: io.kestra.plugin.openai.ChatCompletion
    # this works if I hardcode... idk what's wrong with the secrets... they work in the create bucket flow
    apiKey: "{{ secret('OPENAI_API_KEY') }}"

    model: gpt-4
    prompt: Explain in one sentence why data engineers build data pipelines
  - id: use_output
    type: io.kestra.plugin.core.log.Log
    message: "{{ outputs.prompt.choices | jq('.[].message.content') | first }}"

When I run this the error is that I didn't supply an apiKey, but if I hardcode it in the flow then it works

Environment

• Kestra Version: 0.19.5
• Operating System (OS/Docker/Kubernetes): Docker
• Java Version (if you don't run kestra in Docker):

[anna-geller]

It must be an issue in the OpenAI plugin, we'll investigate, thx for the report! 👍 if you see it with any other plugin, keep us posted. we're working on making all properties dynamic (without losing strong typing) in the next release so this should get better soon

[pypeaday]

Well I actually do have a couple more very similar issues that I'll outline here for documentation's sake

Nextcloud

Testing simple nextcloud integration to download (and then eventually also upload) a file I was not able to pass my credentials to my python script

id: nextcloudTest
namespace: homelab.dev
tasks:
- id: debugging
  type: io.kestra.plugin.core.log.Log
  message: My url "{{ secret('NC_URL') }}"
- id: python
  type: io.kestra.plugin.scripts.shell.Commands
  containerImage: python:3.11-slim
  namespaceFiles:
    enabled: true
    include:
      - nextcloud_test.py
  beforeCommands:
  - python -m pip install webdavclient3
  commands:
  - env
  - python nextcloud_test.py
  env:
    NC_URL: "{{ secret('NC_URL') }}"
    NC_PASSWORD: "{{ secret('NC_PASSWORD') }}"
    NC_USER: "{{ secret('NC_USER') }}"

And in my .env_encoded

SECRET_NC_PASSWORD=base64encoded_password
SECRET_NC_URL=base64encoded_url
SECRET_NC_USER=base64encoded_user

I can see that these are set but masked in the logs - and I also tried to print them from the pyton script but they were ***masked*** so I'm not sure how to confirm if it's working or not. But the same thing holds true that if I just hardcode those credentials in my python script then it works fine
The script is trivial and I doubt is related to the problem but it looks like this

import os
from pathlib import Path

from webdav3.client import Client

# requirements
# ---
# webdavclient3


PASSWORD = os.environ.get("NC_PASSWORD")
URL = os.environ.get("NC_URL")
USER = os.environ.get("NC_USER")

options = {
    "webdav_hostname": URL,
    "webdav_login": USER,
    "webdav_password": PASSWORD,
}
print(options)
client = Client(options)
client.verify = True
file: Path = Path("Readme.md")
client.download_file(remote_path=f"/{file}", local_path=str(file))
# client.upload_file(remote_path=f"/updloaded-{file.name}", local_path=str(file))

In this case the python error is webdav3.exceptions.ConnectionException: No connection adapters were found for '**masked***************************************************/Readme.md' However that "No connection adapters" has means before that the protocol in the URL was missing, but again if I hardcode the value then it works fine so it makes me think there's an issue with the secret() function in kestra's templating

[pypeaday]

I also had issues with a terraform workflow but I kind of stupidely removed the docker volume and didn't back that workflow up so I don't have it or the log immediately handy... the issue was with setting up an s3 provider for remote state storage. I wanted to use my minio instance and that's compatible with the aws cli/boto sdk with passing the AWS secrets, just like in the first post, into the environment - then aws <commands> can work with local minio, it's great.

But I could never get the terraform flow to get through terraform init because it woudl fail to setup the s3 provider. However, when I run the same terraform code from my desktop it works fine - so again it feels to me like an issue with the secret() handling of the variables.

If a flow and logs would be really valuable I can try to recreate this in the next few days