No logs to s3 with remote log folder

[b0kky]

Hello Community!
Faced with a problem lack of logs on s3 bucket, no logs about trying to send or read to\from s3.
My config is:
Nothing in a scheduler logs, and nothing in s3 bucket.
At /opt/airflow/logs folder (in scheduler pod) i see logs, locally, but didn't see any try to upload it to s3.
What i'm doing wrong?
PS: was try to mount s3 as PV to pod's and got slap my face from python, OSError: [Errno 22] python, as i learned about python and s3 it was while script try to create a symlink "latest" to folder with logs instead of s3 not supported it (s3 csi drvier limits)
So, my single way is try to log to s3. Unfortunately i don't have in aws account no one storage with RWX support. Please help guys :)
if you think "here we go again", sorry for that, honestly watched all issues with s3 and logs, and all colleagues had a problem which shows in a log. in my case - don't see any logs about logs =\

airflow:
  executor: CeleryExecutor
  connections:
    - id: aws
      type: aws
      description: logs-connection
      login: {{ .Environment.Values.secrets.airflow.s3.access_key_id  }}
      password: {{ .Environment.Values.secrets.airflow.s3.secret_key }}
      extra: |
        { "region_name":"eu-north-1" }
  extraPipPackages: []

  config:
    AIRFLOW__WEBSERVER__EXPOSE_CONFIG: "True"
    AIRFLOW__LOGGING__REMOTE_LOGGING: "True"
    AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER: "s3://my_bucket/logs"
    AIRFLOW__LOGGING__ENCRYPT_S3_LOGS: "False"
    AIRFLOW__LOGGING__REMOTE_LOG_CONN_ID: aws
  users: []
  resources:
    limits:
      cpu: 1
      memory: 1Gi
    requests:
      cpu: 100m
      memory: 128Mi
  kubernetesPodTemplate:
    resources:
      limits:
        cpu: 1
        memory: 1Gi
      requests:
        cpu: 100m
        memory: 128Mi
  podLabels:
    app.kubernetes.io/name: {{ .Release.Name }}
  dbMigrations:
    labels:
      app.kubernetes.io/name: {{ .Release.Name }}-dbMigrations
    resources:
      limits:
        cpu: 1
        memory: 512Mi
      requests:
        cpu: 100m
        memory: 128Mi
    podLabels: 
      app.kubernetes.io/name: {{ .Release.Name }}-dbMigrations
  sync:
    labels:
      app.kubernetes.io/name: {{ .Release.Name }}-sync
    resources:
      limits:
        cpu: 1
        memory: 512Mi
      requests:
        cpu: 100m
        memory: 128Mi
    podLabels: 
      app.kubernetes.io/name: {{ .Release.Name }}-sync

flower:
  enabled: true
  labels:
    app.kubernetes.io/name: {{ .Release.Name }}-flower
  resources:
    limits:
      cpu: 1
      memory: 512Mi
    requests:
      cpu: 100m
      memory: 128Mi
  podLabels: 
    app.kubernetes.io/name: {{ .Release.Name }}-flower
  {{- if .Environment.Values.airflow.keydb.tls.enabled }}
  extraVolumeMounts:
    - name: keydb-tls
      mountPath: /etc/ssl/keydb/ca.crt
      subPath: ca.crt
      readOnly: true
  {{- end }}
  {{- if .Environment.Values.airflow.keydb.tls.enabled }}
  extraVolumes:
    - name: keydb-tls
      secret:
        secretName: airflow-keydb-tls
        defaultMode: 0644
  {{- end }}

pgbouncer:
  enabled: false
  labels:
    app.kubernetes.io/name: {{ .Release.Name }}-pgbouncer
  resources:
    limits:
      cpu: 1
      memory: 512Mi
    requests:
      cpu: 100m
      memory: 128Mi
  podLabels: 
    app.kubernetes.io/name: {{ .Release.Name }}-pgbouncer

scheduler:
  resources:
    limits:
      cpu: 1
      memory: 1Gi
    requests:
      cpu: 100m
      memory: 128Mi
  podLabels: 
    app.kubernetes.io/name: {{ .Release.Name }}-scheduler
  logCleanup:
    enabled: false

triggerer:
  resources:
    limits:
      cpu: 1
      memory: 512Mi
    requests:
      cpu: 100m
      memory: 128Mi
  podLabels: 
    app.kubernetes.io/name: {{ .Release.Name }}-triggerer

web:
  webserverConfig:
    stringOverride: |
      from flask_appbuilder.security.manager import AUTH_LDAP
      AUTH_TYPE = AUTH_LDAP
      AUTH_LDAP_SERVER = "ldaps:"
      AUTH_LDAP_USE_TLS = False
      AUTH_USER_REGISTRATION = True  # allow users who are not already in the FAB DB
      AUTH_USER_REGISTRATION_ROLE = "Admin"  # this role will be given in addition to any AUTH_ROLES_MAPPING
      AUTH_LDAP_FIRSTNAME_FIELD = "givenName"
      AUTH_LDAP_LASTNAME_FIELD = "sn"
      AUTH_LDAP_EMAIL_FIELD = "mail"  # if null in LDAP, email is set to: "{username}@email.notfound"
      AUTH_LDAP_SEARCH = ",DC=com"  # the LDAP search base (if non-empty, a search will ALWAYS happen)
      AUTH_LDAP_UID_FIELD = "sAMAccountName"  # the username field
      AUTH_LDAP_BIND_USER = "CN="
      AUTH_LDAP_BIND_PASSWORD = {{ .Environment.Values.secrets.airflow.ldap.pass | quote }}
      AUTH_LDAP_TLS_DEMAND = True
      AUTH_LDAP_ALLOW_SELF_SIGNED = True
      AUTH_LDAP_GROUP_FIELD = "memberOf"
      AUTH_ROLES_SYNC_AT_LOGIN = True
      PERMANENT_SESSION_LIFETIME = 1800
  resources:
    limits:
      cpu: 1
      memory: 2Gi
    requests:
      cpu: 100m
      memory: 128Mi
  podLabels: 
    app.kubernetes.io/name: {{ .Release.Name }}-web

workers:
  resources:
    limits:
      memory: 2Gi
    requests:
      cpu: 100m
      memory: 128Mi
  podLabels: 
    app.kubernetes.io/name: {{ .Release.Name }}-workers
  logCleanup:
    enabled: false
  {{- if .Environment.Values.airflow.keydb.tls.enabled }}
  extraVolumeMounts:
    - name: keydb-tls
      mountPath: /etc/ssl/keydb/ca.crt
      subPath: ca.crt
      readOnly: true
  {{- end }}
  {{- if .Environment.Values.airflow.keydb.tls.enabled }}
  extraVolumes:
    - name: keydb-tls
      secret:
        secretName: airflow-keydb-tls
        defaultMode: 0644
  {{- end }}
redis:
  enabled: false

externalRedis:
  host: airflow-keydb
  port: 6379
  password: {{ .Environment.Values.secrets.airflow.keydb.password }}
  properties: "?ssl_cert_reqs=required&ssl_ca_certs=/etc/ssl/keydb/ca.crt"


postgresql:
  enabled: false
externalDatabase:
  type: postgres
  host: dgairflow-postgres-postgresql
  port: 5432
  database: airflow
  userSecret: "airflow.dgairflow-postgres-postgresql.credentials.postgresql.acid.zalan.do"
  userSecretKey: "username"
  passwordSecret: "airflow.dgairflow-postgres-postgresql.credentials.postgresql.acid.zalan.do"
  passwordSecretKey: "password"

ingress:
  enabled: true
  web:
    annotations:
    path: ""
    host: dg-airflow.{{ .Environment.Values.cluster_name }}
    ingressClassName: internal-only
  flower:
    annotations:
    path: ""
    host: dg-airflow-flower.{{ .Environment.Values.cluster_name }}
    ingressClassName: internal-only

network-policies:
  nameOverride: {{ .Release.Name }}
  allow_any_inside_namespace:
    - podSelector: {}  # TODO: create unique rules for each pod
  allow_raw:
    - metadata:
        name: {{ .Release.Name }}-allow-internal-raw
      spec:
        podSelector:
          matchLabels:
            app.kubernetes.io/name: airflow-web
        ingress:
          - ports:
              - port: 8080
                protocol: TCP
          - from:
              - ipBlock:
                  cidr: {{ .Environment.Values.vpc_cidrs }}
        policyTypes:
          - Ingress
    - metadata:
        name: {{ .Release.Name }}-flower-allow-internal-raw
      spec:
        podSelector:
          matchLabels:
            app.kubernetes.io/name: airflow-flower
        ingress:
          - ports:
              - port: 5555
                protocol: TCP
          - from:
              - ipBlock:
                  cidr: {{ .Environment.Values.vpc_cidrs }}
        policyTypes:
          - Ingress

extraManifests:
  - |
    apiVersion: v1
    kind: PersistentVolume
    metadata:
      name: airflow-s3-dags
      namespace:
    spec:
      capacity:
        storage: 120Gi # ignored, required
      accessModes:
        - ReadWriteMany # supported options: ReadWriteMany / ReadOnlyMany
      mountOptions:
        - allow-delete
        - allow-other
        - uid=50000
      csi:
        driver: s3.csi.aws.com # required
        volumeHandle: s3-csi-driver-volume-dags
        volumeAttributes:
          bucketName: k8s-sto-aws-airflow-dags
  - |
    apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      name: airflow-s3-dags
      namespace:
    spec:
      accessModes:
        - ReadWriteMany # supported options: ReadWriteMany / ReadOnlyMany
      storageClassName: "" # required for static provisioning
      resources:
        requests:
          storage: 120Gi # ignored, required
      volumeName: airflow-s3-dags
 
dags:
  path:  /opt/airflow/dags
  persistence:
    enabled: true
    existingClaim: "airflow-s3-dags"
    subPath: ""
    storageClass: ""
    accessMode: ReadOnlyMany
    size: 120Gi
```

[b0kky]

resolve it:
this block is set up your connection config to aws, yeap, but it called assume role for access to s3 bucket, and my test was successfully, because it will check only connect to aws (as i think under hood it call aws cli sts-get-identity and return answer with role and KEY ID). But it not using provisioned access-key and secret-key for connect to s3.

  connections:
    - id: aws
      type: aws
      description: logs-connection
      login: {{ .Environment.Values.secrets.airflow.s3.access_key_id  }}
      password: {{ .Environment.Values.secrets.airflow.s3.secret_key }}

workaround - use extraenv instead of login\password

  extraEnv:
  - name: AWS_ACCESS_KEY_ID
    value: {{ .Environment.Values.secrets.airflow.s3.access_key_id | quote }}
  - name: AWS_SECRET_ACCESS_KEY
    value: {{ .Environment.Values.secrets.airflow.s3.secret_key | quote  }}