Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dags changes from Azure blob fuse PVC not syncing in Airflow Schedular, triggerer, flower #898

Open
2 tasks done
kanika-bhargav opened this issue Nov 29, 2024 · 0 comments
Labels
kind/bug kind - things not working properly

Comments

@kanika-bhargav
Copy link

Checks

Chart Version

2.10.2

Kubernetes Version

Client Version: v1.29.2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.15

Helm Version

version.BuildInfo{Version:"v3.15.2", GitCommit:"1a500d5625419a524fdae4b33de351cc4f58ec35", GitTreeState:"clean", GoVersion:"go1.22.4"}

Description

I have deployed azure blob fuse enabled PVC to load DAG definitions. I have a CI/CD pipeline to push the changes from git branch to azure blob container. We frequently make changes to DAG definition for testing and push to azure blob. The changes are reflected on az blob. But these changes are not syncing on airflow components except webserver.

Relevant Logs

Dont see anything in logs related to this

Custom Helm Values

airflow: 
   config:
      AIRFLOW__CORE__DAGBAG_IMPORT_TIMEOUT: '120.0'
      AIRFLOW__CORE__MIN_SERIALIZED_DAG_FETCH_INTERVAL: '30'
      AIRFLOW__CORE__MIN_SERIALIZED_DAG_UPDATE_INTERVAL: '30'
      AIRFLOW__SCHEDULER__DAG_DIR_LIST_INTERVAL: '120'
      AIRFLOW__SCHEDULER__MIN_FILE_PROCESS_INTERVAL: '60'
      AIRFLOW__SCHEDULER__STALE_DAG_THRESHOLD: '90'
      AIRFLOW__SCHEDULER__ZOMBIE_DETECTION_INTERVAL: '60.0'
      AIRFLOW__SCHEDULER__PARSING_PROCESSES: '3'
      AIRFLOW__USAGE_DATA_COLLECTION__ENABLED: "False"
      AIRFLOW__API__ACCESS_CONTROL_ALLOW_ORIGINS: 'content-type, origin, authorization, accept'
      AIRFLOW__API__ACCESS_CONTROL_ALLOW_HEADERS: 'GET, POST, PATCH, DELETE'
      AIRFLOW__API__ACCESS_CONTROL_ALLOW_METHODS: "*"
      # auth backend (comma separated value for api auth. Default is session auth)
      AIRFLOW__API__AUTH_BACKENDS: airflow.api.auth.backend.session, airflow.api.auth.backend.basic_auth


  ###################################
  ## CONFIG | Airflow DAGs
  ###################################
  dags:
    ## the airflow dags folder
    path: /opt/airflow/dags
    persistence:
      enabled: true
      existingClaim: "pvc-airflow-dags"
      accessMode: ReadWriteMany
@kanika-bhargav kanika-bhargav added the kind/bug kind - things not working properly label Nov 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug kind - things not working properly
Projects
None yet
Development

No branches or pull requests

1 participant