Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Deploy Airflow with DAG example out of box #1

Open
maseiler opened this issue Jun 1, 2022 · 4 comments
Open

Deploy Airflow with DAG example out of box #1

maseiler opened this issue Jun 1, 2022 · 4 comments
Assignees
Labels
new feature Adding a new functionality to the project

Comments

@maseiler
Copy link
Owner

maseiler commented Jun 1, 2022

When we us Helm to install Airflow, our own example should ship with it directly. Let everything handle automatically (no manual copying of DAG to container)

e.g.
https://airflow.apache.org/docs/helm-chart/stable/quick-start.html
https://cloud.google.com/composer/docs/run-apache-airflow-dag

@maseiler
Copy link
Owner Author

maseiler commented Jun 2, 2022

For now we can use the default examples. Unfortunately the runtime of these are very short and it would be better if we could monitor a job of at least a minute. Therefore we need a solution that allows us to add our own example easily.

@JulianLegler JulianLegler added the new feature Adding a new functionality to the project label Jun 4, 2022
@JulianLegler JulianLegler added this to the Midterm Presentation milestone Jun 4, 2022
@JulianLegler
Copy link
Collaborator

I added the default default-airflow-values.yaml file to our repo because there is already an example on how the git sync for DAGs could look like. If we decide to use this in the future.

@escapingpanda
Copy link
Collaborator

escapingpanda commented Jun 6, 2022

if we want the airflow to recognize our DAGs, we can 'kubectl cp' them to the airflow-scheduler pod in the "/opt/airflow/dags/" directory. Or we can create a storage for this purpose and assign it as the default DAG directory for airflow.

@maseiler
Copy link
Owner Author

maseiler commented Jun 6, 2022

FYI: I had to build a custom image of Airflow to include eBPF. Copying DAGs works like charm, but we have to use Docker Hub nonetheless.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
new feature Adding a new functionality to the project
Projects
None yet
Development

No branches or pull requests

3 participants