Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for running dbt tasks in AWS EKS #944

Merged
merged 24 commits into from
May 22, 2024

Conversation

VolkerSchiewe
Copy link
Contributor

@VolkerSchiewe VolkerSchiewe commented May 7, 2024

Description

We are using MWAA in combination with EKS so that all our dags in airflow are running in our EKS. We would like to use the same setup with cosmos.

What changes?

  • New AwsEksOperator classes (inheriting from KubernetesOperators) - Based on the original EksOperator
  • Tests
  • Adjusted documentation

Related Issue(s)

Breaking Change?

No - only an additional feature

Checklist

  • I have made corresponding changes to the documentation (if required)
  • I have added tests that prove my fix is effective or that my feature works

@VolkerSchiewe VolkerSchiewe requested a review from a team as a code owner May 7, 2024 08:12
@dosubot dosubot bot added the size:L This PR changes 100-499 lines, ignoring generated files. label May 7, 2024
Copy link

netlify bot commented May 7, 2024

Deploy Preview for sunny-pastelito-5ecb04 ready!

Name Link
🔨 Latest commit 42fa4c4
🔍 Latest deploy log https://app.netlify.com/sites/sunny-pastelito-5ecb04/deploys/664de8fe8711f20008ea8fa5
😎 Deploy Preview https://deploy-preview-944--sunny-pastelito-5ecb04.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

@dosubot dosubot bot added area:docs Relating to documentation, changes, fixes, improvement area:execution Related to the execution environment/mode, like Docker, Kubernetes, Local, VirtualEnv, etc area:testing Related to testing, like unit tests, integration tests, etc labels May 7, 2024
@tatiana tatiana added this to the 1.5.0 milestone May 7, 2024
Copy link
Collaborator

@tatiana tatiana left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the contribution, @VolkerSchiewe !
Please, could you address the tests currently failing?

@VolkerSchiewe
Copy link
Contributor Author

Thanks for the contribution, @VolkerSchiewe ! Please, could you address the tests currently failing?

Hi @tatiana thanks for getting back to me! I was already working on it, but I ran into this issue: apache/airflow#39103

Seems to be affecting the latest version of the amazon airflow provider. I already tried with pinning an older version of xmlsec1 with no success :/

cosmos/constants.py Outdated Show resolved Hide resolved
Copy link
Contributor

@pankajkoti pankajkoti left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a great contribution @VolkerSchiewe . I think we're close and soon can merge this.

Would it be possible to also paste a snapshot of a successful DAG run using AWS EKS operator?

cosmos/operators/eks.py Outdated Show resolved Hide resolved
pyproject.toml Outdated Show resolved Hide resolved
@VolkerSchiewe
Copy link
Contributor Author

Would it be possible to also paste a snapshot of a successful DAG run using AWS EKS operator?

Hope this is what you mean @pankajkoti (the operator naming is still before the renaming though)
image

Copy link
Contributor

@pankajkoti pankajkoti left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Appreciate your contribution @VolkerSchiewe 👏🏽

@dosubot dosubot bot added the lgtm This PR has been approved by a maintainer label May 22, 2024
Copy link
Collaborator

@tatiana tatiana left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks great, thanks a lot for iterating and improving it so quickly, @VolkerSchiewe !

@tatiana tatiana changed the title Support for EKS operator Support for AWS EKS operator May 22, 2024
@tatiana tatiana changed the title Support for AWS EKS operator Support for running dbt tasks in AWS EKS May 22, 2024
Copy link
Contributor

@pankajastro pankajastro left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks great 👍

docs/getting_started/execution-modes.rst Outdated Show resolved Hide resolved
@pankajkoti pankajkoti merged commit cb2a27a into astronomer:main May 22, 2024
63 checks passed
@pankajkoti pankajkoti mentioned this pull request Jun 27, 2024
tatiana pushed a commit that referenced this pull request Jun 27, 2024
New Features

* Speed up ``LoadMode.DBT_LS`` by caching dbt ls output in Airflow
Variable by @tatiana in #1014
* Support to cache profiles created via ``ProfileMapping`` by
@pankajastro in #1046
* Support for running dbt tasks in AWS EKS in #944 by @VolkerSchiewe
* Add Clickhouse profile mapping by @roadan and @pankajastro in #353 and
#1016
* Add node config to TaskInstance Context by @linchun3 in #1044

Bug fixes

* Support partial parsing when cache is disabled by @tatiana in #1070
* Fix disk permission error in restricted env by @pankajastro in #1051
* Add CSP header to iframe contents by @dwreeves in #1055
* Stop attaching log adaptors to root logger to reduce logging costs by
@glebkrapivin in #1047

Enhancements

* Support ``static_index.html`` docs by @dwreeves in #999
* Support deep linking dbt docs via Airflow UI by @dwreeves in #1038
* Add ability to specify host/port for Snowflake connection by @whummer
in #1063

Docs

* Fix rendering for env ``enable_cache_dbt_ls`` by @pankajastro in #1069

Others

* Update documentation for DbtDocs generator by @arjunanan6 in #1043
* Use uv in CI by @dwreeves in #1013
* Cache hatch folder in the CI by @tatiana in #1056
* Change example DAGs to use ``example_conn`` as opposed to
``airflow_db`` by @tatiana in #1054
* Mark plugin integration tests as integration by @tatiana in #1057
* Ensure compliance with linting rule D300 by using triple quotes for
docstrings by @pankajastro in #1049
* Pre-commit hook updates in #1039, #1050, #1064
* Remove duplicates in changelog by @jedcunningham in #1068
@tatiana tatiana mentioned this pull request Jun 27, 2024
arojasb3 pushed a commit to arojasb3/astronomer-cosmos that referenced this pull request Jul 14, 2024
## Description
We are using MWAA in combination with EKS so that all our dags in
airflow are running in our EKS. We would like to use the same setup with
cosmos.

### What changes?
- New AwsEksOperator classes (inheriting from KubernetesOperators) -
Based on the original
[EksOperator](https://github.com/apache/airflow/blob/main/airflow/providers/amazon/aws/operators/eks.py#L995)
- Tests
- Adjusted documentation

## Related Issue(s)

-

## Breaking Change?
No - only an additional feature

## Checklist

- [x] I have made corresponding changes to the documentation (if
required)
- [x] I have added tests that prove my fix is effective or that my
feature works

---------

Co-authored-by: Pankaj Koti <[email protected]>
arojasb3 pushed a commit to arojasb3/astronomer-cosmos that referenced this pull request Jul 14, 2024
New Features

* Speed up ``LoadMode.DBT_LS`` by caching dbt ls output in Airflow
Variable by @tatiana in astronomer#1014
* Support to cache profiles created via ``ProfileMapping`` by
@pankajastro in astronomer#1046
* Support for running dbt tasks in AWS EKS in astronomer#944 by @VolkerSchiewe
* Add Clickhouse profile mapping by @roadan and @pankajastro in astronomer#353 and
astronomer#1016
* Add node config to TaskInstance Context by @linchun3 in astronomer#1044

Bug fixes

* Support partial parsing when cache is disabled by @tatiana in astronomer#1070
* Fix disk permission error in restricted env by @pankajastro in astronomer#1051
* Add CSP header to iframe contents by @dwreeves in astronomer#1055
* Stop attaching log adaptors to root logger to reduce logging costs by
@glebkrapivin in astronomer#1047

Enhancements

* Support ``static_index.html`` docs by @dwreeves in astronomer#999
* Support deep linking dbt docs via Airflow UI by @dwreeves in astronomer#1038
* Add ability to specify host/port for Snowflake connection by @whummer
in astronomer#1063

Docs

* Fix rendering for env ``enable_cache_dbt_ls`` by @pankajastro in astronomer#1069

Others

* Update documentation for DbtDocs generator by @arjunanan6 in astronomer#1043
* Use uv in CI by @dwreeves in astronomer#1013
* Cache hatch folder in the CI by @tatiana in astronomer#1056
* Change example DAGs to use ``example_conn`` as opposed to
``airflow_db`` by @tatiana in astronomer#1054
* Mark plugin integration tests as integration by @tatiana in astronomer#1057
* Ensure compliance with linting rule D300 by using triple quotes for
docstrings by @pankajastro in astronomer#1049
* Pre-commit hook updates in astronomer#1039, astronomer#1050, astronomer#1064
* Remove duplicates in changelog by @jedcunningham in astronomer#1068
tatiana pushed a commit that referenced this pull request Jul 17, 2024
New Features

* Speed up ``LoadMode.DBT_LS`` by caching dbt ls output in Airflow
Variable by @tatiana in #1014
* Support to cache profiles created via ``ProfileMapping`` by
@pankajastro in #1046
* Support for running dbt tasks in AWS EKS in #944 by @VolkerSchiewe
* Add Clickhouse profile mapping by @roadan and @pankajastro in #353 and
#1016
* Add node config to TaskInstance Context by @linchun3 in #1044

Bug fixes

* Support partial parsing when cache is disabled by @tatiana in #1070
* Fix disk permission error in restricted env by @pankajastro in #1051
* Add CSP header to iframe contents by @dwreeves in #1055
* Stop attaching log adaptors to root logger to reduce logging costs by
@glebkrapivin in #1047

Enhancements

* Support ``static_index.html`` docs by @dwreeves in #999
* Support deep linking dbt docs via Airflow UI by @dwreeves in #1038
* Add ability to specify host/port for Snowflake connection by @whummer
in #1063

Docs

* Fix rendering for env ``enable_cache_dbt_ls`` by @pankajastro in #1069

Others

* Update documentation for DbtDocs generator by @arjunanan6 in #1043
* Use uv in CI by @dwreeves in #1013
* Cache hatch folder in the CI by @tatiana in #1056
* Change example DAGs to use ``example_conn`` as opposed to
``airflow_db`` by @tatiana in #1054
* Mark plugin integration tests as integration by @tatiana in #1057
* Ensure compliance with linting rule D300 by using triple quotes for
docstrings by @pankajastro in #1049
* Pre-commit hook updates in #1039, #1050, #1064
* Remove duplicates in changelog by @jedcunningham in #1068

(cherry picked from commit 18d2c90)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:docs Relating to documentation, changes, fixes, improvement area:execution Related to the execution environment/mode, like Docker, Kubernetes, Local, VirtualEnv, etc area:testing Related to testing, like unit tests, integration tests, etc cosmos epic-assigned lgtm This PR has been approved by a maintainer size:L This PR changes 100-499 lines, ignoring generated files. triage-needed Items need to be reviewed / assigned to milestone
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants