-
Notifications
You must be signed in to change notification settings - Fork 7
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fivetran Connection type not enabled in AWS MWAA with airflow version 2.6.3 #61
Comments
@sunank200 @phanikumv Can you please look into this. |
Have installed this package in your AWS MWAA env? |
Yes i have installed that package and the error message is being shown after successful installation |
Aha, my bad!! you have already installed it. |
removing the deprecated param makes sense but this is a warning, not an error (PR to remove it #62) but I feel the connection type not visible has something to do with your installation. can try to test this PR once if this helps at least you will not get the above warning |
Could you please confirm if this package is tested in MWAA environment |
I have not used MWAA!
The above command requires docker/docker-composer. I did a quick test and I can see the options for the Fivetran connection. |
I would say that this issue should probably be closed, as this is an MWAA deployment issue, not an issue specific to this package. I will explain below the particulars of what I believe to be happening. TLDR-- MWAA is the problem, not this package. There is zero code change that this package needs to make to accommodate MWAA users. Nothing here is actionable on the part of the maintainers of this code base. @Pavan1837 -- MWAA pip installs both to the scheduler instance and the webserver instance. If you are running MWAA with "WebserverAccessMode: PRIVATE_ONLY" mode, the webserver instance will not allow for outbound traffic, meaning that pip installs to PyPI won't work on the webserver, even though it will work on the scheduler instance. The way around this is to use a
A guide here about that: https://docs.aws.amazon.com/mwaa/latest/userguide/best-practices-dependencies.html You can confirm that what I am saying is the issue by going to the Airflow UI, and in the top navigation bar go to As someone very experienced with the peculiarities of MWAA, I have a strong feeling this is the problem you are encountering. At my company, we use a continuous deployment process that has a file containing the lines I mentioned above (plus MWAA is very tedious 🙃 So much for being a "managed" service, when you have to manage so much of it yourself with custom code nonsense. Acts as an advertisement for Astronomer's own managed service... 😆 |
Thanks @dwreeves and @pankajastro . Closing this as per discussion above |
HI,
I am trying to setup Fivetran connection type in AWS MWAA with airflow version 2.6.3 . But i am not able see Fivetran as option in connection type drop down.
Steps:
I have installed airflow-provider-fivetran-async==1.1.2 version and verified the installation successful.
Error:
DeprecationWarning: The provider airflow-provider-fivetran-async uses
hook-class-names
property in provider-info and has noconnection-types
one. The 'hook-class-names' property has been deprecated in favour of 'connection-types' in Airflow 2.2. Use both in case you want to have backwards compatibility with Airflow < 2.2..Could you please suggest on this.
The text was updated successfully, but these errors were encountered: