Fivetran Connection type not enabled in AWS MWAA with airflow version 2.6.3

[Pavan1837]

HI,

I am trying to setup Fivetran connection type in AWS MWAA with airflow version 2.6.3 . But i am not able see Fivetran as option in connection type drop down.

Steps:
I have installed airflow-provider-fivetran-async==1.1.2 version and verified the installation successful.

Error:

DeprecationWarning: The provider airflow-provider-fivetran-async uses hook-class-names property in provider-info and has no connection-types one. The 'hook-class-names' property has been deprecated in favour of 'connection-types' in Airflow 2.2. Use both in case you want to have backwards compatibility with Airflow < 2.2..

Could you please suggest on this.

[Pavan1837]

@sunank200 @phanikumv Can you please look into this.

[pankajastro]

Have installed this package in your AWS MWAA env?

[Pavan1837]

Yes i have installed that package and the error message is being shown after successful installation

[pankajastro]

Aha, my bad!! you have already installed it.

[pankajastro]

removing the deprecated param makes sense but this is a warning, not an error (PR to remove it #62) but I feel the connection type not visible has something to do with your installation. can try to test this PR once if this helps at least you will not get the above warning

[Pavan1837]

Could you please confirm if this package is tested in MWAA environment

[pankajastro]

Could you please confirm if this package is tested in MWAA environment

I have not used MWAA!
If this package works with Airflow and it does not work with MWAA then you should probably raise an issue with MWAA. A quick test can be done by

• clone repo
• run command make build-run from the root folder

The above command requires docker/docker-composer.

I did a quick test and I can see the options for the Fivetran connection.

[dwreeves]

I would say that this issue should probably be closed, as this is an MWAA deployment issue, not an issue specific to this package.

I will explain below the particulars of what I believe to be happening. TLDR-- MWAA is the problem, not this package. There is zero code change that this package needs to make to accommodate MWAA users. Nothing here is actionable on the part of the maintainers of this code base.

---

@Pavan1837 -- MWAA pip installs both to the scheduler instance and the webserver instance.

If you are running MWAA with "WebserverAccessMode: PRIVATE_ONLY" mode, the webserver instance will not allow for outbound traffic, meaning that pip installs to PyPI won't work on the webserver, even though it will work on the scheduler instance.

The way around this is to use a plugins.zip installation. You want to zip all wheels into a plugins.zip (pip download -r requirements.txt -d wheels/) and then add this to the requirements.txt that you install to:

--find-links /usr/local/airflow/plugins
--no-index

A guide here about that: https://docs.aws.amazon.com/mwaa/latest/userguide/best-practices-dependencies.html

You can confirm that what I am saying is the issue by going to the Airflow UI, and in the top navigation bar go to Admin -> Providers. If you do not see airflow-provider-fivetran-async, and you are running in WebserverAccessMode: PRIVATE_ONLY mode, and you also do not see other Airflow provider packages that you have installed, then I would say this is the issue.

As someone very experienced with the peculiarities of MWAA, I have a strong feeling this is the problem you are encountering.

---

At my company, we use a continuous deployment process that has a file containing the lines I mentioned above (plus --constraint constraints.txt, which we zip into plugins.zip; although this is a deliberately left blank file that we touch constraints.txt during deployment), then we append the actual requirements to that file with cat requirements.txt >> mwaa-requirements.txt. This is after zipping the dependencies. Also be aware of gotchas with binary wheels when you download the wheels (will require tweaking your frozen deps a little with some trial and error). We then have a manually triggered build that runs mwaa update-environment and points to the latest version of requirements in the s3 bucket. You can also use a lambda too that checks on upload for a change in the requirements file.

MWAA is very tedious 🙃 So much for being a "managed" service, when you have to manage so much of it yourself with custom code nonsense. Acts as an advertisement for Astronomer's own managed service... 😆

[phanikumv]

Thanks @dwreeves and @pankajastro . Closing this as per discussion above