Splitting AWS to 2 modules, cloud-common and AWS #993
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
davidsloan
changed the title
stheppi
approved these changes
Sep 29, 2023
davidsloan
force-pushed
the
feat/split-s3-cloud-and-aws
branch
from
f5e972d to
5825694
Compare
davidsloan
force-pushed
the
feat/split-s3-cloud-and-aws
branch
from
5825694 to
31ac556
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In order to develop other cloud connectors it would be nice to reuse the non-AWS-specific components of the source and sink.
(I am attempting to break these changes up into regular PRs so as to avoid what happened last time with the master branch moving on.)
A new module has been introduced, cloud-common, with the base source, sink and configuration components. It is hoped that we will be able to replace the transport layers in the AWS/cloud submodules and therefore be able to support other cloud connectors.
There will be more complexity than this but this is a first stab of isolating what could live in a common module.
Behaviour is unchanged with the exception of removing support for the yaml profiles.
Because the connector prefix will change according to the connector, there are many changes designed to facilitate multiple connector prefixes (for example, introducing a mixin for using the key).
Also as a first iteration
S3Locationhas been renamed toCloudLocation. It may be necessary to introduce an interface for these later but will try it like this for now. For validation aCloudLocationValidatoris implicitly supplied, sourced from the ConnectorTask. In the code this has little impact but there is more impact in the tests.