Feat: Use GCS public bucket to get manifest yaml for connectors; adds support for pinning versions and getting prior versions #394
Conversation
…ersions and getting prior versions
|
Warning Rate limit exceeded@aaronsteers has exceeded the limit for the number of commits or files that can be reviewed per hour. Please wait 4 minutes and 17 seconds before requesting another review. How to resolve this issue?After the wait time has elapsed, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout. Please see our FAQ for further information. WalkthroughWalkthroughThe changes involve the Changes
Possibly related PRs
Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media? TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
|
/fix-pr
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
Outside diff range and nitpick comments (2)
airbyte/_executors/util.py (2)
28-32: LGTM! Consider adding type hints for constants?The new constants look good and their purpose is clear. Nice job on structuring the URL for versioned manifests!
Quick thought: wdyt about adding type hints to these constants? It might make the code a tad more self-documenting. Something like:
VERSION_LATEST: str = "latest" DEFAULT_MANIFEST_URL: str = ( "https://storage.googleapis.com/prod-airbyte-cloud-connector-metadata-service/" "metadata/airbyte/{source_name}/{version}/manifest.yaml" )
35-39: Loving the new version support! How about enhancing the docstring?The updated function signature with the optional
versionparameter is spot on! It aligns perfectly with the new URL structure.Quick suggestion: wdyt about adding a bit more detail to the docstring? Maybe something like:
def _try_get_source_manifest( source_name: str, manifest_url: str | None, version: str | None = None, ) -> dict: """Try to get a source manifest from a URL. If the URL is not provided, we'll try the default URL in the public GCS bucket. If version is not provided, it defaults to 'latest'. Args: source_name: The name of the source. manifest_url: Optional custom URL for the manifest. version: Optional version string. Defaults to 'latest' if not provided. Returns: dict: The parsed manifest as a dictionary. Raises: PyAirbyteInputError: If `source_name` is `None`. HTTPError: If fetching the URL was unsuccessful. AirbyteConnectorInstallationError: If parsing the YAML fails. """This gives a bit more context on the parameters and return value. What do you think?
Also applies to: 44-46
There was a problem hiding this comment.
Actionable comments posted: 1
Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Files selected for processing (2)
- airbyte/_executors/util.py (1 hunks)
- airbyte/sources/registry.py (1 hunks)
Files skipped from review as they are similar to previous changes (1)
- airbyte/_executors/util.py
Summary by CodeRabbit
New Features
Bug Fixes
source_nameis always required, preventing potential errors.Refactor
Chores