Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Extend Papersurvey connector to support syncing an individual survey's questions, and a subset of surveys' responses #72

Open
wants to merge 3 commits into
base: feature/source-papersurvey
Choose a base branch
from

Conversation

qixotic
Copy link

@qixotic qixotic commented Feb 20, 2024

(Opening a PR just for @siddhant3030 to see the use case)

What

Using Airbyte's no-code builder, we've added the streams:
survey_questions - https://api.papersurvey.io/surveys/{{stream_partition.manual_survey}} - pulls survey meta data + questions
entries_manual - https://api.papersurvey.io/surveys/{{stream_partition.manual_survey}}/entries - supports incrementally syncing responses for a subset of survey ids.

How

Config params:
survey_ids list of ids from the user for static partitioning of the surveys to sync over. Used by both streams above right now. entries_manual is a separate stream from the one that syncs over everything (entries).

A few caveats w/ airbyte:

  • You can't allow a user to conditionally partition streams dynamically or statically in the same stream definition, resulting in the need to essentially have a duplicate stream of the same object type that gets synced, in order to have different configurability.
  • user variables in stream definitions require configuring those values when setting up the source in Airbyte. If you want to sync using an additional different set of values, you need to set up a new source for that.

…entries.

New stream types:
    survey_questions
    entries_manual
Uses partition streams with the same survey_ids supplied to the user
during source config (same approach as done with the entries_manual
stream).
Copy link

Before Merging a Connector Pull Request

Wow! What a great pull request you have here! 🎉

To merge this PR, ensure the following has been done/considered for each connector added or updated:

  • PR name follows PR naming conventions
  • Breaking changes are considered. If a Breaking Change is being introduced, ensure an Airbyte engineer has created a Breaking Change Plan.
  • Connector version has been incremented in the Dockerfile and metadata.yaml according to our Semantic Versioning for Connectors guidelines
  • You've updated the connector's metadata.yaml file any other relevant changes, including a breakingChanges entry for major version bumps. See metadata.yaml docs
  • Secrets in the connector's spec are annotated with airbyte_secret
  • All documentation files are up to date. (README.md, bootstrap.md, docs.md, etc...)
  • Changelog updated in docs/integrations/<source or destination>/<name>.md with an entry for the new version. See changelog example
  • Migration guide updated in docs/integrations/<source or destination>/<name>-migrations.md with an entry for the new version, if the version is a breaking change. See migration guide example
  • If set, you've ensured the icon is present in the platform-internal repo. (Docs)

If the checklist is complete, but the CI check is failing,

  1. Check for hidden checklists in your PR description

  2. Toggle the github label checklist-action-run on/off to re-run the checklist CI.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant