Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: refactored adding support for RETL and Profiles #35

Merged
merged 5 commits into from
Oct 9, 2024

Conversation

RanjeetMishra
Copy link
Collaborator

@RanjeetMishra RanjeetMishra commented Oct 3, 2024

Description

  1. Refactored to have explicit operators for Profiles and Reverse ETL.
  2. Updated ReadME.
  3. Added version in headers for API requests.

Testing

  1. Tested with airflow.
*** Reading local file: /Users/ranjeetmishra/airflow/logs/dag_id=rudderstack-sample/run_id=scheduled__2024-10-03T13:01:39.570003+00:00/task_id=rs_trigger_sync/attempt=1.log
[2024-10-03, 13:06:50 UTC] {taskinstance.py:1084} INFO - Dependencies all met for dep_context=non-requeueable deps ti=<TaskInstance: rudderstack-sample.rs_trigger_sync scheduled__2024-10-03T13:01:39.570003+00:00 [queued]>
[2024-10-03, 13:06:50 UTC] {taskinstance.py:1084} INFO - Dependencies all met for dep_context=requeueable deps ti=<TaskInstance: rudderstack-sample.rs_trigger_sync scheduled__2024-10-03T13:01:39.570003+00:00 [queued]>
[2024-10-03, 13:06:50 UTC] {taskinstance.py:1282} INFO - 
--------------------------------------------------------------------------------
[2024-10-03, 13:06:50 UTC] {taskinstance.py:1283} INFO - Starting attempt 1 of 2
[2024-10-03, 13:06:50 UTC] {taskinstance.py:1284} INFO - 
--------------------------------------------------------------------------------
[2024-10-03, 13:06:50 UTC] {taskinstance.py:1303} INFO - Executing <Task(RudderstackRETLOperator): rs_trigger_sync> on 2024-10-03 13:01:39.570003+00:00
[2024-10-03, 13:06:50 UTC] {standard_task_runner.py:55} INFO - Started process 15951 to run task
[2024-10-03, 13:06:50 UTC] {standard_task_runner.py:82} INFO - Running: ['airflow', 'tasks', 'run', 'rudderstack-sample', 'rs_trigger_sync', 'scheduled__2024-10-03T13:01:39.570003+00:00', '--job-id', '29', '--raw', '--subdir', 'DAGS_FOLDER/retl_dag.py', '--cfg-path', '/var/folders/j3/0xg77yp13n79lpysgzml60780000gn/T/tmprlpdlbwc']
[2024-10-03, 13:06:50 UTC] {standard_task_runner.py:83} INFO - Job 29: Subtask rs_trigger_sync
[2024-10-03, 13:06:50 UTC] {task_command.py:388} INFO - Running <TaskInstance: rudderstack-sample.rs_trigger_sync scheduled__2024-10-03T13:01:39.570003+00:00 [running]> on host ranjeets-macbook-pro.local
[2024-10-03, 13:06:50 UTC] {taskinstance.py:1509} INFO - Exporting the following env vars:
[email protected]
AIRFLOW_CTX_DAG_OWNER=airflow
AIRFLOW_CTX_DAG_ID=rudderstack-sample
AIRFLOW_CTX_TASK_ID=rs_trigger_sync
AIRFLOW_CTX_EXECUTION_DATE=2024-10-03T13:01:39.570003+00:00
AIRFLOW_CTX_TRY_NUMBER=1
AIRFLOW_CTX_DAG_RUN_ID=scheduled__2024-10-03T13:01:39.570003+00:00
[2024-10-03, 13:06:50 UTC] {rudderstack.py:165} INFO - Triggering sync for retl connection id: 2cwRi0iuL5XMnnZCRj9BnPwUxau
[2024-10-03, 13:06:50 UTC] {base.py:73} INFO - Using connection ID 'rudderstack_default' for task execution.
[2024-10-03, 13:06:50 UTC] {base.py:73} INFO - Using connection ID 'rudderstack_default' for task execution.
[2024-10-03, 13:06:51 UTC] {rudderstack.py:65} INFO - poll and wait for sync to finish for retl-connectionId: 2cwRi0iuL5XMnnZCRj9BnPwUxau, syncId: crv9dqt52mjsic2pne00
[2024-10-03, 13:06:51 UTC] {base.py:73} INFO - Using connection ID 'rudderstack_default' for task execution.
[2024-10-03, 13:06:51 UTC] {base.py:73} INFO - Using connection ID 'rudderstack_default' for task execution.
[2024-10-03, 13:06:52 UTC] {rudderstack.py:196} INFO - Polled status for syncId: crv9dqt52mjsic2pne00 for retl connection: 2cwRi0iuL5XMnnZCRj9BnPwUxau, status: running
[2024-10-03, 13:07:02 UTC] {base.py:73} INFO - Using connection ID 'rudderstack_default' for task execution.
[2024-10-03, 13:07:02 UTC] {base.py:73} INFO - Using connection ID 'rudderstack_default' for task execution.
[2024-10-03, 13:07:03 UTC] {rudderstack.py:196} INFO - Polled status for syncId: crv9dqt52mjsic2pne00 for retl connection: 2cwRi0iuL5XMnnZCRj9BnPwUxau, status: running
[2024-10-03, 13:07:13 UTC] {base.py:73} INFO - Using connection ID 'rudderstack_default' for task execution.
[2024-10-03, 13:07:13 UTC] {base.py:73} INFO - Using connection ID 'rudderstack_default' for task execution.
[2024-10-03, 13:07:13 UTC] {rudderstack.py:196} INFO - Polled status for syncId: crv9dqt52mjsic2pne00 for retl connection: 2cwRi0iuL5XMnnZCRj9BnPwUxau, status: running

...

Copy link

codecov bot commented Oct 3, 2024

Welcome to Codecov 🎉

Once you merge this PR into your default branch, you're all set! Codecov will compare coverage reports and display results in all future pull requests.

Thanks for integrating Codecov - We've got you covered ☂️

pyproject.toml Outdated Show resolved Hide resolved
@RanjeetMishra RanjeetMishra merged commit aa0b58a into main Oct 9, 2024
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants