Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feature(partner-sdk): added python connector example #29

Merged
merged 16 commits into from
May 13, 2024
Merged

feature(partner-sdk): added python connector example #29

merged 16 commits into from
May 13, 2024

Conversation

manjutapali
Copy link
Collaborator

@manjutapali manjutapali commented Apr 1, 2024 •
edited by ediril
Loading

Closes: T-647875

Description

  • Added new python connector example

@manjutapali manjutapali requested a review from ediril April 1, 2024 09:22
@manjutapali manjutapali force-pushed the T-647875-partner-sdk-python-example branch from 57ca765 to 642f8fd Compare April 1, 2024 11:12
fivetran-niketkhandelwal
fivetran-niketkhandelwal approved these changes Apr 2, 2024
View reviewed changes
Copy link
Contributor

@fivetran-niketkhandelwal fivetran-niketkhandelwal left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added some suggestions, please see if keeping them makes sense. example code is working fine. approved

examples/connector/python/connector/build.sh Outdated Show resolved Hide resolved
examples/connector/python/connector/build.sh Outdated Show resolved Hide resolved
examples/connector/python/connector/main.py Outdated Show resolved Hide resolved
examples/connector/python/connector/main.py Outdated Show resolved Hide resolved
examples/connector/python/connector/main.py Outdated Show resolved Hide resolved
examples/connector/python/connector/main.py Outdated Show resolved Hide resolved
examples/connector/python/connector/main.py Outdated Show resolved Hide resolved
examples/connector/python/connector/main.py Outdated Show resolved Hide resolved
examples/connector/python/connector/main.py Outdated Show resolved Hide resolved
@ediril ediril changed the title feature(partner-sdk): added python example feature(partner-sdk): added python connector example Apr 7, 2024
ediril
ediril requested changes Apr 7, 2024
View reviewed changes
Copy link
Collaborator

@ediril ediril left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's get rid of the last connector folder. All the files should go into examples/connector/python

examples/connector/python/connector/README.md Outdated Show resolved Hide resolved
@manjutapali manjutapali requested a review from ediril April 8, 2024 06:15
ediril
ediril requested changes Apr 14, 2024
View reviewed changes
Copy link
Collaborator

@ediril ediril left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When I run this example with the connector tester, I get the following error:

java.lang.RuntimeException: java.sql.SQLException: Binder Error: Cannot change the type of a column that has a UNIQUE or PRIMARY KEY constraint specified
	at testers.util.MockWarehouse.lambda$changeColumnType$12(MockWarehouse.java:281)
	at testers.util.MockWarehouse.runOnConnection(MockWarehouse.java:53)
	at testers.util.MockWarehouse.changeColumnType(MockWarehouse.java:268)
	at testers.util.MockConnectorOutput.handleColumnChanges(MockConnectorOutput.java:307)
	at testers.util.MockConnectorOutput.handleUpsert(MockConnectorOutput.java:123)
	at testers.util.MockConnectorOutput.enqueueOperation(MockConnectorOutput.java:76)
	at client.connector.SdkConnectorClient.update(SdkConnectorClient.java:85)
	at testers.SdkConnectorTester.run(SdkConnectorTester.java:240)
	at testers.SdkConnectorTester.main(SdkConnectorTester.java:87)

This is a limitation of the connector tester that currently it cannot change the data type of primary key.

Can you please update the example so it does work with the connector tester?

examples/connector/python/build.sh Show resolved Hide resolved
examples/connector/python/run.sh Outdated Show resolved Hide resolved
examples/connector/python/requirements.txt
@@ -0,0 +1,2 @@
grpcio==1.60.1
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should match the versions we are showing in the development guide. Can we not use 1.61.1?

Copy link
Collaborator Author

@manjutapali manjutapali Apr 15, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think there is no 1.61.1
Screenshot 2024-04-15 at 10 37 25 AM

@manjutapali
Copy link
Collaborator Author

When I run this example with the connector tester, I get the following error:

java.lang.RuntimeException: java.sql.SQLException: Binder Error: Cannot change the type of a column that has a UNIQUE or PRIMARY KEY constraint specified
	at testers.util.MockWarehouse.lambda$changeColumnType$12(MockWarehouse.java:281)
	at testers.util.MockWarehouse.runOnConnection(MockWarehouse.java:53)
	at testers.util.MockWarehouse.changeColumnType(MockWarehouse.java:268)
	at testers.util.MockConnectorOutput.handleColumnChanges(MockConnectorOutput.java:307)
	at testers.util.MockConnectorOutput.handleUpsert(MockConnectorOutput.java:123)
	at testers.util.MockConnectorOutput.enqueueOperation(MockConnectorOutput.java:76)
	at client.connector.SdkConnectorClient.update(SdkConnectorClient.java:85)
	at testers.SdkConnectorTester.run(SdkConnectorTester.java:240)
	at testers.SdkConnectorTester.main(SdkConnectorTester.java:87)

This is a limitation of the connector tester that currently it cannot change the data type of primary key.

Can you please update the example so it does work with the connector tester?

It's working with the connector tester which I am using it for test.

@manjutapali manjutapali force-pushed the T-647875-partner-sdk-python-example branch from 0b5f5e5 to 5094420 Compare April 16, 2024 05:23
@manjutapali manjutapali requested a review from ediril April 16, 2024 05:24
manjutapali and others added 14 commits April 16, 2024 12:43
@manjutapali manjutapali force-pushed the T-647875-partner-sdk-python-example branch from 5094420 to 2c01991 Compare April 16, 2024 07:13
@manjutapali manjutapali force-pushed the T-647875-partner-sdk-python-example branch from 23d7abc to a67478f Compare April 16, 2024 07:18
ediril
ediril requested changes Apr 16, 2024
View reviewed changes
examples/connector/python/build.sh Show resolved Hide resolved
examples/connector/python/main.py
t1.columns.add(name="a1", type=common_pb2.DataType.UNSPECIFIED, primary_key=True)
t1.columns.add(name="a2", type=common_pb2.DataType.DOUBLE)

t2 = table_list.tables.add(name="table2")
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This example doesn't use table2. Can you add an upsert for it so it gets a row in the destination?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copied from the java connector example, even it does not use it table2. I'll add simple upsert operation for it.

@manjutapali manjutapali requested a review from ediril April 17, 2024 09:39
@manjutapali manjutapali merged commit 7387dbc into main May 13, 2024
1 check passed
@manjutapali manjutapali deleted the T-647875-partner-sdk-python-example branch May 13, 2024 07:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@fivetran-niketkhandelwal fivetran-niketkhandelwal fivetran-niketkhandelwal approved these changes

@ediril ediril ediril approved these changes

Assignees

@fivetran-niketkhandelwal fivetran-niketkhandelwal

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@manjutapali @ediril @fivetran-niketkhandelwal