Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[connector] CDC doesn't support primary key only tables #142

Open
aymkhalil opened this issue Apr 11, 2023 · 0 comments
Open

[connector] CDC doesn't support primary key only tables #142

aymkhalil opened this issue Apr 11, 2023 · 0 comments

Comments

@aymkhalil
Copy link
Contributor

A table with primary key only columns with cdc enabled, will results in errors in the connector. My initial guess is primary key only tables are not supported because we rely on the value part of the key value schema (that is generated from non-primary key columns) to indicate a delete by passing null values. In the case of primary key only, it is not clear how deletes will be differentiated from inserts because the value is always null.

To reproduce:

  1. Create a table with PK only
CREATE TABLE source.results (
    uuid           text,
    value          int,
	PRIMARY KEY ((uuid, value))
)
WITH cdc = TRUE;
  1. Enable CDC connectors
  2. The following error will be logged and no events will be sent to the data topic
2023-04-11T15:35:02,097+0000 [public/cdc-test/origin-results-0] ERROR org.apache.pulsar.functions.instance.JavaInstanceRunnable - [public/cdc-test/origin-results:0] Uncaught exception in Java Instance
java.util.concurrent.CompletionException: com.datastax.oss.driver.api.core.servererrors.SyntaxError: line 1:7 no viable alternative at input 'FROM' (SELECT [FROM]...)
	at java.util.concurrent.CompletableFuture.reportJoin(CompletableFuture.java:412) ~[?:?]
	at java.util.concurrent.CompletableFuture.join(CompletableFuture.java:2044) ~[?:?]
	at com.datastax.oss.pulsar.source.CassandraSource.batchRead(CassandraSource.java:574) ~[?:?]
	at com.datastax.oss.pulsar.source.CassandraSource.maybeBatchRead(CassandraSource.java:463) ~[?:?]
	at com.datastax.oss.pulsar.source.CassandraSource.read(CassandraSource.java:455) ~[?:?]

The desired behavior

  1. Have the events go through (consider using a metadata in the schema to indicate DELETE vs INSERT)
  2. Skip the records with a clear user feedback
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant