Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FLINK-35982] Fix transform metadata config doesn't work if no projection block was provided #3592

Merged
merged 2 commits into from
Nov 7, 2024

Conversation

yuxiqian
Copy link
Contributor

@yuxiqian yuxiqian commented Sep 2, 2024

This closes FLINK-35982.

One may readjust source table primary keys, partition keys, table options by specifying them in a Transform block like this:

transform:
  - source-table: db.tbl
    projection: '*'
    primary-keys: order_id, product_name
    partition-keys: order_id
    table-options: bucket=1

However, if the projection field is omitted (which fallbacks to default behavior that does not change any source table columns), such configuration will not take effect:

transform:
  - source-table: db.tbl
    primary-keys: order_id, product_name
    partition-keys: order_id
    table-options: bucket=1 # These options will not apply

The underlying reason is transform blocks without projection was simply ignored by PreTransformOperator, which isn't accurate since other options like primary keys, partition keys should work without projecting original columns.

# Conflicts:
#	flink-cdc-composer/src/test/java/org/apache/flink/cdc/composer/flink/FlinkPipelineTransformITCase.java

# Conflicts:
#	flink-cdc-composer/src/test/java/org/apache/flink/cdc/composer/flink/FlinkPipelineTransformITCase.java

# Conflicts:
#	flink-cdc-composer/src/test/java/org/apache/flink/cdc/composer/flink/FlinkPipelineTransformITCase.java
#	flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/transform/TransformExpressionCompiler.java
@yuxiqian
Copy link
Contributor Author

yuxiqian commented Sep 2, 2024

cc @ruanhang1993 @lvyanquan

Copy link
Contributor

@leonardBang leonardBang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@leonardBang leonardBang merged commit d620c58 into apache:master Nov 7, 2024
2 checks passed
Gianzie pushed a commit to Gianzie/flink-cdc that referenced this pull request Nov 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants