[FLINK-33971]Specifies whether to use HBase table that supports dynamic columns. #36

MOBIN-F · 2024-01-03T07:10:05Z

Specifies whether to use HBase table that supports dynamic columns.

Refer to the dynamic.table parameter in this document: https://www.alibabacloud.com/help/en/flink/developer-reference/apsaradb-for-hbase-connector#section-ltp-3fy-9qv

Sample code for a result table that supports dynamic columns

CREATE TEMPORARY TABLE datagen_source (
  id INT,
  f1hour STRING,
  f1deal BIGINT,
  f2day STRING,
  f2deal BIGINT
) WITH (
  'connector'='datagen'
);

CREATE TEMPORARY TABLE hbase_sink (
  rowkey INT,
  f1 ROW<`hour` STRING, deal BIGINT>,
  f2 ROW<`day` STRING, deal BIGINT>
) WITH (
  'connector'='hbase-2.2',
  'table-name'='<yourTableName>',
  'zookeeper.quorum'='<yourZookeeperQuorum>',
  'dynamic.table'='true'
);

INSERT INTO hbase_sink
SELECT id, ROW(f1hour, f1deal), ROW(f2day, f2deal) FROM datagen_source;

If dynamic.table is set to true, HBase table that supports dynamic columns is used.
Two fields must be declared in the rows that correspond to each column family. The value of the first field indicates the dynamic column, and the value of the second field indicates the value of the dynamic column.

For example, the datagen_source table contains a row of data The row of data indicates that the ID of the commodity is 1, the transaction amount of the commodity between 10:00 and 11:00 is 100, and the transaction amount of the commodity on July 26, 2020 is 10000. In this case, a row whose rowkey is 1 is inserted into the HBase table. f1:10 is 100, and f2:2020-7-26 is 10000.

boring-cyborg · 2024-01-03T07:10:09Z

Thanks for opening this pull request! Please check out our contributing guidelines. (https://flink.apache.org/contributing/how-to-contribute.html)

MOBIN-F · 2024-01-08T15:22:10Z

@MartijnVisser What do you think of this pr, thks.

MartijnVisser · 2024-01-08T16:46:14Z

@ferenc-csaky Want to take a look?

ferenc-csaky · 2024-01-10T18:54:12Z

@ferenc-csaky Want to take a look?

I'll review it in the next 2 days. Thanks for your contribution @MOBIN-F!

MOBIN-F · 2024-01-12T09:14:49Z

@ferenc-csaky Want to take a look?

I'll review it in the next 2 days. Thanks for your contribution @MOBIN-F!

I added some test cases

ferenc-csaky

I added some comments to the implementation, but my understanding is that this feature is ApsaraDB specific, which means Alibaba Cloud specific.

Correct me if I'm wrong here, but in that case, the main question is that is it a good idea to introduce platform-specific features and flags to the IMO common upstream connector. I am open to discuss this, I don't know if we have other examples to these kind of stories or any rule of thumbs to follow here, @MartijnVisser WDYT?

ferenc-csaky · 2024-01-10T18:42:37Z

...r-hbase-base/src/main/java/org/apache/flink/connector/hbase/table/HBaseConnectorOptions.java

@@ -60,6 +60,12 @@ public class HBaseConnectorOptions {
                            "Representation for null values for string fields. HBase source and "
                                    + "sink encodes/decodes empty bytes as null values for all types except string type.");

+    public static final ConfigOption<Boolean> DYNAMIC_TABLE =
+            ConfigOptions.key("dynamic.table")


I think dynamic-table would comply more to the other similar options like null-string-literal or table-name.

ferenc-csaky · 2024-01-10T18:43:42Z

...ase-base/src/main/java/org/apache/flink/connector/hbase/sink/RowDataToMutationConverter.java

@@ -43,23 +43,26 @@ public class RowDataToMutationConverter implements HBaseMutationConverter<RowDat
    private final TimestampMetadata timestampMetadata;
    private final TimeToLiveMetadata timeToLiveMetadata;
    private transient HBaseSerde serde;
+    private final boolean dynamicTable;


nit: Can we move the new field above serde to keep the grouping of the private final fields?

ferenc-csaky · 2024-01-15T12:32:44Z

...tor-hbase-2.2/src/main/java/org/apache/flink/connector/hbase2/HBase2DynamicTableFactory.java

@@ -121,6 +122,7 @@ public DynamicTableSink createDynamicTableSink(Context context) {
        validatePrimaryKey(context.getPhysicalRowDataType(), context.getPrimaryKeyIndexes());

        String tableName = tableOptions.get(TABLE_NAME);
+        boolean dynamicTable = tableOptions.get(DYNAMIC_TABLE);


This variable is unused, can be removed.

ferenc-csaky · 2024-01-15T12:41:55Z

flink-connector-hbase-base/src/main/java/org/apache/flink/connector/hbase/util/HBaseSerde.java

At this point I think it would worth to make HBaseSerde to an abstract class, e.g. BaseHBaseSerde (or AbstractHBaseSerde, cause base HBase sounds a bit weird) , which contains any common logic and introduce 2 child classes (for example DefaultHBaseSerde and DynamicHBaseSerde) that cointains the different implementations where it makes sense.

Furthermore, I would add an error msg constant and use that message in the thrown UnsupportedOperationExceptions to make it more meaningful and explicit.

I agree with this point of view. If you have no objection to the dynamic-table parameter, I am willing to do this part of the refactoring work.

MartijnVisser · 2024-01-15T19:02:07Z

the main question is that is it a good idea to introduce platform-specific features and flags to the IMO common upstream connector.

If this isn't something that isn't supported in open source HBase itself, then at least we should first have a discussion on this topic in the Dev mailing list.

MOBIN-F · 2024-01-16T02:19:09Z

the main question is that is it a good idea to introduce platform-specific features and flags to the IMO common upstream connector.主要问题是，将特定于平台的功能和标志引入 IMO 通用上游连接器是否是一个好主意。

If this isn't something that isn't supported in open source HBase itself, then at least we should first have a discussion on this topic in the Dev mailing list.如果开源 HBase 本身不支持这一点，那么至少我们应该首先在 Dev 邮件列表中讨论这个话题。

@MartijnVisser @ferenc-csaky
open source hbase is supported. We have used this function internally online. The version is also open source hbase and has nothing to do with the cloud platform.

Tan-JiaLiang · 2024-01-17T03:07:22Z

@MOBIN-F @ferenc-csaky @MartijnVisser Hi, sorry to jump into this discussion.

This option will make FlinkSQL behave strange IMO, because we define the ROW type and the actual behavior is the MAP type, so why not just implement it as a MAP type? And looking at the current implementation as long as user set this option in their table, scan and get will throw an exception, this case never appear in any other connector in my mind.

So I think is better to have a discussion on this topic in the dev mailing list.

MartijnVisser · 2024-01-17T09:59:56Z

So I think is better to have a discussion on this topic in the dev mailing list.

+1

ferenc-csaky · 2024-01-17T13:09:36Z

So I think is better to have a discussion on this topic in the dev mailing list.

Agreed!

Specifies whether to use an HBase table that supports dynamic columns.

da25e12

boring-cyborg bot added the component=Connectors/HBase label Jan 3, 2024

MOBIN-F changed the title ~~Specifies whether to use HBase table that supports dynamic columns.~~ [FLINK-33971]Specifies whether to use HBase table that supports dynamic columns. Jan 3, 2024

MOBIN-F changed the title ~~[FLINK-33971]Specifies whether to use HBase table that supports dynamic columns.~~ Specifies whether to use HBase table that supports dynamic columns. Jan 3, 2024

MOBIN-F changed the title ~~Specifies whether to use HBase table that supports dynamic columns.~~ [FLINK-33971]Specifies whether to use HBase table that supports dynamic columns. Jan 3, 2024

Remove the wrong method

b955d2b

004292 added 3 commits January 12, 2024 17:04

fix

7f2ff2f

add test

1241336

fix

ecb41fa

ferenc-csaky requested changes Jan 15, 2024

View reviewed changes

MOBIN-F closed this Jan 22, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[FLINK-33971]Specifies whether to use HBase table that supports dynamic columns. #36

[FLINK-33971]Specifies whether to use HBase table that supports dynamic columns. #36

MOBIN-F commented Jan 3, 2024 •

edited

Loading

boring-cyborg bot commented Jan 3, 2024

MOBIN-F commented Jan 8, 2024

MartijnVisser commented Jan 8, 2024

ferenc-csaky commented Jan 10, 2024

MOBIN-F commented Jan 12, 2024

ferenc-csaky left a comment

ferenc-csaky Jan 10, 2024

MOBIN-F Jan 16, 2024

ferenc-csaky Jan 10, 2024

MOBIN-F Jan 16, 2024

ferenc-csaky Jan 15, 2024

ferenc-csaky Jan 15, 2024

MOBIN-F Jan 16, 2024

MartijnVisser commented Jan 15, 2024

MOBIN-F commented Jan 16, 2024

Tan-JiaLiang commented Jan 17, 2024

MartijnVisser commented Jan 17, 2024

ferenc-csaky commented Jan 17, 2024

[FLINK-33971]Specifies whether to use HBase table that supports dynamic columns. #36

[FLINK-33971]Specifies whether to use HBase table that supports dynamic columns. #36

Conversation

MOBIN-F commented Jan 3, 2024 • edited Loading

boring-cyborg bot commented Jan 3, 2024

MOBIN-F commented Jan 8, 2024

MartijnVisser commented Jan 8, 2024

ferenc-csaky commented Jan 10, 2024

MOBIN-F commented Jan 12, 2024

ferenc-csaky left a comment

Choose a reason for hiding this comment

ferenc-csaky Jan 10, 2024

Choose a reason for hiding this comment

MOBIN-F Jan 16, 2024

Choose a reason for hiding this comment

ferenc-csaky Jan 10, 2024

Choose a reason for hiding this comment

MOBIN-F Jan 16, 2024

Choose a reason for hiding this comment

ferenc-csaky Jan 15, 2024

Choose a reason for hiding this comment

ferenc-csaky Jan 15, 2024

Choose a reason for hiding this comment

MOBIN-F Jan 16, 2024

Choose a reason for hiding this comment

MartijnVisser commented Jan 15, 2024

MOBIN-F commented Jan 16, 2024

Tan-JiaLiang commented Jan 17, 2024

MartijnVisser commented Jan 17, 2024

ferenc-csaky commented Jan 17, 2024

MOBIN-F commented Jan 3, 2024 •

edited

Loading