Skip to content

Commit

Permalink
Add (non-public) method to expose iceberg schema for each column of a…
Browse files Browse the repository at this point in the history
… channel (#856)

For schema evolution of structured data types, Kafka Connector needs a way to discover the schema of the column that had a validation error.
We already have an API on StreamingIngestChannel called getTableSchema() that returns a Map<String, ColumnProperties>.
Am adding a method (currently non-public) to the ColumnProperties type called getIcebergSchema() that returns a spec-compliant JSON of the column's iceberg schema.

We'll make this method public when we make setIsIceberg() public.
  • Loading branch information
sfc-gh-hmadan authored Oct 4, 2024
1 parent f8a684c commit e5ec33a
Showing 1 changed file with 22 additions and 7 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -6,19 +6,21 @@
* that this is slightly different than the internal column metadata used elsewhere in this SDK.
*/
public class ColumnProperties {
private String type;
private final String type;

private String logicalType;
private final String logicalType;

private Integer precision;
private final Integer precision;

private Integer scale;
private final Integer scale;

private Integer byteLength;
private final Integer byteLength;

private Integer length;
private final Integer length;

private boolean nullable;
private final boolean nullable;

private final String icebergColumnSchema;

ColumnProperties(ColumnMetadata columnMetadata) {
this.type = columnMetadata.getType();
Expand All @@ -28,6 +30,7 @@ public class ColumnProperties {
this.byteLength = columnMetadata.getByteLength();
this.length = columnMetadata.getLength();
this.nullable = columnMetadata.getNullable();
this.icebergColumnSchema = columnMetadata.getSourceIcebergDataType();
}

public String getType() {
Expand Down Expand Up @@ -57,4 +60,16 @@ public Integer getLength() {
public boolean isNullable() {
return nullable;
}

/**
* Return the value of sourceIcebergDataType() as returned by the service. It is populated only
* when this object represents an iceberg table's column, null otherwise. The String returned from
* here is meant to conform to the json schema specified here:
* https://iceberg.apache.org/spec/#appendix-c-json-serialization
*
* <p>Make this a public API when the Builder.setIsIceberg API is made public.
*/
String getIcebergSchema() {
return icebergColumnSchema;
}
}

0 comments on commit e5ec33a

Please sign in to comment.