You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Hive types have no concept of annotations or metadata, but we can use a special column to preserve the necessary information.
Original SerDe property proposal:
Specify the column that will be used to keep all annotation information for the whole row. This column must be
specified in the table and will not be used to map any values. The annotations are kept as a list of symbols, one for
each column name annotated with it's value annotations in order.
Important when mapping containers to a Hive collection type, e.g. array and map, this will only work for top level
annotations. Currently there is no mechanism to preserve the nested mapped values annotations.
WITH SERDEPROPERTIES (
"annotation.column" = "<colum_name_to_keep_annotations>"
)
The proposal above needs to be extended to support nested values annotations, example: {my_column: ["first", annon::"second"]}
Need to define a UDF to allow users to create queries that use annotations
Zack suggested using a Map instead of String to keep the annotation data. This makes it easier for users to manipulate the annotation column with existing UDFs
The text was updated successfully, but these errors were encountered:
Hive types have no concept of annotations or metadata, but we can use a special column to preserve the necessary information.
Original SerDe property proposal:
Specify the column that will be used to keep all annotation information for the whole row. This column must be
specified in the table and will not be used to map any values. The annotations are kept as a list of symbols, one for
each column name annotated with it's value annotations in order.
Important when mapping containers to a Hive collection type, e.g. array and map, this will only work for top level
annotations. Currently there is no mechanism to preserve the nested mapped values annotations.
Example:
Other considerations
{my_column: ["first", annon::"second"]}
The text was updated successfully, but these errors were encountered: