[FEATURE] Non hive partition discovery and update.

[penghuo]

Is your feature request related to a problem?

1. After create table, user need to run MSCK statement to refresh table partition. if new partition added, user need to refresh partition of table.
2. spark data source table does not support non hive partition.

What solution would you like?

1. user don't need to run MSCK or ALTER to refresh partition.
2. support non hive partition.

What alternatives have you considered?
not yet.

Do you have any additional context?
n/a

[penghuo]

Glue crawler support discovery non-hive partition schema. Glue crawler detect /2022/01/01 as partition_0, partition_1 and partition_2, user can rename partition_0, partition_1 and partition_2 as year, month and day. More reading in https://aws.amazon.com/blogs/big-data/catalog-and-analyze-application-load-balancer-logs-more-efficiently-with-aws-glue-custom-classifiers-and-amazon-athena/.

[penghuo]

Solutions

1. add recursiveFileLookup = true when creating table. recursiveFileLookup is used to recursively load files and it disables partition inferring.
2. create skipping index of table. note: User can not add Partition columns to skipping index if table does not include partition columns.