…pache#9634)

Unify the encoding of Java `String` to byte array in Hudi,
especially for writing bytes to the storage,
by using `UTF_8` encoding only.

---------

Co-authored-by: Sagar Sumit <[email protected]>

…emove dependency (apache#9649)

…e#9700)

Co-authored-by: Jonathan Vexler <=>
Co-authored-by: Y Ethan Guo <[email protected]>

…nt-common to hudi-common (apache#9103)

Co-authored-by: Y Ethan Guo <[email protected]>

This commit fixes the bundling of RoaringBitmap dependency in Hudi bundles by including it in the shade rules and shading the classes, to avoid dependency conflict with engine-provided jars, e.g., Spark. Before this fix, with Hudi Spark bundle, NoSuchMethodError exception is thrown by Spark 3.2.

* Address build failures in older Flink Versions
* Remove unnecessary dependency on flink-connector-hive
* Fix Flink 1.18 Validate-bundles

---------

Signed-off-by: Prabhu Joseph <[email protected]>
Co-authored-by: Prabhu Joseph <[email protected]>
Co-authored-by: root <[email protected]>

…#10066)

…pache#10083)

---------

Co-authored-by: Shawn Chang <[email protected]>
Co-authored-by: Y Ethan Guo <[email protected]>

…ache#10123)

… prevent inconsistency during data reads (apache#10711)

…rmats (apache#10265)

* [HUDI-7190] Fix legacy parquet format nested columns vectorized read for spark3.3+
* Fix nested type implicit schema evolution
* fix legacy format support batch read
* Add exception messages when vectorized read nested type with type change

… npe (apache#10300)

…ool (apache#10374)

Co-authored-by: jp0317 <[email protected]>

…ansitionPendingState (apache#10392)

…pache#10411)

Co-authored-by: xuyu <[email protected]>

…pache#10388)

… with append mode (apache#10430)

Co-authored-by: xuyu <[email protected]>

…to specify hoodie.properties path (apache#10413)

Co-authored-by: dongsj <[email protected]>

…pache#10438)

…leak (apache#10441)

Co-authored-by: leixin1 <[email protected]>

Co-authored-by: Jonathan Vexler <=>

…cutor type (apache#10459)

* make sampling rate configurable for BOUNDED_IN_MEMORY executor type
* add sinceVersion for new configs
---------
Co-authored-by: wei.kong <[email protected]>

… mode (apache#10261)

* Add cachedSchema per batch, fix idempotency with getSourceSchema calls
---------
Co-authored-by: danielfordfc <[email protected]>

…#10467)

Co-authored-by: Jonathan Vexler <=>

…parkSessionExtension (apache#10373)

* [HUDI-7241] Avoid always broadcast HUDI relation if not using HoodieSparkSessionExtension
* Update the logical to check whether HoodieExtension is enabled

…ionedTable created by 0.10/0.11 HUDI version (apache#10482)

…#10477)

…elation. (apache#10480)

This speedups the check for large datasets when a very large number of files need to be checked for existence.

…ow table with cluster configuration when the index is bucket. (apache#10475)

The reason is to clean up any orphan resources.

Changes:
1. Set initial memory 128M.

* make bloom filter skippable for CPU saving
---------
Co-authored-by: wei.kong <[email protected]>

…apache#10490)

…0476)

After we set this parameter, the surefire will try to ping the forked JVM
after the timeout.

)

* [MINOR] Clean default Hadoop configurations for SparkContext

These default Hadoop configurations are not used in Hudi tests.

* Consolidating the code into a helper class

---------

Co-authored-by: vinoth chandar <[email protected]>

Signed-off-by: wulingqi <[email protected]>

…ests (apache#10492)

* reduce combos of tests

* build success

---------

Co-authored-by: Jonathan Vexler <=>

… Involve Both Columns with and without Column Stats (apache#10389)

This commit adds a Hudi-native HFile reader implementation independent of HBase.

fixed a test.

…apache#10518)

…hema mismatches that in parquet file (apache#10497)

…er pushing down for any of its operand's logical type for is unsupported in ExpressionPredicates::toParquetPredicate (apache#10524)

Co-authored-by: Jonathan Vexler <=>

Co-authored-by: xuyu <[email protected]>
Co-authored-by: Y Ethan Guo <[email protected]>

… with no-partitioned table (apache#10532)

Signed-off-by: wulingqi <[email protected]>

…o schema does not exist in hudi table (apache#10527)

…wn its wrapped filter unsupported, as its operand's primitive value is incomparable (apache#10537)

…apache#10541)

Co-authored-by: xuyu <[email protected]>

…ng date comparison operator (apache#10517)

* [MINOR] Reduce UT spark-datasource test times

* Reverting the parallelism change

…leanups (apache#10342)

…taClient in order to reduce occurences of executors making file listing calls when reloading active timeline (apache#10540)

Summary:
Currently some implementors of  AbstractHoodieLogRecordReader create a HoodieTableMetaClient on construction, which implicitly reloads active timeline, causing a `listStatus` HDFS call. Since these are created in executors, each of the hundreds to thousands of executors will make a `listStatus` call at the same time during a stage. To avoid these redundant calls to HDFS NameNode, AbstractHoodieLogRecordReader and the following implementations have been updated to allow an existing HoodieTableMetaClient to be passed in.
- HoodieUnMergedLogRecordScanner
- HoodieMergedLogRecordScanner
- HoodieMetadataMergedLogRecordReader
As long as the caller passed in a HoodieTableMetaClient with active timeline already loaded, and the implementation doesn't need to re-load the timeline (such as in order to get a more "fresh" timeline) than `listStatus` calls can be avoided in the executor, without causing the logic to be incorrect.

Co-authored-by: Krishen Bhan <[email protected]>

…own (apache#10531)

…cord reader (apache#10340)

… failing stream (apache#10500)

Cases:
- No transformers, with schema provider. Records will go to the error table if they cannot be rewritten in the deduced schema.
- recordkey is null, even if the column is nullable in the schema

…0549)

…che#10556)

---------

Co-authored-by: Jonathan Vexler <=>

…apache#10562)

Co-authored-by: dongsj <[email protected]>

…trics if lockDurationTimer has not been started (apache#10523)

…ion (apache#10564)

This commit creates a new module `hudi-hadoop-common` for hadoop-specific implementation.  This serves as the first step to decouple `hudi-common` module from hadoop dependencies.

…ieval (apache#10572)

This commit introduces `HoodieStorage` abstraction and Hudi's counterpart classes for Hadoop File System classes (`org.apache.hadoop.fs.`[`FileSystem`, `Path`, `PathFilter`, `FileStatus`]) to decouple Hudi's implementation from Hadoop classes, so it's much easier to plugin different file system implementation.

…PartitionMetadata (apache#10568)

…he#10330)

…e#10574)

…che#10570)

…e#10571)

…put>Stream when possible (apache#10573)

…he#10575)

…ve Sync (apache#10577)

Co-authored-by: Danny Chan <[email protected]>

…tPendingClusterInstant (apache#10613)

Co-authored-by: Jonathan Vexler <=>