Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[HUDI-1234] DO NOT MERGE use fg reader in cdc test #11401

Closed
wants to merge 100 commits into from
Closed

[HUDI-1234] DO NOT MERGE use fg reader in cdc test #11401

wants to merge 100 commits into from

Commits on Apr 2, 2024

  1. add spark 3.3 reader

    Jonathan Vexler committed Apr 2, 2024
    Configuration menu
    Copy the full SHA
    283d7c3 View commit details
    Browse the repository at this point in the history

  2. add spark3.4

    Jonathan Vexler committed Apr 2, 2024
    Configuration menu
    Copy the full SHA
    ef65428 View commit details
    Browse the repository at this point in the history

  3. add spark 3.5

    Jonathan Vexler committed Apr 2, 2024
    Configuration menu
    Copy the full SHA
    8168147 View commit details
    Browse the repository at this point in the history

  4. add spark 3.2

    Jonathan Vexler committed Apr 2, 2024
    Configuration menu
    Copy the full SHA
    1a53f1e View commit details
    Browse the repository at this point in the history

  5. add spark 3.1

    Jonathan Vexler committed Apr 2, 2024
    Configuration menu
    Copy the full SHA
    97d9920 View commit details
    Browse the repository at this point in the history

  6. add spark 3.0

    Jonathan Vexler committed Apr 2, 2024
    Configuration menu
    Copy the full SHA
    b9d7ce4 View commit details
    Browse the repository at this point in the history

  7. add spark 2.4

    Jonathan Vexler committed Apr 2, 2024
    Configuration menu
    Copy the full SHA
    a20e9d4 View commit details
    Browse the repository at this point in the history

Commits on Apr 3, 2024

  1. spark 3.3 use properties class

    Jonathan Vexler committed Apr 3, 2024
    Configuration menu
    Copy the full SHA
    abe7839 View commit details
    Browse the repository at this point in the history

  2. spark 3.2 add props class

    Jonathan Vexler committed Apr 3, 2024
    Configuration menu
    Copy the full SHA
    865526e View commit details
    Browse the repository at this point in the history

Commits on Apr 4, 2024

  1. spark 3.4 add properties

    Jonathan Vexler committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    bab974a View commit details
    Browse the repository at this point in the history

  2. add spark 3.5 properties

    Jonathan Vexler committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    0eb2185 View commit details
    Browse the repository at this point in the history

  3. add properties spark 3.1

    Jonathan Vexler committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    10a577f View commit details
    Browse the repository at this point in the history

  4. add props spark 3.0

    Jonathan Vexler committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    3c7ecf1 View commit details
    Browse the repository at this point in the history

  5. add properties spark 2.4

    Jonathan Vexler committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    700013b View commit details
    Browse the repository at this point in the history

  6. fix 3.0

    Jonathan Vexler committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    37f52eb View commit details
    Browse the repository at this point in the history

  7. refactor to get rid of properties, spark 3.1

    Jonathan Vexler committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    b9c1592 View commit details
    Browse the repository at this point in the history

  8. remove props spark 3.0

    Jonathan Vexler committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    e3957c5 View commit details
    Browse the repository at this point in the history

  9. use class model for spark 3.3

    Jonathan Vexler committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    7345f6b View commit details
    Browse the repository at this point in the history

  10. remove props spark 3.3

    Jonathan Vexler committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    2942a6c View commit details
    Browse the repository at this point in the history

  11. remove props spark 3.4

    Jonathan Vexler committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    2012131 View commit details
    Browse the repository at this point in the history

  12. remove props spark 3.5

    Jonathan Vexler committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    e40072e View commit details
    Browse the repository at this point in the history

  13. remove props spark 2.4

    Jonathan Vexler committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    5813cbf View commit details
    Browse the repository at this point in the history

  14. remove change

    Jonathan Vexler committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    0f00822 View commit details
    Browse the repository at this point in the history

  15. remove bad import

    Jonathan Vexler committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    867593d View commit details
    Browse the repository at this point in the history

  16. add spark 3.3

    Jonathan Vexler committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    64965e6 View commit details
    Browse the repository at this point in the history

  17. add spark 3.4

    Jonathan Vexler committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    6b0ca88 View commit details
    Browse the repository at this point in the history

  18. add spark 3.5

    Jonathan Vexler committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    5a557e1 View commit details
    Browse the repository at this point in the history

  19. add spark 3.2

    Jonathan Vexler committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    2f48bdd View commit details
    Browse the repository at this point in the history

  20. add spark 3.1

    Jonathan Vexler committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    9845f37 View commit details
    Browse the repository at this point in the history

  21. add spark 3.0

    Jonathan Vexler committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    5e536fe View commit details
    Browse the repository at this point in the history

  22. add spark 2.4

    Jonathan Vexler committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    9dc9ade View commit details
    Browse the repository at this point in the history

  23. fix 2.4

    Jonathan Vexler committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    b12c018 View commit details
    Browse the repository at this point in the history

  24. integrate schema evolution into the fg reader

    Jonathan Vexler committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    749c651 View commit details
    Browse the repository at this point in the history

  25. finish rebase

    Jonathan Vexler committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    d59fd1b View commit details
    Browse the repository at this point in the history

Commits on Apr 5, 2024

  1. use augment hadoop conf

    Jonathan Vexler committed Apr 5, 2024
    Configuration menu
    Copy the full SHA
    b7a7c27 View commit details
    Browse the repository at this point in the history

  2. clone hadoop conf for log file read

    Jonathan Vexler committed Apr 5, 2024
    Configuration menu
    Copy the full SHA
    8319069 View commit details
    Browse the repository at this point in the history

  3. try copying conf in reader

    Jonathan Vexler committed Apr 5, 2024
    Configuration menu
    Copy the full SHA
    28200fd View commit details
    Browse the repository at this point in the history

  4. create a copy of the conf when reading

    Jonathan Vexler committed Apr 5, 2024
    Configuration menu
    Copy the full SHA
    8ca12f2 View commit details
    Browse the repository at this point in the history

  5. make conf copy during read

    Jonathan Vexler committed Apr 5, 2024
    Configuration menu
    Copy the full SHA
    088f69e View commit details
    Browse the repository at this point in the history

Commits on Apr 9, 2024

  1. recordkey filters is reverse

    Jonathan Vexler committed Apr 9, 2024
    Configuration menu
    Copy the full SHA
    c5d77d5 View commit details
    Browse the repository at this point in the history

Commits on Apr 10, 2024

  1. fix position based merging

    Jonathan Vexler committed Apr 10, 2024
    Configuration menu
    Copy the full SHA
    3b8b4f1 View commit details
    Browse the repository at this point in the history

  2. fix mor

    Jonathan Vexler committed Apr 10, 2024
    Configuration menu
    Copy the full SHA
    70cef40 View commit details
    Browse the repository at this point in the history

  3. fix some tests

    Jonathan Vexler committed Apr 10, 2024
    Configuration menu
    Copy the full SHA
    56125af View commit details
    Browse the repository at this point in the history

  4. add validations

    Jonathan Vexler committed Apr 10, 2024
    Configuration menu
    Copy the full SHA
    31eb84b View commit details
    Browse the repository at this point in the history

  5. fixed broken test

    Jonathan Vexler committed Apr 10, 2024
    Configuration menu
    Copy the full SHA
    9c723d0 View commit details
    Browse the repository at this point in the history

  6. fix set and use wrong var

    Jonathan Vexler committed Apr 10, 2024
    Configuration menu
    Copy the full SHA
    f475aa9 View commit details
    Browse the repository at this point in the history

  7. only update and use reader state in the constructor

    Jonathan Vexler committed Apr 10, 2024
    Configuration menu
    Copy the full SHA
    1e4657a View commit details
    Browse the repository at this point in the history

  8. use data block schema to read untransformed record

    Jonathan Vexler committed Apr 10, 2024
    Configuration menu
    Copy the full SHA
    15acc2e View commit details
    Browse the repository at this point in the history

Commits on Apr 11, 2024

  1. add test

    Jonathan Vexler committed Apr 11, 2024
    Configuration menu
    Copy the full SHA
    8205971 View commit details
    Browse the repository at this point in the history

  2. allow vectorized read and comment better

    Jonathan Vexler committed Apr 11, 2024
    Configuration menu
    Copy the full SHA
    815b6fd View commit details
    Browse the repository at this point in the history

  3. address review comments 3.5

    Jonathan Vexler committed Apr 11, 2024
    Configuration menu
    Copy the full SHA
    120226a View commit details
    Browse the repository at this point in the history

  4. rename spark 3.4

    Jonathan Vexler committed Apr 11, 2024
    Configuration menu
    Copy the full SHA
    dbdefad View commit details
    Browse the repository at this point in the history

  5. rename for spark3.3

    Jonathan Vexler committed Apr 11, 2024
    Configuration menu
    Copy the full SHA
    f950835 View commit details
    Browse the repository at this point in the history

  6. rename for spark 3.2

    Jonathan Vexler committed Apr 11, 2024
    Configuration menu
    Copy the full SHA
    75da5dd View commit details
    Browse the repository at this point in the history

  7. rename spark 3.1

    Jonathan Vexler committed Apr 11, 2024
    Configuration menu
    Copy the full SHA
    e7e4b51 View commit details
    Browse the repository at this point in the history

  8. rename spark 30

    Jonathan Vexler committed Apr 11, 2024
    Configuration menu
    Copy the full SHA
    81da1a7 View commit details
    Browse the repository at this point in the history

  9. rename for spark 2

    Jonathan Vexler committed Apr 11, 2024
    Configuration menu
    Copy the full SHA
    1c68439 View commit details
    Browse the repository at this point in the history

  10. remove empty line

    Jonathan Vexler committed Apr 11, 2024
    Configuration menu
    Copy the full SHA
    f6c5beb View commit details
    Browse the repository at this point in the history

Commits on Apr 12, 2024

  1. address hidden review comments

    Jonathan Vexler committed Apr 12, 2024
    Configuration menu
    Copy the full SHA
    8f1ba6d View commit details
    Browse the repository at this point in the history

  2. Merge branch 'create_spark_file_readers' into add_schema_evolution_to… 

    …_spark_file_readers
    Jonathan Vexler committed Apr 12, 2024
    Configuration menu
    Copy the full SHA
    37bc97b View commit details
    Browse the repository at this point in the history

  3. Merge branch 'add_schema_evolution_to_spark_file_readers' into add_sc… 

    …hema_evolution_to_fg_reader
    Jonathan Vexler committed Apr 12, 2024
    Configuration menu
    Copy the full SHA
    26c1df8 View commit details
    Browse the repository at this point in the history

  4. finish merge

    Jonathan Vexler committed Apr 12, 2024
    Configuration menu
    Copy the full SHA
    ee7a0e3 View commit details
    Browse the repository at this point in the history

  5. Merge branch 'master' into add_schema_evolution_to_spark_file_readers

    Jonathan Vexler committed Apr 12, 2024
    Configuration menu
    Copy the full SHA
    a73f955 View commit details
    Browse the repository at this point in the history

  6. Merge branch 'add_schema_evolution_to_spark_file_readers' into add_sc… 

    …hema_evolution_to_fg_reader
    Jonathan Vexler committed Apr 12, 2024
    Configuration menu
    Copy the full SHA
    966e8c8 View commit details
    Browse the repository at this point in the history

  7. add missing import

    Jonathan Vexler committed Apr 12, 2024
    Configuration menu
    Copy the full SHA
    c8f507b View commit details
    Browse the repository at this point in the history

Commits on Apr 15, 2024

  1. address comments and add changes to legacy 3.5

    Jonathan Vexler committed Apr 15, 2024
    Configuration menu
    Copy the full SHA
    be77950 View commit details
    Browse the repository at this point in the history

  2. spark 3.4 update legacy

    Jonathan Vexler committed Apr 15, 2024
    Configuration menu
    Copy the full SHA
    8943bb4 View commit details
    Browse the repository at this point in the history

  3. make changes to spark 3.3 and restore legacy for 3.4 and 3.5

    Jonathan Vexler committed Apr 15, 2024
    Configuration menu
    Copy the full SHA
    4c32421 View commit details
    Browse the repository at this point in the history

  4. update spark 3.2

    Jonathan Vexler committed Apr 15, 2024
    Configuration menu
    Copy the full SHA
    a08eacb View commit details
    Browse the repository at this point in the history

  5. update spark 3.1

    Jonathan Vexler committed Apr 15, 2024
    Configuration menu
    Copy the full SHA
    1edf6bf View commit details
    Browse the repository at this point in the history

  6. update spark 3.0

    Jonathan Vexler committed Apr 15, 2024
    Configuration menu
    Copy the full SHA
    eb58a1a View commit details
    Browse the repository at this point in the history

  7. Merge branch 'add_schema_evolution_to_spark_file_readers' into add_sc… 

    …hema_evolution_to_fg_reader
    Jonathan Vexler committed Apr 15, 2024
    Configuration menu
    Copy the full SHA
    aed8113 View commit details
    Browse the repository at this point in the history

  8. Merge branch 'master' into add_schema_evolution_to_fg_reader

    Jonathan Vexler committed Apr 15, 2024
    Configuration menu
    Copy the full SHA
    94171e2 View commit details
    Browse the repository at this point in the history

Commits on Apr 16, 2024

  1. remove some unnecessary changes to make it easier to review

    Jonathan Vexler committed Apr 16, 2024
    Configuration menu
    Copy the full SHA
    72e09f6 View commit details
    Browse the repository at this point in the history

Commits on Apr 18, 2024

  1. fix most review comments

    Jonathan Vexler committed Apr 18, 2024
    Configuration menu
    Copy the full SHA
    24be896 View commit details
    Browse the repository at this point in the history

  2. address review comments

    Jonathan Vexler committed Apr 18, 2024
    Configuration menu
    Copy the full SHA
    d1fbbf6 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'master' into add_schema_evolution_to_fg_reader

    Jonathan Vexler committed Apr 18, 2024
    Configuration menu
    Copy the full SHA
    96a371f View commit details
    Browse the repository at this point in the history

  4. use more complex projection

    Jonathan Vexler committed Apr 18, 2024
    Configuration menu
    Copy the full SHA
    89078f3 View commit details
    Browse the repository at this point in the history

Commits on Apr 24, 2024

  1. Merge branch 'master' into add_schema_evolution_to_fg_reader

    @jonvex
    jonvex authored Apr 24, 2024
    Configuration menu
    Copy the full SHA
    6df21d8 View commit details
    Browse the repository at this point in the history

Commits on Apr 29, 2024

  1. Merge branch 'master' into add_schema_evolution_to_fg_reader

    @jonvex
    jonvex authored Apr 29, 2024
    Configuration menu
    Copy the full SHA
    95b6d62 View commit details
    Browse the repository at this point in the history

  2. add batch support

    Jonathan Vexler committed Apr 29, 2024
    Configuration menu
    Copy the full SHA
    879e07c View commit details
    Browse the repository at this point in the history

Commits on Apr 30, 2024

  1. Merge branch 'master' into add_schema_evolution_to_fg_reader

    Jonathan Vexler committed Apr 30, 2024
    Configuration menu
    Copy the full SHA
    c98242b View commit details
    Browse the repository at this point in the history

  2. add comment and allow vector read when positional merge is used

    Jonathan Vexler committed Apr 30, 2024
    Configuration menu
    Copy the full SHA
    9d1ac2a View commit details
    Browse the repository at this point in the history

Commits on May 7, 2024

  1. Merge branch 'master' into add_schema_evolution_to_fg_reader

    Jonathan Vexler committed May 7, 2024
    Configuration menu
    Copy the full SHA
    9d0a471 View commit details
    Browse the repository at this point in the history

Commits on May 14, 2024

  1. Merge branch 'master' into add_schema_evolution_to_fg_reader

    Jonathan Vexler committed May 14, 2024
    Configuration menu
    Copy the full SHA
    540d122 View commit details
    Browse the repository at this point in the history

Commits on May 30, 2024

  1. Merge branch 'master' into add_schema_evolution_to_fg_reader

    Jonathan Vexler committed May 30, 2024
    Configuration menu
    Copy the full SHA
    f126475 View commit details
    Browse the repository at this point in the history

  2. don't unwrap copy, we need to original so that the hadoopconf gets mo… 

    …dified by build
    Jonathan Vexler committed May 30, 2024
    Configuration menu
    Copy the full SHA
    63737ca View commit details
    Browse the repository at this point in the history

  3. fix another case where hadoopconf copy isolates config changes

    Jonathan Vexler committed May 30, 2024
    Configuration menu
    Copy the full SHA
    475a1bc View commit details
    Browse the repository at this point in the history

Commits on May 31, 2024

  1. disable problem test

    Jonathan Vexler committed May 31, 2024
    Configuration menu
    Copy the full SHA
    d504e37 View commit details
    Browse the repository at this point in the history

  2. disable vectorized read for mor and bootstrap

    Jonathan Vexler committed May 31, 2024
    Configuration menu
    Copy the full SHA
    ec6fa62 View commit details
    Browse the repository at this point in the history

Commits on Jun 3, 2024

  1. revert test change

    Jonathan Vexler committed Jun 3, 2024
    Configuration menu
    Copy the full SHA
    2284613 View commit details
    Browse the repository at this point in the history

Commits on Jun 4, 2024

  1. Address self comments

    @codope
    codope committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    72eb6de View commit details
    Browse the repository at this point in the history

  2. allow batch for cdc

    Jonathan Vexler committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    5f257f0 View commit details
    Browse the repository at this point in the history

  3. add comments

    Jonathan Vexler committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    29c604c View commit details
    Browse the repository at this point in the history

  4. add testing back/ add new testing

    Jonathan Vexler committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    4045388 View commit details
    Browse the repository at this point in the history

  5. add spark test

    Jonathan Vexler committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    575b206 View commit details
    Browse the repository at this point in the history

  6. fix build errors

    Jonathan Vexler committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    36d0b15 View commit details
    Browse the repository at this point in the history

Commits on Jun 5, 2024

  1. Merge branch 'master' into add_schema_evolution_to_fg_reader

    Jonathan Vexler committed Jun 5, 2024
    Configuration menu
    Copy the full SHA
    11862a3 View commit details
    Browse the repository at this point in the history

  2. make default value -1 for position column

    Jonathan Vexler committed Jun 5, 2024
    Configuration menu
    Copy the full SHA
    e710020 View commit details
    Browse the repository at this point in the history

  3. replace use of parquet reader with fg reader in the cdc iterator

    Jonathan Vexler committed Jun 5, 2024
    Configuration menu
    Copy the full SHA
    a4f3d9a View commit details
    Browse the repository at this point in the history