Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix(ingest): stateful redundant run skip handler #8467

Merged
merged 33 commits into from
Aug 28, 2023
Merged

fix(ingest): stateful redundant run skip handler #8467

merged 33 commits into from
Aug 28, 2023

Commits on Jul 19, 2023

  1. fix(ingest): stateful redundant run skip handler WIP

    @mayurinehate
    mayurinehate committed Jul 19, 2023
    Configuration menu
    Copy the full SHA
    46babb6 View commit details
    Browse the repository at this point in the history

  2. changes to consider suggested changes in time window

    @mayurinehate
    mayurinehate committed Jul 19, 2023
    Configuration menu
    Copy the full SHA
    02fc770 View commit details
    Browse the repository at this point in the history

  3. changes to update state only in case of complete success

    @mayurinehate
    mayurinehate committed Jul 19, 2023
    Configuration menu
    Copy the full SHA
    3f1487a View commit details
    Browse the repository at this point in the history

Commits on Jul 20, 2023

  1. add unit tests for should_skip_this_run and suggest_run_time_window

    @mayurinehate
    mayurinehate committed Jul 20, 2023
    Configuration menu
    Copy the full SHA
    d8be681 View commit details
    Browse the repository at this point in the history

Commits on Jul 21, 2023

  1. more tests, remove |

    @mayurinehate
    mayurinehate committed Jul 21, 2023
    Configuration menu
    Copy the full SHA
    da3904e View commit details
    Browse the repository at this point in the history

Commits on Jul 31, 2023

  1. Merge branch 'master' into redundant_run_fix

    @anshbansal
    anshbansal authored Jul 31, 2023
    Configuration menu
    Copy the full SHA
    887d3b7 View commit details
    Browse the repository at this point in the history

Commits on Aug 1, 2023

  1. Merge branch 'master' into redundant_run_fix

    @mayurinehate
    mayurinehate authored Aug 1, 2023
    Configuration menu
    Copy the full SHA
    2a1ac2e View commit details
    Browse the repository at this point in the history

Commits on Aug 3, 2023

  1. Merge branch 'master' into redundant_run_fix

    @mayurinehate
    mayurinehate authored Aug 3, 2023
    Configuration menu
    Copy the full SHA
    1b81436 View commit details
    Browse the repository at this point in the history

  2. small refractor and test for snowflake stateful ingestion

    @mayurinehate
    mayurinehate committed Aug 3, 2023
    Configuration menu
    Copy the full SHA
    6405f21 View commit details
    Browse the repository at this point in the history

Commits on Aug 4, 2023

  1. Merge branch 'master' into redundant_run_fix

    @mayurinehate
    mayurinehate authored Aug 4, 2023
    Configuration menu
    Copy the full SHA
    e35203a View commit details
    Browse the repository at this point in the history

  2. fix lint

    @mayurinehate
    mayurinehate committed Aug 4, 2023
    Configuration menu
    Copy the full SHA
    686e5f9 View commit details
    Browse the repository at this point in the history

Commits on Aug 8, 2023

  1. Merge branch 'master' into redundant_run_fix

    @mayurinehate
    mayurinehate authored Aug 8, 2023
    Configuration menu
    Copy the full SHA
    b0dc114 View commit details
    Browse the repository at this point in the history

Commits on Aug 14, 2023

  1. Merge branch 'master' into redundant_run_fix

    @mayurinehate
    mayurinehate authored Aug 14, 2023
    Configuration menu
    Copy the full SHA
    201ccbc View commit details
    Browse the repository at this point in the history

Commits on Aug 17, 2023

  1. Merge branch 'master' into redundant_run_fix

    @mayurinehate
    mayurinehate authored Aug 17, 2023
    Configuration menu
    Copy the full SHA
    3127e2e View commit details
    Browse the repository at this point in the history

  2. address review comments

    @mayurinehate
    mayurinehate committed Aug 17, 2023
    Configuration menu
    Copy the full SHA
    8ceb21c View commit details
    Browse the repository at this point in the history

Commits on Aug 18, 2023

  1. add timestamps in report

    @mayurinehate
    mayurinehate committed Aug 18, 2023
    Configuration menu
    Copy the full SHA
    9c275c9 View commit details
    Browse the repository at this point in the history

  2. rewrite redundant run conditions, more tests

    @mayurinehate
    mayurinehate committed Aug 18, 2023
    Configuration menu
    Copy the full SHA
    45c09d9 View commit details
    Browse the repository at this point in the history

  3. fix state for ignore_start_time_lineage and time reporting for lineage

    @mayurinehate
    mayurinehate committed Aug 18, 2023
    Configuration menu
    Copy the full SHA
    39ceb6a View commit details
    Browse the repository at this point in the history

  4. Merge branch 'master' into redundant_run_fix

    @mayurinehate
    mayurinehate authored Aug 18, 2023
    Configuration menu
    Copy the full SHA
    c3fe256 View commit details
    Browse the repository at this point in the history

  5. refractor to prefer mypy check over runtime failure

    @mayurinehate
    mayurinehate committed Aug 18, 2023
    Configuration menu
    Copy the full SHA
    b366ce6 View commit details
    Browse the repository at this point in the history

  6. Merge branch 'master' into redundant_run_fix

    @mayurinehate
    mayurinehate authored Aug 18, 2023
    Configuration menu
    Copy the full SHA
    73043c5 View commit details
    Browse the repository at this point in the history

  7. more changes, tests

    @mayurinehate
    mayurinehate committed Aug 18, 2023
    Configuration menu
    Copy the full SHA
    3013168 View commit details
    Browse the repository at this point in the history

  8. Merge remote-tracking branch 'refs/remotes/origin/redundant_run_fix' … 

    …into redundant_run_fix
    @mayurinehate
    mayurinehate committed Aug 18, 2023
    Configuration menu
    Copy the full SHA
    75a7abc View commit details
    Browse the repository at this point in the history

Commits on Aug 22, 2023

  1. Merge branch 'master' into redundant_run_fix

    @mayurinehate
    mayurinehate authored Aug 22, 2023
    Configuration menu
    Copy the full SHA
    126804c View commit details
    Browse the repository at this point in the history

  2. do not floor start time if absolute start time is specified

    @mayurinehate
    mayurinehate committed Aug 22, 2023
    Configuration menu
    Copy the full SHA
    7afc105 View commit details
    Browse the repository at this point in the history

Commits on Aug 23, 2023

  1. nit changes

    @mayurinehate
    mayurinehate committed Aug 23, 2023
    Configuration menu
    Copy the full SHA
    59b21d7 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'master' into redundant_run_fix

    @mayurinehate
    mayurinehate authored Aug 23, 2023
    Configuration menu
    Copy the full SHA
    8851ff2 View commit details
    Browse the repository at this point in the history

  3. suggested time in logs

    @mayurinehate
    mayurinehate committed Aug 23, 2023
    Configuration menu
    Copy the full SHA
    663e5e2 View commit details
    Browse the repository at this point in the history

  4. Merge remote-tracking branch 'refs/remotes/origin/redundant_run_fix' … 

    …into redundant_run_fix
    @mayurinehate
    mayurinehate committed Aug 23, 2023
    Configuration menu
    Copy the full SHA
    fd09d3a View commit details
    Browse the repository at this point in the history

  5. fix issue with zero usage aspects

    @mayurinehate
    mayurinehate committed Aug 23, 2023
    Configuration menu
    Copy the full SHA
    c286a8d View commit details
    Browse the repository at this point in the history

  6. Update metadata-ingestion/src/datahub/utilities/time.py

    @asikowitz
    asikowitz authored Aug 23, 2023
    Configuration menu
    Copy the full SHA
    a7a2c1c View commit details
    Browse the repository at this point in the history

Commits on Aug 25, 2023

  1. Merge branch 'master' into redundant_run_fix

    @asikowitz
    asikowitz committed Aug 25, 2023
    Configuration menu
    Copy the full SHA
    8c79d75 View commit details
    Browse the repository at this point in the history

Commits on Aug 28, 2023

  1. Merge branch 'master' into redundant_run_fix

    @mayurinehate
    mayurinehate authored Aug 28, 2023
    Configuration menu
    Copy the full SHA
    fe4702c View commit details
    Browse the repository at this point in the history