Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

test: Addition of test cases for the Data Preprocessor code #4

Open
wants to merge 30 commits into
base: dataloader-v2-impl
Choose a base branch
from
Open

test: Addition of test cases for the Data Preprocessor code #4

wants to merge 30 commits into from

Commits on Nov 8, 2024

  1. add first cut dataloader v2 ADR 

    Signed-off-by: Dushyant Behl <[email protected]>
    @dushyantbehl
    dushyantbehl committed Nov 8, 2024
    Configuration menu
    Copy the full SHA
    aa5dd18 View commit details
    Browse the repository at this point in the history

  2. add initial implementation of dataloader v2 

    Signed-off-by: Dushyant Behl <[email protected]>
    @dushyantbehl
    dushyantbehl committed Nov 8, 2024
    Configuration menu
    Copy the full SHA
    251902b View commit details
    Browse the repository at this point in the history

  3. Update tuning/data/data_processors.py 

    Co-authored-by: Will Johnson <[email protected]>
Signed-off-by: Dushyant Behl <[email protected]>
    @dushyantbehl @willmj
    dushyantbehl and willmj committed Nov 8, 2024
    Configuration menu
    Copy the full SHA
    ac148eb View commit details
    Browse the repository at this point in the history

Commits on Nov 11, 2024

  1. tests: reformat mock.patch to inside unit tests 

    Signed-off-by: Will Johnson <[email protected]>

fmt

Signed-off-by: Will Johnson <[email protected]>
    @willmj
    willmj committed Nov 11, 2024
    Configuration menu
    Copy the full SHA
    3fbeed0 View commit details
    Browse the repository at this point in the history

Commits on Nov 12, 2024

  1. Merge pull request foundation-model-stack#1 from willmj/dataloader-v2… 

    …-impl-unit-fix

tests: reformat `mock.patch` to inside unit tests
Signed-off-by: Dushyant Behl <[email protected]>
    @dushyantbehl
    dushyantbehl committed Nov 12, 2024
    Configuration menu
    Copy the full SHA
    4c0b109 View commit details
    Browse the repository at this point in the history

Commits on Nov 20, 2024

  1. Add data config argument 

    Signed-off-by: Dushyant Behl <[email protected]>
    @dushyantbehl
    dushyantbehl committed Nov 20, 2024
    Configuration menu
    Copy the full SHA
    96b9467 View commit details
    Browse the repository at this point in the history

Commits on Nov 21, 2024

  1. fix: Changes to support current implementation 

    Signed-off-by: Abhishek <[email protected]>
    @Abhishek-TAMU
    Abhishek-TAMU committed Nov 21, 2024
    Configuration menu
    Copy the full SHA
    163fe34 View commit details
    Browse the repository at this point in the history

Commits on Nov 22, 2024

  1. add initial implementation of dataloader v2 

    Signed-off-by: Dushyant Behl <[email protected]>
    @dushyantbehl
    dushyantbehl committed Nov 22, 2024
    Configuration menu
    Copy the full SHA
    137c2d4 View commit details
    Browse the repository at this point in the history

  2. tests: reformat mock.patch to inside unit tests 

    Signed-off-by: Will Johnson <[email protected]>

fmt

Signed-off-by: Will Johnson <[email protected]>
    @willmj @dushyantbehl
    willmj authored and dushyantbehl committed Nov 22, 2024
    Configuration menu
    Copy the full SHA
    b63a9a5 View commit details
    Browse the repository at this point in the history

  3. Add data config argument to data preprocessor 

    Signed-off-by: Dushyant Behl <[email protected]>
    @dushyantbehl
    dushyantbehl committed Nov 22, 2024
    Configuration menu
    Copy the full SHA
    84cf65c View commit details
    Browse the repository at this point in the history

  4. fix: Changes to support current implementation 

    Signed-off-by: Abhishek <[email protected]>
    @Abhishek-TAMU @dushyantbehl
    Abhishek-TAMU authored and dushyantbehl committed Nov 22, 2024
    Configuration menu
    Copy the full SHA
    f299b39 View commit details
    Browse the repository at this point in the history

  5. Ensure data handling is done within process dataargs 

    Removes unused dead code after adding the new framework and refactors
some test cases and files.

Signed-off-by: Dushyant Behl <[email protected]>
    @dushyantbehl
    dushyantbehl committed Nov 22, 2024
    Configuration menu
    Copy the full SHA
    32c8733 View commit details
    Browse the repository at this point in the history

  6. Remove accelerator in favor of torch distributed check for multi node 

    data preprocessing

Signed-off-by: Dushyant Behl <[email protected]>
    @dushyantbehl
    dushyantbehl committed Nov 22, 2024
    Configuration menu
    Copy the full SHA
    cea24ab View commit details
    Browse the repository at this point in the history

  7. rename tests/data to tests/testdata 

    Signed-off-by: Dushyant Behl <[email protected]>
    @dushyantbehl
    dushyantbehl committed Nov 22, 2024
    Configuration menu
    Copy the full SHA
    43626ed View commit details
    Browse the repository at this point in the history

  8. Refactor data util tests as data handler tests. 

    Signed-off-by: Dushyant Behl <[email protected]>
    @dushyantbehl
    dushyantbehl committed Nov 22, 2024
    Configuration menu
    Copy the full SHA
    3bd42b5 View commit details
    Browse the repository at this point in the history

  9. fix: add __init__.py to add tuning.data to python package 

    Signed-off-by: Will Johnson <[email protected]>
    @willmj
    willmj committed Nov 22, 2024
    Configuration menu
    Copy the full SHA
    7da1bf8 View commit details
    Browse the repository at this point in the history

  10. Merge remote-tracking branch 'dushyant_fms_hf_tuning/dataloader-v2-im… 

    …pl' into data_preprocessor_dushyant

Signed-off-by: Abhishek <[email protected]>
    @Abhishek-TAMU
    Abhishek-TAMU committed Nov 22, 2024
    Configuration menu
    Copy the full SHA
    13ebab9 View commit details
    Browse the repository at this point in the history

  11. fix: multi GPU prepare training dataset 

    Signed-off-by: Will Johnson <[email protected]>
    @willmj
    willmj committed Nov 22, 2024
    Configuration menu
    Copy the full SHA
    098e9d8 View commit details
    Browse the repository at this point in the history

  12. test: Changes by Dushyant 

    Signed-off-by: Abhishek <[email protected]>
    @Abhishek-TAMU
    Abhishek-TAMU committed Nov 22, 2024
    Configuration menu
    Copy the full SHA
    f4b17ba View commit details
    Browse the repository at this point in the history

  13. fix: minor changes 

    Signed-off-by: Abhishek <[email protected]>
    @Abhishek-TAMU
    Abhishek-TAMU committed Nov 22, 2024
    Configuration menu
    Copy the full SHA
    aab08a3 View commit details
    Browse the repository at this point in the history

  14. fix: lint 

    Signed-off-by: Will Johnson <[email protected]>
    @willmj
    willmj committed Nov 22, 2024
    Configuration menu
    Copy the full SHA
    643313a View commit details
    Browse the repository at this point in the history

  15. Merge remote-tracking branch 'dushyant_fms_hf_tuning/dataloader-v2-im… 

    …pl' into data_preprocessor_dushyant
    @Abhishek-TAMU
    Abhishek-TAMU committed Nov 22, 2024
    Configuration menu
    Copy the full SHA
    14575af View commit details
    Browse the repository at this point in the history

Commits on Nov 23, 2024

  1. fix: Add TODO 

    Signed-off-by: Will Johnson <[email protected]>
    @willmj
    willmj committed Nov 23, 2024
    Configuration menu
    Copy the full SHA
    c078d07 View commit details
    Browse the repository at this point in the history

  2. Merge remote-tracking branch 'dushyant_fms_hf_tuning/dataloader-v2-im… 

    …pl' into data_preprocessor_dushyant
    @Abhishek-TAMU
    Abhishek-TAMU committed Nov 23, 2024
    Configuration menu
    Copy the full SHA
    8aace79 View commit details
    Browse the repository at this point in the history

  3. tests: Added test case for load_dataset function of class HFBasedData… 

    …PreProcessor

Signed-off-by: Abhishek <[email protected]>
    @Abhishek-TAMU
    Abhishek-TAMU committed Nov 23, 2024
    Configuration menu
    Copy the full SHA
    6cad504 View commit details
    Browse the repository at this point in the history

Commits on Nov 25, 2024

  1. test: _process_dataconfig_file 

    Signed-off-by: Abhishek <[email protected]>
    @Abhishek-TAMU
    Abhishek-TAMU committed Nov 25, 2024
    Configuration menu
    Copy the full SHA
    086b8a1 View commit details
    Browse the repository at this point in the history

  2. test: _process_dataconfig_file 

    Signed-off-by: Abhishek <[email protected]>
    @Abhishek-TAMU
    Abhishek-TAMU committed Nov 25, 2024
    Configuration menu
    Copy the full SHA
    5b17f66 View commit details
    Browse the repository at this point in the history

  3. test: _process_dataconfig_file 

    Signed-off-by: Abhishek <[email protected]>
    @Abhishek-TAMU
    Abhishek-TAMU committed Nov 25, 2024
    Configuration menu
    Copy the full SHA
    a0bd939 View commit details
    Browse the repository at this point in the history

Commits on Nov 26, 2024

  1. test: add test for process_dataset_configs in HFBasedDataPreProcessor 

    Signed-off-by: Will Johnson <[email protected]>
    @willmj
    willmj committed Nov 26, 2024
    Configuration menu
    Copy the full SHA
    2a0777c View commit details
    Browse the repository at this point in the history

  2. Merge remote-tracking branch 'dushyant_fms_hf_tuning/dataloader-v2-im… 

    …pl' into data_preprocessor_dushyant

Signed-off-by: Abhishek <[email protected]>
    @Abhishek-TAMU
    Abhishek-TAMU committed Nov 26, 2024
    Configuration menu
    Copy the full SHA
    65748d4 View commit details
    Browse the repository at this point in the history