Add more unittest #304

pan-x-c · 2024-04-24T03:59:36Z

setup local actions runner on a GPU machine
use docker-compose to setup a cluster
add more unit tests
- single process
- multi process
- multi process with GPU
- ray with CPU
- ray with GPU

github-actions · 2024-05-29T09:32:20Z

This PR is marked as stale because there has been no activity for 21 days. Remove stale label or add new comments or this PR will be closed in 3 day.

.github/workflows/docker/docker-compose.yml

data_juicer/utils/unittest_utils.py

.github/workflows/unittest.yml

* modelscope-sora news (#323) * News/modelscope sora (#327) * modelscope-sora news * remove empower * debug for gpu rank for analyser (#329) * debug for gpu rank for analyser * spec_numprocs -> num_proc * Add more unittest (#304) * add unittest env with gpu * fix unittest yml * add environment for unittest * update workflow trigger * update install step * fix install command * update working dir * update container * update working dir * change working directory * change working directory * change working directory * change working directory * change unittest * use test tag * finish tag support * support run op with different executro * fix pre-commit * add hf mirror * add hf mirror * run all test in standalone mode by default * ignore image face ratio * update tags * add ray testcase * add ray test in workflow * update ray unittest workflow * delete old unittest --------- Co-authored-by: root <panxuchen> * Add source tag (#317) * add source tag for some mapper op * fix no attribute 'current_tag' when executing local tests * move op process logic from executor to base op * fix typo * move export outside op * init refactor * update analyser * fix format * clean up * bring back batch mapper * Improve fault tolerance & Fix Ray executor * fix wrapper * fix batched filter * Remove use_actor as it is not compatible with the refactored OP clas, unless the dataset class is refactored * make wrappers work with unittests * Compatible with unit tests and works with ray * fix unittest * fix wrappers with ray, map, filter * unify unittests * wrap deduplicators * Compatible with non-batched calls * Class-level wrappers - compatible with dataset.filter - bring back nested wrappers * Instance-level wrappers * Refined instance-level wrappers - Remove incomplete dataset.filter wrappers - Simplify code - Stack wrappers * fix use_cuda * Refactor dataset (#348) * refactor dataset * update unittest with DJDataset * fix unittest * update ray data load * add test * ray read json * update docker image version * actor is no longer supported * Regress filter's stats export logic --------- Co-authored-by: BeachWang <[email protected]> Co-authored-by: Xuchen Pan <[email protected]> Co-authored-by: chenhesen <[email protected]> Co-authored-by: garyzhang99 <[email protected]>

* Refactor OP & Dataset (#336) * modelscope-sora news (#323) * News/modelscope sora (#327) * modelscope-sora news * remove empower * debug for gpu rank for analyser (#329) * debug for gpu rank for analyser * spec_numprocs -> num_proc * Add more unittest (#304) * add unittest env with gpu * fix unittest yml * add environment for unittest * update workflow trigger * update install step * fix install command * update working dir * update container * update working dir * change working directory * change working directory * change working directory * change working directory * change unittest * use test tag * finish tag support * support run op with different executro * fix pre-commit * add hf mirror * add hf mirror * run all test in standalone mode by default * ignore image face ratio * update tags * add ray testcase * add ray test in workflow * update ray unittest workflow * delete old unittest --------- Co-authored-by: root <panxuchen> * Add source tag (#317) * add source tag for some mapper op * fix no attribute 'current_tag' when executing local tests * move op process logic from executor to base op * fix typo * move export outside op * init refactor * update analyser * fix format * clean up * bring back batch mapper * Improve fault tolerance & Fix Ray executor * fix wrapper * fix batched filter * Remove use_actor as it is not compatible with the refactored OP clas, unless the dataset class is refactored * make wrappers work with unittests * Compatible with unit tests and works with ray * fix unittest * fix wrappers with ray, map, filter * unify unittests * wrap deduplicators * Compatible with non-batched calls * Class-level wrappers - compatible with dataset.filter - bring back nested wrappers * Instance-level wrappers * Refined instance-level wrappers - Remove incomplete dataset.filter wrappers - Simplify code - Stack wrappers * fix use_cuda * Refactor dataset (#348) * refactor dataset * update unittest with DJDataset * fix unittest * update ray data load * add test * ray read json * update docker image version * actor is no longer supported * Regress filter's stats export logic --------- Co-authored-by: BeachWang <[email protected]> Co-authored-by: Xuchen Pan <[email protected]> Co-authored-by: chenhesen <[email protected]> Co-authored-by: garyzhang99 <[email protected]> * minor fix * fix num_proc default None --------- Co-authored-by: Ce Ge (戈策) <[email protected]> Co-authored-by: BeachWang <[email protected]> Co-authored-by: Xuchen Pan <[email protected]> Co-authored-by: chenhesen <[email protected]> Co-authored-by: garyzhang99 <[email protected]> Co-authored-by: null <[email protected]>

add unittest env with gpu

2e4c50d

pan-x-c added the enhancement New feature or request label Apr 24, 2024

pan-x-c self-assigned this Apr 24, 2024

root added 3 commits April 24, 2024 04:12

fix unittest yml

3edd333

add environment for unittest

aa59509

update workflow trigger

5d7c5dd

pan-x-c had a problem deploying to Testing April 24, 2024 04:28 — with GitHub Actions Failure

pan-x-c had a problem deploying to Testing April 24, 2024 05:26 — with GitHub Actions Failure

pan-x-c had a problem deploying to Testing April 24, 2024 05:30 — with GitHub Actions Failure

update install step

b9e9f0c

pan-x-c had a problem deploying to Testing April 24, 2024 05:39 — with GitHub Actions Failure

fix install command

fe7a71a

pan-x-c had a problem deploying to Testing April 24, 2024 05:52 — with GitHub Actions Failure

pan-x-c had a problem deploying to Testing April 24, 2024 05:57 — with GitHub Actions Failure

update working dir

06141ca

pan-x-c had a problem deploying to Testing April 24, 2024 05:58 — with GitHub Actions Failure

update container

c89a92a

pan-x-c had a problem deploying to Testing April 24, 2024 07:14 — with GitHub Actions Failure

pan-x-c had a problem deploying to Testing April 24, 2024 07:32 — with GitHub Actions Failure

pan-x-c had a problem deploying to Testing April 24, 2024 08:25 — with GitHub Actions Failure

pan-x-c had a problem deploying to Testing April 24, 2024 08:26 — with GitHub Actions Error

update working dir

1f5f522

pan-x-c temporarily deployed to Testing April 24, 2024 10:24 — with GitHub Actions Inactive

change working directory

9142f50

pan-x-c had a problem deploying to Testing April 24, 2024 11:26 — with GitHub Actions Failure

change working directory

926fbb5

pan-x-c had a problem deploying to Testing April 24, 2024 11:42 — with GitHub Actions Failure

change working directory

905488f

pan-x-c had a problem deploying to Testing April 24, 2024 11:49 — with GitHub Actions Failure

change working directory

74c3465

add hf mirror

319c450

pan-x-c had a problem deploying to Testing April 26, 2024 05:16 — with GitHub Actions Failure

add hf mirror

de318af

pan-x-c temporarily deployed to Testing April 26, 2024 05:18 — with GitHub Actions Inactive

run all test in standalone mode by default

6863cd0

pan-x-c had a problem deploying to Testing April 26, 2024 05:21 — with GitHub Actions Failure

ignore image face ratio

1a251c9

pan-x-c temporarily deployed to Testing April 26, 2024 06:02 — with GitHub Actions Inactive

pan-x-c added 3 commits May 7, 2024 15:43

update tags

aef6739

Merge branch 'main' into test/pxc/use_local_runner

accba06

add ray testcase

6e6409d

pan-x-c had a problem deploying to Testing May 7, 2024 12:19 — with GitHub Actions Failure

add ray test in workflow

a2091d8

pan-x-c had a problem deploying to Testing May 7, 2024 12:24 — with GitHub Actions Failure

pan-x-c changed the title ~~[WIP] Add more unittest~~ Add more unittest May 7, 2024

update ray unittest workflow

be81489

pan-x-c temporarily deployed to Testing May 8, 2024 01:49 — with GitHub Actions Inactive

github-actions bot added the stale-pr label May 29, 2024

yxdyc reviewed May 31, 2024

View reviewed changes

.github/workflows/docker/docker-compose.yml Show resolved Hide resolved

data_juicer/utils/unittest_utils.py Show resolved Hide resolved

.github/workflows/unittest.yml Outdated Show resolved Hide resolved

github-actions bot removed the stale-pr label May 31, 2024

merge main

50a45b2

pan-x-c had a problem deploying to Testing June 17, 2024 02:51 — with GitHub Actions Failure

pan-x-c temporarily deployed to Testing June 17, 2024 02:51 — with GitHub Actions Inactive

delete old unittest

aa68b49

pan-x-c temporarily deployed to Testing June 17, 2024 06:07 — with GitHub Actions Inactive

drcege approved these changes Jun 26, 2024

View reviewed changes

yxdyc merged commit c749a28 into main Jun 26, 2024
3 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add more unittest #304

Add more unittest #304

pan-x-c commented Apr 24, 2024

github-actions bot commented May 29, 2024

Add more unittest #304

Add more unittest #304

Conversation

pan-x-c commented Apr 24, 2024

github-actions bot commented May 29, 2024