Skip to content
CI

fix torch multi-gpu training hanging and timeout problem #4946

Sign in to view logs

fix torch multi-gpu training hanging and timeout problem

fix torch multi-gpu training hanging and timeout problem #4946

Re-run triggered November 15, 2023 11:07
@albertzalbertz
#1461
multi_gpu_syncronization_fix
Status Success
Total duration 4m 4s
Artifacts –

main.yml

on: pull_request
Matrix: base-tests
Matrix: pycharm-inspect
Matrix: rf-tests
Matrix: tf-tests
Matrix: torch-tests
Fit to window
Zoom out
Zoom in

Annotations

53 warnings
base-tests (3.7, TEST=tensor)
The following actions uses node12 which is deprecated and will be forced to run on node16: actions/cache@v2. For more info: https://github.blog/changelog/2023-06-13-github-actions-all-actions-will-run-on-node16-instead-of-node12-by-default/
base-tests (3.7, TEST=TranslationDataset)
The following actions uses node12 which is deprecated and will be forced to run on node16: actions/cache@v2. For more info: https://github.blog/changelog/2023-06-13-github-actions-all-actions-will-run-on-node16-instead-of-node12-by-default/
base-tests (3.7, TEST=TaskSystem_SharedMem)
The following actions uses node12 which is deprecated and will be forced to run on node16: actions/cache@v2. For more info: https://github.blog/changelog/2023-06-13-github-actions-all-actions-will-run-on-node16-instead-of-node12-by-default/
base-tests (3.7, TEST=Util)
The following actions uses node12 which is deprecated and will be forced to run on node16: actions/cache@v2. For more info: https://github.blog/changelog/2023-06-13-github-actions-all-actions-will-run-on-node16-instead-of-node12-by-default/
base-tests (3.7, TEST=Pretrain)
The following actions uses node12 which is deprecated and will be forced to run on node16: actions/cache@v2. For more info: https://github.blog/changelog/2023-06-13-github-actions-all-actions-will-run-on-node16-instead-of-node12-by-default/
base-tests (3.7, TEST=MultiProcDataset)
The following actions uses node12 which is deprecated and will be forced to run on node16: actions/cache@v2. For more info: https://github.blog/changelog/2023-06-13-github-actions-all-actions-will-run-on-node16-instead-of-node12-by-default/
base-tests (3.7, TEST=TaskSystem)
The following actions uses node12 which is deprecated and will be forced to run on node16: actions/cache@v2. For more info: https://github.blog/changelog/2023-06-13-github-actions-all-actions-will-run-on-node16-instead-of-node12-by-default/
base-tests (3.7, TEST=GeneratingDataset)
The following actions uses node12 which is deprecated and will be forced to run on node16: actions/cache@v2. For more info: https://github.blog/changelog/2023-06-13-github-actions-all-actions-will-run-on-node16-instead-of-node12-by-default/
base-tests (3.7, TEST=SprintDataset)
The following actions uses node12 which is deprecated and will be forced to run on node16: actions/cache@v2. For more info: https://github.blog/changelog/2023-06-13-github-actions-all-actions-will-run-on-node16-instead-of-node12-by-default/
base-tests (3.7, TEST=HDFDataset)
The following actions uses node12 which is deprecated and will be forced to run on node16: actions/cache@v2. For more info: https://github.blog/changelog/2023-06-13-github-actions-all-actions-will-run-on-node16-instead-of-node12-by-default/
base-tests (3.7, TEST=LearningRateControl)
The following actions uses node12 which is deprecated and will be forced to run on node16: actions/cache@v2. For more info: https://github.blog/changelog/2023-06-13-github-actions-all-actions-will-run-on-node16-instead-of-node12-by-default/
base-tests (3.7, TEST=hdf_dump)
The following actions uses node12 which is deprecated and will be forced to run on node16: actions/cache@v2. For more info: https://github.blog/changelog/2023-06-13-github-actions-all-actions-will-run-on-node16-instead-of-node12-by-default/
rf-tests (3.7, 1.13.1, 2.10.0, TEST=rf_signal)
The following actions uses node12 which is deprecated and will be forced to run on node16: actions/cache@v2. For more info: https://github.blog/changelog/2023-06-13-github-actions-all-actions-will-run-on-node16-instead-of-node12-by-default/
pycharm-inspect (3.7, 2.4.2)
The following actions uses node12 which is deprecated and will be forced to run on node16: actions/cache@v2. For more info: https://github.blog/changelog/2023-06-13-github-actions-all-actions-will-run-on-node16-instead-of-node12-by-default/
base-tests (3.7, TEST=Dataset)
The following actions uses node12 which is deprecated and will be forced to run on node16: actions/cache@v2. For more info: https://github.blog/changelog/2023-06-13-github-actions-all-actions-will-run-on-node16-instead-of-node12-by-default/
rf-tests (3.7, 1.13.1, 2.10.0, TEST=rf_normalization)
The following actions uses node12 which is deprecated and will be forced to run on node16: actions/cache@v2. For more info: https://github.blog/changelog/2023-06-13-github-actions-all-actions-will-run-on-node16-instead-of-node12-by-default/
base-tests (3.7, TEST=fork_exec)
The following actions uses node12 which is deprecated and will be forced to run on node16: actions/cache@v2. For more info: https://github.blog/changelog/2023-06-13-github-actions-all-actions-will-run-on-node16-instead-of-node12-by-default/
rf-tests (3.7, 1.13.1, 2.10.0, TEST=rf_rec)
The following actions uses node12 which is deprecated and will be forced to run on node16: actions/cache@v2. For more info: https://github.blog/changelog/2023-06-13-github-actions-all-actions-will-run-on-node16-instead-of-node12-by-default/
rf-tests (3.7, 1.13.1, 2.10.0, TEST=rf_math)
The following actions uses node12 which is deprecated and will be forced to run on node16: actions/cache@v2. For more info: https://github.blog/changelog/2023-06-13-github-actions-all-actions-will-run-on-node16-instead-of-node12-by-default/
rf-tests (3.7, 1.13.1, 2.10.0, TEST=rf_reduce)
The following actions uses node12 which is deprecated and will be forced to run on node16: actions/cache@v2. For more info: https://github.blog/changelog/2023-06-13-github-actions-all-actions-will-run-on-node16-instead-of-node12-by-default/
rf-tests (3.7, 1.13.1, 2.10.0, TEST=rf_loop)
The following actions uses node12 which is deprecated and will be forced to run on node16: actions/cache@v2. For more info: https://github.blog/changelog/2023-06-13-github-actions-all-actions-will-run-on-node16-instead-of-node12-by-default/
rf-tests (3.7, 1.13.1, 2.10.0, TEST=rf_label_smoothing)
The following actions uses node12 which is deprecated and will be forced to run on node16: actions/cache@v2. For more info: https://github.blog/changelog/2023-06-13-github-actions-all-actions-will-run-on-node16-instead-of-node12-by-default/
rf-tests (3.7, 1.13.1, 2.10.0, TEST=rf_encoder_conformer)
The following actions uses node12 which is deprecated and will be forced to run on node16: actions/cache@v2. For more info: https://github.blog/changelog/2023-06-13-github-actions-all-actions-will-run-on-node16-instead-of-node12-by-default/
rf-tests (3.7, 1.13.1, 2.10.0, TEST=rf_conv)
The following actions uses node12 which is deprecated and will be forced to run on node16: actions/cache@v2. For more info: https://github.blog/changelog/2023-06-13-github-actions-all-actions-will-run-on-node16-instead-of-node12-by-default/
rf-tests (3.7, 1.13.1, 2.10.0, TEST=rf_const)
The following actions uses node12 which is deprecated and will be forced to run on node16: actions/cache@v2. For more info: https://github.blog/changelog/2023-06-13-github-actions-all-actions-will-run-on-node16-instead-of-node12-by-default/
rf-tests (3.7, 1.13.1, 2.10.0, TEST=rf_gradient)
The following actions uses node12 which is deprecated and will be forced to run on node16: actions/cache@v2. For more info: https://github.blog/changelog/2023-06-13-github-actions-all-actions-will-run-on-node16-instead-of-node12-by-default/
rf-tests (3.7, 1.13.1, 2.10.0, TEST=rf_cond)
The following actions uses node12 which is deprecated and will be forced to run on node16: actions/cache@v2. For more info: https://github.blog/changelog/2023-06-13-github-actions-all-actions-will-run-on-node16-instead-of-node12-by-default/
rf-tests (3.7, 1.13.1, 2.10.0, TEST=rf_attention)
The following actions uses node12 which is deprecated and will be forced to run on node16: actions/cache@v2. For more info: https://github.blog/changelog/2023-06-13-github-actions-all-actions-will-run-on-node16-instead-of-node12-by-default/
rf-tests (3.7, 1.13.1, 2.10.0, TEST=rf_container)
The following actions uses node12 which is deprecated and will be forced to run on node16: actions/cache@v2. For more info: https://github.blog/changelog/2023-06-13-github-actions-all-actions-will-run-on-node16-instead-of-node12-by-default/
rf-tests (3.7, 1.13.1, 2.10.0, TEST=rf_array)
The following actions uses node12 which is deprecated and will be forced to run on node16: actions/cache@v2. For more info: https://github.blog/changelog/2023-06-13-github-actions-all-actions-will-run-on-node16-instead-of-node12-by-default/
tf-tests (TEST=TFNetworkSigProcLayer, 3.7, 1.15.3)
The following actions uses node12 which is deprecated and will be forced to run on node16: actions/cache@v2. For more info: https://github.blog/changelog/2023-06-13-github-actions-all-actions-will-run-on-node16-instead-of-node12-by-default/
tf-tests (TEST=demos, 3.7, 2.3.0)
The following actions uses node12 which is deprecated and will be forced to run on node16: actions/cache@v2. For more info: https://github.blog/changelog/2023-06-13-github-actions-all-actions-will-run-on-node16-instead-of-node12-by-default/
rf-tests (3.7, 1.13.1, 2.10.0, TEST=rf_base)
The following actions uses node12 which is deprecated and will be forced to run on node16: actions/cache@v2. For more info: https://github.blog/changelog/2023-06-13-github-actions-all-actions-will-run-on-node16-instead-of-node12-by-default/
tf-tests (TEST=TFEngine, 3.7, 2.3.0)
The following actions uses node12 which is deprecated and will be forced to run on node16: actions/cache@v2. For more info: https://github.blog/changelog/2023-06-13-github-actions-all-actions-will-run-on-node16-instead-of-node12-by-default/
tf-tests (TEST=TFNativeOp, 3.7, 1.14.0)
The following actions uses node12 which is deprecated and will be forced to run on node16: actions/cache@v2. For more info: https://github.blog/changelog/2023-06-13-github-actions-all-actions-will-run-on-node16-instead-of-node12-by-default/
tf-tests (TEST=TFEngine, 3.7, 1.15.3)
The following actions uses node12 which is deprecated and will be forced to run on node16: actions/cache@v2. For more info: https://github.blog/changelog/2023-06-13-github-actions-all-actions-will-run-on-node16-instead-of-node12-by-default/
tf-tests (3.7, 2.10.0, TEST=tools)
The following actions uses node12 which is deprecated and will be forced to run on node16: actions/cache@v2. For more info: https://github.blog/changelog/2023-06-13-github-actions-all-actions-will-run-on-node16-instead-of-node12-by-default/
tf-tests (3.7, 2.10.0, TEST=TFUtil)
The following actions uses node12 which is deprecated and will be forced to run on node16: actions/cache@v2. For more info: https://github.blog/changelog/2023-06-13-github-actions-all-actions-will-run-on-node16-instead-of-node12-by-default/
tf-tests (TEST=TFUtil, 3.7, 1.15.3)
The following actions uses node12 which is deprecated and will be forced to run on node16: actions/cache@v2. For more info: https://github.blog/changelog/2023-06-13-github-actions-all-actions-will-run-on-node16-instead-of-node12-by-default/
tf-tests (3.7, 2.10.0, TEST=TFUpdater)
The following actions uses node12 which is deprecated and will be forced to run on node16: actions/cache@v2. For more info: https://github.blog/changelog/2023-06-13-github-actions-all-actions-will-run-on-node16-instead-of-node12-by-default/
tf-tests (3.7, 2.10.0, TEST=TFNetworkLayer)
The following actions uses node12 which is deprecated and will be forced to run on node16: actions/cache@v2. For more info: https://github.blog/changelog/2023-06-13-github-actions-all-actions-will-run-on-node16-instead-of-node12-by-default/
tf-tests (3.7, 2.10.0, TEST=TFEngine)
The following actions uses node12 which is deprecated and will be forced to run on node16: actions/cache@v2. For more info: https://github.blog/changelog/2023-06-13-github-actions-all-actions-will-run-on-node16-instead-of-node12-by-default/
tf-tests (3.7, 2.10.0, TEST=TFNativeOp)
The following actions uses node12 which is deprecated and will be forced to run on node16: actions/cache@v2. For more info: https://github.blog/changelog/2023-06-13-github-actions-all-actions-will-run-on-node16-instead-of-node12-by-default/
tf-tests (3.7, 2.10.0, TEST=TFNetworkRecLayer)
The following actions uses node12 which is deprecated and will be forced to run on node16: actions/cache@v2. For more info: https://github.blog/changelog/2023-06-13-github-actions-all-actions-will-run-on-node16-instead-of-node12-by-default/
torch-tests (3.7, 1.13.1, TEST=torch_frontend)
The following actions uses node12 which is deprecated and will be forced to run on node16: actions/cache@v2. For more info: https://github.blog/changelog/2023-06-13-github-actions-all-actions-will-run-on-node16-instead-of-node12-by-default/
tf-tests (3.7, 2.10.0, TEST=SprintInterface)
The following actions uses node12 which is deprecated and will be forced to run on node16: actions/cache@v2. For more info: https://github.blog/changelog/2023-06-13-github-actions-all-actions-will-run-on-node16-instead-of-node12-by-default/
torch-tests (3.7, 1.13.1, TEST=torch_engine)
The following actions uses node12 which is deprecated and will be forced to run on node16: actions/cache@v2. For more info: https://github.blog/changelog/2023-06-13-github-actions-all-actions-will-run-on-node16-instead-of-node12-by-default/
tf-tests (3.7, 2.10.0, TEST=Fsa)
The following actions uses node12 which is deprecated and will be forced to run on node16: actions/cache@v2. For more info: https://github.blog/changelog/2023-06-13-github-actions-all-actions-will-run-on-node16-instead-of-node12-by-default/
tf-tests (3.7, 2.10.0, TEST=Log)
The following actions uses node12 which is deprecated and will be forced to run on node16: actions/cache@v2. For more info: https://github.blog/changelog/2023-06-13-github-actions-all-actions-will-run-on-node16-instead-of-node12-by-default/
torch-tests (3.7, 1.13.1, TEST=torch_internal_frontend)
The following actions uses node12 which is deprecated and will be forced to run on node16: actions/cache@v2. For more info: https://github.blog/changelog/2023-06-13-github-actions-all-actions-will-run-on-node16-instead-of-node12-by-default/
tf-tests (3.7, 2.10.0, TEST=Config)
The following actions uses node12 which is deprecated and will be forced to run on node16: actions/cache@v2. For more info: https://github.blog/changelog/2023-06-13-github-actions-all-actions-will-run-on-node16-instead-of-node12-by-default/
torch-tests (3.7, 1.13.1, TEST=torch_dataset)
The following actions uses node12 which is deprecated and will be forced to run on node16: actions/cache@v2. For more info: https://github.blog/changelog/2023-06-13-github-actions-all-actions-will-run-on-node16-instead-of-node12-by-default/
torch-tests (3.7, 1.13.1, TEST=demos RETURNN_DISABLE_TF=1)
The following actions uses node12 which is deprecated and will be forced to run on node16: actions/cache@v2. For more info: https://github.blog/changelog/2023-06-13-github-actions-all-actions-will-run-on-node16-instead-of-node12-by-default/