Error: Fatal: UD endpoint 0x129b9b0 to <no debug data>: unhandled timeout error

[adfaure]

Hello,

First of all, I hope this is the right place to post my question, I am trying to run an MPI application in a cluster with omnipath.

However, I am a out of idea to debug because:

• The error is (or looks like it is) random. On a cluster of 30 nodes it occurs on a few nodes (If I remove them from the hostfile it works).
• The error disappear. If I insist (which happens if I try to debug) the error eventually disappears and everything works fine.
• I can't get information from strace because, the PID crashing is the only one that doesn't write an output file.

I am using ucx 1.15 (I did have the issue with 1.14 as well). And my setup is a bit uncommon as I am using mpi with nixos (the packages are defined here: https://github.com/oar-team/nur-kapack/tree/ucx-upgrade)

If anyone as an idea on how to debug this I'll gladly accept your help.
Thanks!

This is the command that I execute:

mpirun -n 128 --hostfile $OAR_NODE_FILE -mca pls_rsh_agent oarsh -mca pml ucx --mca btl ^vader,tcp,openib,uct -x UCX_NET_DEVICES=mlx5_1:1 cg.C.mpi

and this is the error:

[node19:3747 :0:3747]       ud_ep.c:278  Fatal: UD endpoint 0x129b9b0 to <no debug data>: unhandled timeout error
[node3:3949 :0:3949]       ud_ep.c:278  Fatal: UD endpoint 0x1622850 to <no debug data>: unhandled timeout error
[node3:3957 :0:3957]       ud_ep.c:278  Fatal: UD endpoint 0x910850 to <no debug data>: unhandled timeout error
==== backtrace (tid:   3747) ====
0 0x00000000000317f1 ucs_fatal_error_format.localalias()  ???:0
1 0x00000000000852c8 uct_ud_ep_deferred_timeout_handler()  ???:0
2 0x000000000002640f ucs_callbackq_slow_proxy()  ???:0
3 0x000000000005ed0a ucp_worker_progress()  ???:0
4 0x00000000000435bc opal_progress()  ???:0
5 0x0000000000088e85 ompi_request_default_wait_all()  ???:0
6 0x00000000000ea78f ompi_coll_base_bcast_intra_generic()  ???:0
7 0x000000000010b48c ompi_coll_tuned_bcast_intra_dec_fixed()  ???:0
8 0x00000000000a20a6 MPI_Bcast()  ???:0
9 0x0000000000052082 pmpi_bcast_()  ???:0
10 0x000000000040476f bcast_inputs_()  ???:0
11 0x0000000000403c17 read_input_()  ???:0
12 0x0000000000402400 MAIN__()  ???:0
13 0x0000000000402934 main()  ???:0
14 0x0000000000023ace __libc_start_call_main()  ???:0
15 0x0000000000023b89 __libc_start_main_alias_2()  ???:0
16 0x00000000004022c5 _start()  ???:0
=================================

Program received signal SIGABRT: Process abort signal.

Backtrace for this error:
==== backtrace (tid:   3949) ====
0 0x00000000000317f1 ucs_fatal_error_format.localalias()  ???:0
1 0x00000000000852c8 uct_ud_ep_deferred_timeout_handler()  ???:0
2 0x000000000002640f ucs_callbackq_slow_proxy()  ???:0
3 0x000000000005ed0a ucp_worker_progress()  ???:0
4 0x00000000000435bc opal_progress()  ???:0
5 0x0000000000088e85 ompi_request_default_wait_all()  ???:0
6 0x00000000000ea78f ompi_coll_base_bcast_intra_generic()  ???:0
7 0x000000000010b48c ompi_coll_tuned_bcast_intra_dec_fixed()  ???:0
==== backtrace (tid:   3957) ====
0 0x00000000000317f1 ucs_fatal_error_format.localalias()  ???:0
1 0x00000000000852c8 uct_ud_ep_deferred_timeout_handler()  ???:0
2 0x000000000002640f ucs_callbackq_slow_proxy()  ???:0
3 0x000000000005ed0a ucp_worker_progress()  ???:0
4 0x00000000000435bc opal_progress()  ???:0
5 0x0000000000088e85 ompi_request_default_wait_all()  ???:0
6 0x00000000000ea78f ompi_coll_base_bcast_intra_generic()  ???:0
7 0x000000000010b48c ompi_coll_tuned_bcast_intra_dec_fixed()  ???:0
8 0x00000000000a20a6 MPI_Bcast()  ???:0
9 0x0000000000052082 pmpi_bcast_()  ???:0
10 0x000000000040476f bcast_inputs_()  ???:0
11 0x0000000000403c17 read_input_()  ???:0
12 0x0000000000402400 MAIN__()  ???:0
13 0x0000000000402934 main()  ???:0
14 0x0000000000023ace __libc_start_call_main()  ???:0
15 0x0000000000023b89 __libc_start_main_alias_2()  ???:0
16 0x00000000004022c5 _start()  ???:0
=================================

Program received signal SIGABRT: Process abort signal.

Backtrace for this error:
8 0x00000000000a20a6 MPI_Bcast()  ???:0
9 0x0000000000052082 pmpi_bcast_()  ???:0
10 0x000000000040476f bcast_inputs_()  ???:0
11 0x0000000000403c17 read_input_()  ???:0
12 0x0000000000402400 MAIN__()  ???:0
13 0x0000000000402934 main()  ???:0
14 0x0000000000023ace __libc_start_call_main()  ???:0
15 0x0000000000023b89 __libc_start_main_alias_2()  ???:0
16 0x00000000004022c5 _start()  ???:0
=================================

Program received signal SIGABRT: Process abort signal.

Backtrace for this error:
#0  0x7f865be52d5f in ???
#1  0x7f865bea1adc in __pthread_kill_implementation
#2  0x7f865be52cb5 in __GI_raise
#3  0x7f865be3c8b9 in abort
#4  0x7f865bac1706 in ucs_fatal_error_message
#5  0x7f865bac17f0 in ucs_fatal_error_format.localalias
#6  0x7f86599342c7 in uct_ud_ep_deferred_timeout_handler
#7  0x7f865bab640e in ucs_callbackq_slow_proxy
#8  0x7f865bbcbd09 in ucp_worker_progress
#9  0x7f865bd155bb in opal_progress
#10  0x7f865c488e84 in ompi_request_default_wait_all
#11  0x7f865c4ea78e in ompi_coll_base_bcast_intra_generic
#12  0x7f865c50b48b in ompi_coll_tuned_bcast_intra_dec_fixed
#13  0x7f865c4a20a5 in MPI_Bcast
#14  0x7f865c959081 in MPI_Bcast_f08
#15  0x40476e in bcast_inputs_
#16  0x403c16 in read_input_
#17  0x4023ff in MAIN__
#18  0x402933 in main
#0  0x7f4e74252d5f in ???
#1  0x7f4e742a1adc in __pthread_kill_implementation
#2  0x7f4e74252cb5 in __GI_raise
#3  0x7f4e7423c8b9 in abort
#4  0x7f4e73d30706 in ucs_fatal_error_message
#5  0x7f4e73d307f0 in ucs_fatal_error_format.localalias
#6  0x7f4e71b8f2c7 in uct_ud_ep_deferred_timeout_handler
#7  0x7f4e73d2540e in ucs_callbackq_slow_proxy
#8  0x7f4e73e3ad09 in ucp_worker_progress
#0  0x7f9333652d5f in ???
#1  0x7f93336a1adc in __pthread_kill_implementation
#9  0x7f4e73f845bb in opal_progress
#2  0x7f9333652cb5 in __GI_raise
#3  0x7f933363c8b9 in abort
#4  0x7f93332c1706 in ucs_fatal_error_message
#5  0x7f93332c17f0 in ucs_fatal_error_format.localalias
#6  0x7f93311242c7 in uct_ud_ep_deferred_timeout_handler
#7  0x7f93332b640e in ucs_callbackq_slow_proxy
#8  0x7f93333cbd09 in ucp_worker_progress
#9  0x7f93335155bb in opal_progress
#10  0x7f9333c88e84 in ompi_request_default_wait_all
#11  0x7f9333cea78e in ompi_coll_base_bcast_intra_generic
#12  0x7f9333d0b48b in ompi_coll_tuned_bcast_intra_dec_fixed
#13  0x7f9333ca20a5 in MPI_Bcast
#14  0x7f9334149081 in MPI_Bcast_f08
#15  0x40476e in bcast_inputs_
#16  0x403c16 in read_input_
#17  0x4023ff in MAIN__
#18  0x402933 in main
#10  0x7f4e74888e84 in ompi_request_default_wait_all
#11  0x7f4e748ea78e in ompi_coll_base_bcast_intra_generic
#12  0x7f4e7490b48b in ompi_coll_tuned_bcast_intra_dec_fixed
#13  0x7f4e748a20a5 in MPI_Bcast
#14  0x7f4e74bb4081 in MPI_Bcast_f08
#15  0x40476e in bcast_inputs_
#16  0x403c16 in read_input_
#17  0x4023ff in MAIN__
#18  0x402933 in main
[node3:3943 :0:3943]       ud_ep.c:278  Fatal: UD endpoint 0x1875ad0 to <no debug data>: unhandled timeout error
[node3:3945 :0:3945]       ud_ep.c:278  Fatal: UD endpoint 0x892ae0 to <no debug data>: unhandled timeout error
[node3:3946 :0:3946]       ud_ep.c:278  Fatal: UD endpoint 0x686ad0 to <no debug data>: unhandled timeout error
[node19:3744 :0:3744]       ud_ep.c:278  Fatal: UD endpoint 0x1f79af0 to <no debug data>: unhandled timeout error
[node3:3944 :0:3944]       ud_ep.c:278  Fatal: UD endpoint 0x1b35ac0 to <no debug data>: unhandled timeout error
[node3:3941 :0:3941]       ud_ep.c:278  Fatal: UD endpoint 0x1355490 to <no debug data>: unhandled timeout error
==== backtrace (tid:   3943) ====
0 0x00000000000317f1 ucs_fatal_error_format.localalias()  ???:0
1 0x00000000000852c8 uct_ud_ep_deferred_timeout_handler()  ???:0
2 0x000000000002640f ucs_callbackq_slow_proxy()  ???:0
3 0x000000000005ed0a ucp_worker_progress()  ???:0
4 0x00000000002321ef mca_pml_ucx_send()  ???:0
5 0x00000000000c5a6b PMPI_Send()  ???:0
6 0x000000000005abb6 MPI_Send_f08()  ???:0
7 0x000000000040be91 exchange_3_()  ???:0
8 0x00000000004074d4 erhs_()  ???:0
9 0x0000000000402475 MAIN__()  ???:0
10 0x0000000000402934 main()  ???:0
11 0x0000000000023ace __libc_start_call_main()  ???:0
12 0x0000000000023b89 __libc_start_main_alias_2()  ???:0
13 0x00000000004022c5 _start()  ???:0
=================================

Program received signal SIGABRT: Process abort signal.

Backtrace for this error:
==== backtrace (tid:   3945) ====
0 0x00000000000317f1 ucs_fatal_error_format.localalias()  ???:0
1 0x00000000000852c8 uct_ud_ep_deferred_timeout_handler()  ???:0
2 0x000000000002640f ucs_callbackq_slow_proxy()  ???:0
3 0x000000000005ed0a ucp_worker_progress()  ???:0
4 0x00000000002321ef mca_pml_ucx_send()  ???:0
5 0x00000000000c5a6b PMPI_Send()  ???:0
6 0x000000000005abb6 MPI_Send_f08()  ???:0
7 0x000000000040be91 exchange_3_()  ???:0
8 0x00000000004074d4 erhs_()  ???:0
9 0x0000000000402475 MAIN__()  ???:0
10 0x0000000000402934 main()  ???:0
11 0x0000000000023ace __libc_start_call_main()  ???:0
12 0x0000000000023b89 __libc_start_main_alias_2()  ???:0
13 0x00000000004022c5 _start()  ???:0
=================================

Program received signal SIGABRT: Process abort signal.

Backtrace for this error:
[node3:3942 :0:3942]       ud_ep.c:278  Fatal: UD endpoint 0x1a8ea80 to <no debug data>: unhandled timeout error
==== backtrace (tid:   3946) ====
0 0x00000000000317f1 ucs_fatal_error_format.localalias()  ???:0
1 0x00000000000852c8 uct_ud_ep_deferred_timeout_handler()  ???:0
2 0x000000000002640f ucs_callbackq_slow_proxy()  ???:0
3 0x000000000005ed0a ucp_worker_progress()  ???:0
4 0x00000000002321ef mca_pml_ucx_send()  ???:0
5 0x00000000000c5a6b PMPI_Send()  ???:0
6 0x000000000005abb6 MPI_Send_f08()  ???:0
7 0x000000000040be91 exchange_3_()  ???:0
8 0x00000000004074d4 erhs_()  ???:0
9 0x0000000000402475 MAIN__()  ???:0
10 0x0000000000402934 main()  ???:0
11 0x0000000000023ace __libc_start_call_main()  ???:0
12 0x0000000000023b89 __libc_start_main_alias_2()  ???:0
13 0x00000000004022c5 _start()  ???:0
=================================

Program received signal SIGABRT: Process abort signal.

Backtrace for this error:
==== backtrace (tid:   3744) ====
0 0x00000000000317f1 ucs_fatal_error_format.localalias()  ???:0
1 0x00000000000852c8 uct_ud_ep_deferred_timeout_handler()  ???:0
2 0x000000000002640f ucs_callbackq_slow_proxy()  ???:0
3 0x000000000005ed0a ucp_worker_progress()  ???:0
4 0x00000000002321ef mca_pml_ucx_send()  ???:0
5 0x00000000000c5a6b PMPI_Send()  ???:0
6 0x000000000005abb6 MPI_Send_f08()  ???:0
7 0x000000000040be91 exchange_3_()  ???:0
8 0x00000000004074d4 erhs_()  ???:0
9 0x0000000000402475 MAIN__()  ???:0
10 0x0000000000402934 main()  ???:0
11 0x0000000000023ace __libc_start_call_main()  ???:0
12 0x0000000000023b89 __libc_start_main_alias_2()  ???:0
13 0x00000000004022c5 _start()  ???:0
=================================

Program received signal SIGABRT: Process abort signal.

Backtrace for this error:
==== backtrace (tid:   3944) ====
0 0x00000000000317f1 ucs_fatal_error_format.localalias()  ???:0
1 0x00000000000852c8 uct_ud_ep_deferred_timeout_handler()  ???:0
2 0x000000000002640f ucs_callbackq_slow_proxy()  ???:0
3 0x000000000005ed0a ucp_worker_progress()  ???:0
4 0x00000000002321ef mca_pml_ucx_send()  ???:0
5 0x00000000000c5a6b PMPI_Send()  ???:0
6 0x000000000005abb6 MPI_Send_f08()  ???:0
7 0x000000000040be91 exchange_3_()  ???:0
8 0x00000000004074d4 erhs_()  ???:0
9 0x0000000000402475 MAIN__()  ???:0
10 0x0000000000402934 main()  ???:0
11 0x0000000000023ace __libc_start_call_main()  ???:0
12 0x0000000000023b89 __libc_start_main_alias_2()  ???:0
13 0x00000000004022c5 _start()  ???:0
=================================

Program received signal SIGABRT: Process abort signal.

Backtrace for this error:
==== backtrace (tid:   3941) ====
0 0x00000000000317f1 ucs_fatal_error_format.localalias()  ???:0
1 0x00000000000852c8 uct_ud_ep_deferred_timeout_handler()  ???:0
2 0x000000000002640f ucs_callbackq_slow_proxy()  ???:0
3 0x000000000005ed0a ucp_worker_progress()  ???:0
4 0x00000000002321ef mca_pml_ucx_send()  ???:0
5 0x00000000000c5a6b PMPI_Send()  ???:0
6 0x000000000005abb6 MPI_Send_f08()  ???:0
7 0x000000000040be91 exchange_3_()  ???:0
8 0x00000000004074d4 erhs_()  ???:0
9 0x0000000000402475 MAIN__()  ???:0
10 0x0000000000402934 main()  ???:0
11 0x0000000000023ace __libc_start_call_main()  ???:0
12 0x0000000000023b89 __libc_start_main_alias_2()  ???:0
13 0x00000000004022c5 _start()  ???:0
=================================

Program received signal SIGABRT: Process abort signal.

Backtrace for this error:
==== backtrace (tid:   3942) ====
0 0x00000000000317f1 ucs_fatal_error_format.localalias()  ???:0
1 0x00000000000852c8 uct_ud_ep_deferred_timeout_handler()  ???:0
2 0x000000000002640f ucs_callbackq_slow_proxy()  ???:0
3 0x000000000005ed0a ucp_worker_progress()  ???:0
4 0x00000000002321ef mca_pml_ucx_send()  ???:0
5 0x00000000000c5a6b PMPI_Send()  ???:0
6 0x000000000005abb6 MPI_Send_f08()  ???:0
7 0x000000000040be91 exchange_3_()  ???:0
8 0x00000000004074d4 erhs_()  ???:0
9 0x0000000000402475 MAIN__()  ???:0
10 0x0000000000402934 main()  ???:0
11 0x0000000000023ace __libc_start_call_main()  ???:0
12 0x0000000000023b89 __libc_start_main_alias_2()  ???:0
13 0x00000000004022c5 _start()  ???:0
=================================

Program received signal SIGABRT: Process abort signal.

Backtrace for this error:
#0  0x7f7e45452d5f in ???
#1  0x7f7e454a1adc in __pthread_kill_implementation
#2  0x7f7e45452cb5 in __GI_raise
#3  0x7f7e4543c8b9 in abort
#4  0x7f7e44f4e706 in ucs_fatal_error_message
#5  0x7f7e44f4e7f0 in ucs_fatal_error_format.localalias
#6  0x7f7e42da62c7 in uct_ud_ep_deferred_timeout_handler
#7  0x7f7e44f4340e in ucs_callbackq_slow_proxy
#8  0x7f7e4503ad09 in ucp_worker_progress
#9  0x7f7e45c321ee in mca_pml_ucx_send
#10  0x7f7e45ac5a6a in PMPI_Send
#11  0x7f7e45dd5bb5 in MPI_Send_f08
#12  0x40be90 in exchange_3_
#13  0x4074d3 in erhs_
#14  0x402474 in MAIN__
#15  0x402933 in main
#0  0x7f18c0e52d5f in ???
#1  0x7f18c0ea1adc in __pthread_kill_implementation
#2  0x7f18c0e52cb5 in __GI_raise
#3  0x7f18c0e3c8b9 in abort
#4  0x7f18c0a03706 in ucs_fatal_error_message
#5  0x7f18c0a037f0 in ucs_fatal_error_format.localalias
#6  0x7f18bffb52c7 in uct_ud_ep_deferred_timeout_handler
#7  0x7f18c09f840e in ucs_callbackq_slow_proxy
#8  0x7f18c0aa3d09 in ucp_worker_progress
#9  0x7f18c16321ee in mca_pml_ucx_send
#10  0x7f18c14c5a6a in PMPI_Send
#11  0x7f18c18c5bb5 in MPI_Send_f08
#12  0x40be90 in exchange_3_
#13  0x4074d3 in erhs_
#14  0x402474 in MAIN__
#15  0x402933 in main
#0  0x7f78cda52d5f in ???
#1  0x7f78cdaa1adc in __pthread_kill_implementation
#2  0x7f78cda52cb5 in __GI_raise
#3  0x7f78cda3c8b9 in abort
#4  0x7f78cdf09706 in ucs_fatal_error_message
#5  0x7f78cdf097f0 in ucs_fatal_error_format.localalias
#6  0x7f78cb4ed2c7 in uct_ud_ep_deferred_timeout_handler
#7  0x7f78cdefe40e in ucs_callbackq_slow_proxy
#8  0x7f78cd6a3d09 in ucp_worker_progress
#9  0x7f78ce2321ee in mca_pml_ucx_send
#10  0x7f78ce0c5a6a in PMPI_Send
#11  0x7f78ce51bbb5 in MPI_Send_f08
#12  0x40be90 in exchange_3_
#13  0x4074d3 in erhs_
#14  0x402474 in MAIN__
#15  0x402933 in main
#0  0x7f81e8e52d5f in ???
#1  0x7f81e8ea1adc in __pthread_kill_implementation
#2  0x7f81e8e52cb5 in __GI_raise
#3  0x7f81e8e3c8b9 in abort
#4  0x7f81e8ac1706 in ucs_fatal_error_message
#5  0x7f81e8ac17f0 in ucs_fatal_error_format.localalias
#6  0x7f81e692a2c7 in uct_ud_ep_deferred_timeout_handler
#7  0x7f81e8ab640e in ucs_callbackq_slow_proxy
#8  0x7f81e8bcbd09 in ucp_worker_progress
#9  0x7f81e96321ee in mca_pml_ucx_send
#10  0x7f81e94c5a6a in PMPI_Send
#11  0x7f81e9957bb5 in MPI_Send_f08
#12  0x40be90 in exchange_3_
#13  0x4074d3 in erhs_
#14  0x402474 in MAIN__
#15  0x402933 in main
#0  0x7f44c3052d5f in ???
#1  0x7f44c30a1adc in __pthread_kill_implementation
#2  0x7f44c3052cb5 in __GI_raise
#3  0x7f44c303c8b9 in abort
#4  0x7f44c2c03706 in ucs_fatal_error_message
#5  0x7f44c2c037f0 in ucs_fatal_error_format.localalias
#6  0x7f44c0aa82c7 in uct_ud_ep_deferred_timeout_handler
#7  0x7f44c2bf840e in ucs_callbackq_slow_proxy
#8  0x7f44c2ca3d09 in ucp_worker_progress
#9  0x7f44c38321ee in mca_pml_ucx_send
#10  0x7f44c36c5a6a in PMPI_Send
#11  0x7f44c3ad6bb5 in MPI_Send_f08
#12  0x40be90 in exchange_3_
#13  0x4074d3 in erhs_
#14  0x402474 in MAIN__
#15  0x402933 in main
#0  0x7f1600c52d5f in ???
#1  0x7f1600ca1adc in __pthread_kill_implementation
#2  0x7f1600c52cb5 in __GI_raise
#3  0x7f1600c3c8b9 in abort
#4  0x7f1600803706 in ucs_fatal_error_message
#5  0x7f16008037f0 in ucs_fatal_error_format.localalias
#6  0x7f15ffda82c7 in uct_ud_ep_deferred_timeout_handler
#7  0x7f16007f840e in ucs_callbackq_slow_proxy
#8  0x7f16008a3d09 in ucp_worker_progress
#9  0x7f16014321ee in mca_pml_ucx_send
#10  0x7f16012c5a6a in PMPI_Send
#11  0x7f16016b8bb5 in MPI_Send_f08
#12  0x40be90 in exchange_3_
#13  0x4074d3 in erhs_
#14  0x402474 in MAIN__
#15  0x402933 in main
#0  0x7ffad2452d5f in ???
#1  0x7ffad24a1adc in __pthread_kill_implementation
#2  0x7ffad2452cb5 in __GI_raise
#3  0x7ffad243c8b9 in abort
#4  0x7ffad2003706 in ucs_fatal_error_message
#5  0x7ffad20037f0 in ucs_fatal_error_format.localalias
#6  0x7ffad15ab2c7 in uct_ud_ep_deferred_timeout_handler
#7  0x7ffad1ff840e in ucs_callbackq_slow_proxy
#8  0x7ffad20a3d09 in ucp_worker_progress
#9  0x7ffad2c321ee in mca_pml_ucx_send
#10  0x7ffad2ac5a6a in PMPI_Send
#11  0x7ffad2ebbbb5 in MPI_Send_f08
#12  0x40be90 in exchange_3_
#13  0x4074d3 in erhs_
#14  0x402474 in MAIN__
#15  0x402933 in main
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun noticed that process rank 376 with PID 3957 on node node3 exited on signal 6 (Aborted).
--------------------------------------------------------------------------
2 total processes killed (some possibly by mpirun during cleanup)

[yosefe]

Hi @adfaure , can you pls try adding -x UCX_UD_TIMEOUT=120s to mpirun command (to export that env var)?

[adfaure]

Hi !

Thanks for the reply. With -x UCX_UD_TIMEOUT=120s, the error still occurs after two minutes.

[yosefe]

Hi, @adfaure, according to the description "If I remove them from the hostfile it works" this could be a hardware issue. I'd suggest to take it with the fabric vendor to see if there is any way to check for packet drops on the "bad" nodes.

[adfaure]

Hi, I doubt that it is a hardware issue because:

• It works after some time.
• It works with the default cluster environment (I am in a cluster where I can install my own images, but with the default environment it works).
• If I remove the nodes, it works; but what is funny is that if I only keep the failing nodes it works too 😅

[yosefe]

what is the difference between your image and the default environment?

[adfaure]

Unfortunately, the environments are really different.

The base environment is based on debian, and the one that I am using (with the bug) is a nixos image.
On debian the openmpi version used openmpi 4.1.0, on nixos I am using openmpi 4.1.5.

[adfaure]

Hello, I don't think the issue was related to UCX.
I fact I had two network interfaces connected and configured on the same network, which causes MPI to fail on some timeouts.
Now that I identified the issue, shutting down one interface fixes the issue.

Thank for your time :)