-
Notifications
You must be signed in to change notification settings - Fork 424
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
What is UCX's policy of choosing transport? #5320
Labels
Comments
@rzambre pls see https://openucx.readthedocs.io/en/master/faq.html#selecting-networks-and-transports |
Thanks! Wasn't aware of the new documentation. That is helpful. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the bug
In the past (with UCX 1.5.0), I used to set
UCX_NET_DEVICES=mlx5_0:1
andUCX_TLS=rc_mlx5,rc
and hope thatrc_mlx5
would be used during the fast-path operations. If I setUCX_TLS=rc_mlx5
only, I would get an error duringucp_init
.With the latest UCX
master
, I see withucx_info -d
that there arerc_verbs
andrc_mlx5
transports. But when I setUCX_TLS=rc_mlx5,rc_verbs
, I get an error during initialization. But after playing around, I discovered that settingUCX_TLS=rc_mlx5,rc
(as I had done earlier) works even thoughrc
is not listed inucx_info -d
.(1) What is the difference between setting
UCX_TLS=rc_mlx5,rc_verbs
andUCX_TLS=rc_mlx5,rc
?What works with using the transports listed in
ucx_info -d
isUCX_TLS=rc_mlx5,ud_[mlx5|verbs]
.(2) More generally, is there an overview of how UCX chooses which transport to use for its critical-path operations such as
ucp_tag_send_nb
?Steps to Reproduce
mpiexec -n 2 -ppn 1 -hosts <node1>,<node2> -env UCX_NET_DEVICES mlx5_0:1 -env UCX_TLS=rc_mlx5,rc_verbs ./osu_mbw_mr
master
@eaad8e2
+ UCX configure flags:--disable-logging --disable-debug --disable-assertions --disable-params-check --enable-mt
d1e673a
Setup and versions
CentOS Linux release 7.7.1908 (Core)
Linux hpc3-14-12 3.10.0-1062.12.1.el7.x86_64 Add basic types and functions, initial makefile, and smoke test. #1 SMP Tue Feb 4 23:02:59 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
The text was updated successfully, but these errors were encountered: