Flushing only one endpoint without polling all QPs

[sobc]

Dear community,

TL;DR

Is there a way to flush an endpoint in a blocking manner without progressing the ucp worker?

In-depth

I'm currently struggling with an implementation of a middleware utilizing UCX and its RMA capabilities.

In my current setup, I'm utilizing one-sided RC write/reads for ucx_get_nbx and ucx_put_nbx in a blocking manner, as outlined in the documentation. This involves checking the request status after running ucp_worker_progress in a loop.

While ucx_ep_flush_nbx seems like a potential solution, it returns a request handle which necessitates further progression of the worker. But I want to avoid a complete progression of the worker, because: As far as I understand ucp_worker_progress polls all active QPs on messages, which leads to a massive overhead when multiple endpoints are connected and thus resulting in a bad scaling of my library. In the library one can assume, if configured right, there will only be one-sided communication and there shall be no other messages than the current request in the QP. I can confirm that my library uses RC write/read messages to complete put/get operations.

Why am I sure that ucp_worker_progress is the problem?

I have observed that as UCX is configured to allow a maximum of 256 endpoints for using RC verbs, beyond which UD (two-sided) is employed for RMA operations, my library's scaling performance suffers until it reaches this threshold. Subsequently, the transition to UD transport significantly improves scaling, as depicted in the provided figure (tpn = "tasks per node").

While I am confident in my understanding that ucp_worker_progress is contributing to the scalability issues, I remain open to alternative explanations. However, based on my observations, this appears to be the most plausible explanation, as I find it unlikely that two-sided UD communication would outperform one-sided RC transport.

Thank you for your help!
Max

[yosefe]

The progress should not affect scalability as we poll on CQs (one per UCT iface) and not on QPs. The performance change above can we caused by switching from RC to UD. Which HCA/NIC is being used? Can you try forcing RC transport by running with UCX_TLS=self,sm,rc also on large scale?

[sobc]

Thank you for your quick reply.

First of all, I reconsidered the idea that having too many QPs per process would slow down RC one-sided transport and concluded that this is nonsense. For example, besides the fact that ucp_worker_progress will only poll the CQ and not the QPs, using 21 processes with 12 processes per node will actually create more endpoints per process (20*12 = 240) than 10 processes with 24 tasks per node (9 * 24 = 216), but has a much higher throughput, at around 256 total processes.

The problem remains: Why does RC one-sided communication scale worse than UD two-sided?

So as you suggested, I rerun the benchmark now with only RC transport. Only when using 12 tasks per node there is linear scaling. However, there is a drop from 2 to 3 nodes with 12 tpn and the overall achieved MOPS are still not comparable to UD transport.

The nodes are equipped with InfiniBand FDR HCAs (MCX353A-FCBT) and are connected using one InfiniBand FDR interconnect (Mellanox SX6025). I'm using UCX 1.15.

[sobc]

After conducting several investigations to identify the bad scaling, we've determined that the issue does not lie in ucx, but rather with our hardware.

Sorry for the inconvenience and thanks for the reply.