ENH: Fix cuda storage transfer deadlock on multiple GPUs

[luweizheng]

Fix storage transfer deadlock of CUDA storage on multiple GPUs. I do this work based on #488

The current implementation of the transfer function leads to a deadlock when executing Xorbits on multiple GPUs. The issue arises from the StorageHandlerActor.fetch_batch function, which invokes SenderManagerActor.send_batch_data and subsequently calls StorageHandlerActor.request_quota_with_spill. Due to the locking mechanism within the StorageHandlerActor method call, a deadlock arises.

NOTEs:

• fix dead lock on GPU
• fix GPU buffer size 0 issue
• test on TPC-H SF10 and it works. Much faster than Dask-cuDF.
• known issue: test_transfer_gpu.py of ucx channel has bugs while socket channel works.
• known issue: too many actor calls in this implement. May lead to performance down.

Check code requirements

• tests added / passed (if needed)
• Ensure all linting tests pass

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 78.53%. Comparing base (9984250) to head (44fa88a).

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #788      +/-   ##
==========================================
- Coverage   82.97%   78.53%   -4.44%     
==========================================
  Files        1070     1070              
  Lines       80004    80016      +12     
  Branches    16537    16541       +4     
==========================================
- Hits        66380    62840    -3540     
- Misses      12031    15425    +3394     
- Partials     1593     1751     +158     

Flag	Coverage Δ
unittests	78.42% <ø> (-4.46%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.