[CUDA] add multi-node multi-GPU support for new CUDA version and improve efficiency

[Xuweijia-buaa]

Summary

Current multi-GPU version just support one node scenario, we can add support for multi-node Multi-GPU scenario for new CUDA version, to deal with big data that can not fit in one node.

And we can also improve efficiency for such scenario. In the current Multi-GPU version, each device loads a complete histogram and performs follow-up work based on it. When dealing with datasets that have too many features, the full histogram may be too large to fit in one device. Even the histogram can fit in, each device will do repetitive work afterwards.

We can use a mixed data and feature parallel strategy in such scenario, like in the distributed data parallel scenario, alleviate loading and computing pressure on each device.

[jameslamb]

Added this to #2302 with other feature requests. Thanks for documenting it!

Per this repo's policy described there, I'm closing this discussion for now. Anyone wishing to implement this, leave a comment saying that you're working on it and we can re-open this discussion.