[jvm-packages] Support Ranker

[wbo4958]

No description provided.

[wbo4958]

Hi @trivialfis, @eordentlich Could you help review it? Thx

[eordentlich]

Would be good to explain how this resolves the issue raised in this comment and issue linked therein:
#10639 (comment)

[eordentlich]

Some questions re plugin and preprocess.

[eordentlich]

How does this operation interact with spark-rapids plugin if enabled? Any implications on GPU memory?

[eordentlich]

Does this preprocess even get called if plugin is enabled? If not, partition might not be sorted.

[wbo4958]

My bad. Fixed this issue. Please help review it again. Thx very much.

[wbo4958]

Hi @eordentlich @trivialfis, Could you help take a look at it. Thx very much.

[eordentlich]

This resolves the issue with the plugin and sorted partitions (and nice to see the test for this case too), but still wondering how that partition sort is computed by the spark-rapids plugin when enabled. Is done on the GPU?

Also, does this PR resolve the issue I reference in an earlier comment?

@trivialfis should take a look as well.

[wbo4958]

HI @eordentlich,

I just tried the below case which has the same pattern with XGBoost

df.repartition(2).sortWithinPartitions("class").collect()

and got below Physical plans

== Physical Plan ==
AdaptiveSparkPlan (10)
+- == Final Plan ==
   GpuColumnarToRow (6)
   +- GpuSort (5)
      +- GpuShuffleCoalesce (4)
         +- ShuffleQueryStage (3), Statistics(sizeInBytes=6.4 KiB, rowCount=150)
            +- GpuColumnarExchange (2)
               +- GpuScan parquet  (1)
+- == Initial Plan ==
   Sort (9)
   +- Exchange (8)
      +- Scan parquet  (7)

XGBoost leverages ColumnarRdd to extract the CUDF table. ColumnarRdd is going to do some case match to filter out the RDDs that involves row-wised operations. So after converting,

we we get below corresponding RDDs which are coming from below GPU plans. So you can see the final cudf table was coming from GpuSort which will run on GPUs

   +- GpuSort (5)
      +- GpuShuffleCoalesce (4)
         +- ShuffleQueryStage (3), Statistics(sizeInBytes=6.4 KiB, rowCount=150)
            +- GpuColumnarExchange (2)
               +- GpuScan parquet  (1)

[eordentlich]

👍

[trivialfis]

Will look into this tomorrow.

[trivialfis]

Also, does this PR resolve the issue I reference in an earlier comment?

Thank you for raising that. I will be looking into it along with other LTR issues/feature requests after sorting out some of the work on external memory. I still think within-partition sort is sufficient for most of the use cases. The worst case is adding these qid-based partitioning, which might be as expensive as a global sort.

[trivialfis]

Looks good to me, assuming all tests can pass.