Skip to content

Commit

Permalink
[SPARK-45737][SQL] Remove unnecessary .toArray[InternalRow] in `Spa…
Browse files Browse the repository at this point in the history
…rkPlan#executeTake`

### What changes were proposed in this pull request?
https://github.com/apache/spark/blob/8dd3ec87e26969df6fe08f5fddc3f8d6efc2420d/sql/core/src/main/scala/org/apache/spark/sql/execution/SparkPlan.scala#L535-L559

In the above code, the input parameters of `mutable.Buffer#prependAll` and `mutable.Growable#++=` functions are `IterableOnce`

- `mutable.Buffer#prependAll`

```scala
  def prependAll(elems: IterableOnce[A]): this.type = { insertAll(0, elems); this }
```

- `mutable.Growable#++=`

```
  `inline` final def ++= (xs: IterableOnce[A]): this.type = addAll(xs)
```

and the type of `rows` is `Iterator[InternalRow]`, which inherits from `IterableOnce`

```
val rows = decodeUnsafeRows(res(i)._2)
private def decodeUnsafeRows(bytes: ChunkedByteBuffer): Iterator[InternalRow]
```

So there is no need to cast to an `Array` of `InternalRow` anymore.

### Why are the changes needed?
Remove unnecessary `.toArray[InternalRow]`

### Does this PR introduce _any_ user-facing change?
No

### How was this patch tested?
Pass GitHub Actions

### Was this patch authored or co-authored using generative AI tooling?
No

Closes #43599 from LuciferYang/sparkplan.

Authored-by: yangjie01 <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
  • Loading branch information
LuciferYang authored and dongjoon-hyun committed Oct 31, 2023
1 parent 9169d93 commit 936c1cd
Showing 1 changed file with 4 additions and 4 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -536,23 +536,23 @@ abstract class SparkPlan extends QueryPlan[SparkPlan] with Logging with Serializ
while (buf.length < n && i < res.length) {
val rows = decodeUnsafeRows(res(i)._2)
if (n - buf.length >= res(i)._1) {
buf.prependAll(rows.toArray[InternalRow])
buf.prependAll(rows)
} else {
val dropUntil = res(i)._1 - (n - buf.length)
// Same as Iterator.drop but this only takes a long.
var j: Long = 0L
while (j < dropUntil) { rows.next(); j += 1L}
buf.prependAll(rows.toArray[InternalRow])
buf.prependAll(rows)
}
i += 1
}
} else {
while (buf.length < n && i < res.length) {
val rows = decodeUnsafeRows(res(i)._2)
if (n - buf.length >= res(i)._1) {
buf ++= rows.toArray[InternalRow]
buf ++= rows
} else {
buf ++= rows.take(n - buf.length).toArray[InternalRow]
buf ++= rows.take(n - buf.length)
}
i += 1
}
Expand Down

0 comments on commit 936c1cd

Please sign in to comment.