Skip to content

Commit

Permalink
[DOCS][MINOR] Remove spark-sql-api dependency from connect docs
Browse files Browse the repository at this point in the history
### What changes were proposed in this pull request?
Remove `spark-sql-api` dependency from documentation.

### Why are the changes needed?
Dependency `spark-connect-client-jvm` is sufficient, as it includes (shaded) the `spark-sql-api` package.

In fact, adding the `spark-sql-api` dependency breaks runtime:
1. transient dependency `io.netty:netty-buffer`, that is also included in `spark-connect-client-jvm` (shaded) cannot be found: `NoClassDefFoundError: io/netty/buffer/PooledByteBufAllocator`
2. method `ArrowUtils$.toArrowSchema` provided by `spark-sql-api` cannot be found: `NoSuchMethodError: 'org.sparkproject.org.apache.arrow.vector.types.pojo.Schema org.apache.spark.sql.util.ArrowUtils$.toArrowSchema(org.apache.spark.sql.types.StructType, java.lang.String, boolean, boolean)'`

### Does this PR introduce _any_ user-facing change?
No.

### How was this patch tested?
Manually with minimal maven project.

### Was this patch authored or co-authored using generative AI tooling?
No.

Closes #48119 from EnricoMi/docs-connect-jvm-deps.

Authored-by: Enrico Minack <[email protected]>
Signed-off-by: Herman van Hovell <[email protected]>
  • Loading branch information
EnricoMi authored and hvanhovell committed Sep 16, 2024
1 parent 2113f10 commit 8d78f5b
Showing 1 changed file with 0 additions and 1 deletion.
1 change: 0 additions & 1 deletion docs/spark-connect-overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -335,7 +335,6 @@ Lines with a: 72, lines with b: 39
To use Spark Connect as part of a Scala application/project, we first need to include the right dependencies.
Using the `sbt` build system as an example, we add the following dependencies to the `build.sbt` file:
{% highlight sbt %}
libraryDependencies += "org.apache.spark" %% "spark-sql-api" % "3.5.0"
libraryDependencies += "org.apache.spark" %% "spark-connect-client-jvm" % "3.5.0"
{% endhighlight %}

Expand Down

0 comments on commit 8d78f5b

Please sign in to comment.