Upgrade to Spark 3.4.x #2187
Comments
codefromthecrypt
pushed a commit
to openzipkin/zipkin-dependencies
that referenced
this issue
Dec 30, 2023
This updates all dependencies we can, except we can't update to Scala 2.13 due to a conflict between the Elasticsearch and Cassandra connectors. Notably, Cassandra requires Spark 3.4 on Scala 2.13, but Elastic requires Spark 3.3. See elastic/elasticsearch-hadoop#2187 Signed-off-by: Adrian Cole <[email protected]>
codefromthecrypt
pushed a commit
to openzipkin/zipkin-dependencies
that referenced
this issue
Dec 30, 2023
This drops support of Elasticsearch 6.x, including writing to old index templates, as it is no longer possible with the driver that supports 7.x and 8.x. See elastic/elasticsearch-hadoop#2188 This also updates to the latest Spark 3.x version possible, currently 3.3.x. Note: we can't update to Scala 2.13 due to a conflict between connectors: (DataStax) Cassandra requires Spark 3.4 on Scala 2.13, but Elastic requires Spark 3.3. See elastic/elasticsearch-hadoop#2187 Signed-off-by: Adrian Cole <[email protected]>
codefromthecrypt
added a commit
to openzipkin/zipkin-dependencies
that referenced
this issue
Dec 30, 2023
This drops support of Elasticsearch 6.x, including writing to old index templates, as it is no longer possible with the driver that supports 7.x and 8.x. See elastic/elasticsearch-hadoop#2188 This also updates to the latest Spark 3.x version possible, currently 3.3.x. Note: we can't update to Scala 2.13 due to a conflict between connectors: (DataStax) Cassandra requires Spark 3.4 on Scala 2.13, but Elastic requires Spark 3.3. See elastic/elasticsearch-hadoop#2187 Signed-off-by: Adrian Cole <[email protected]>
|
Spark 3.5 is out, so updating to this also works |
|
I tested this, and it works fine. Thanks @masseyke! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
What kind an issue is this?
The easier it is to track down the bug, the faster it is solved.
Often a solution already exists! Don’t send pull requests to implement new features without
first getting our support. Sometimes we leave features out on purpose to keep the project small.
Issue description
Please update the Spark version to 3.4.x
Steps to reproduce
N/A
Version Info
N/A
Feature description
The current codebase is on Spark 3.3.x and causes conflicts moving to Scala 2.13 where other projects are at Spark 3.4.x. Please update the dependency or keep this open with a note about timeframe or external event which would result in the same.
The text was updated successfully, but these errors were encountered: