Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-49703] Publish Java 21 Docker image for preview1 #69

Closed
wants to merge 3 commits into from
Closed

[SPARK-49703] Publish Java 21 Docker image for preview1 #69

wants to merge 3 commits into from

Conversation

dongjoon-hyun
Copy link
Member

@dongjoon-hyun dongjoon-hyun commented Sep 18, 2024 •
edited
Loading

What changes were proposed in this pull request?

This PR aims to publish Java 21 Docker image for preview1 and will be extended for preview2.

Why are the changes needed?

Apache Spark supports Java 21 via SPARK-43831.

Does this PR introduce any user-facing change?

No, this is a new image.

How was this patch tested?

Pass the CIs.

HyukjinKwon
HyukjinKwon approved these changes Sep 19, 2024
View reviewed changes
Copy link
Member

@HyukjinKwon HyukjinKwon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cc @Yikun

@dongjoon-hyun
Copy link
Member Author

Thank you, @HyukjinKwon .

viirya
viirya reviewed Sep 19, 2024
View reviewed changes
README.md Outdated Show resolved Hide resolved
@dongjoon-hyun
Copy link
Member Author

For the record, while preparing 4.0.0-preview2 release, I'm making this repo up-to-date by re-implementing the Apache Spark patches like the following. The following three are published to https://hub.docker.com/r/apache/spark/ .

@dongjoon-hyun @viirya
Update README.md
494f5b0 
Co-authored-by: Liang-Chi Hsieh <[email protected]>
@dongjoon-hyun
Copy link
Member Author

Thank you, @viirya .

Yikun
Yikun reviewed Sep 19, 2024
View reviewed changes
versions.json Outdated Show resolved Hide resolved
versions.json Outdated Show resolved Hide resolved
@dongjoon-hyun
Copy link
Member Author

Thank you, @Yikun .

@dongjoon-hyun
Copy link
Member Author

Thank you, @HyukjinKwon , @viirya , @Yikun .
Merged to master.

This was referenced Sep 19, 2024
Closed
Closed
dongjoon-hyun added a commit to apache/spark-kubernetes-operator that referenced this pull request Sep 19, 2024
@dongjoon-hyun
[SPARK-49714] Add Java 21-based SparkPi example
815cefe 
### What changes were proposed in this pull request?

This PR aims to add `Java 21`-based SparkPi example.

### Why are the changes needed?

Apache Spark starts to publish Java 21-based image from Today (2024-09-19). This new example will illustrate how to use it.
- apache/spark-docker#69

### Does this PR introduce _any_ user-facing change?

No, this is a new example.

### How was this patch tested?

Manual review.

```
$ kubectl apply -f examples/pi-java21.yaml
```

```
$ kubectl get sparkapp
NAME        CURRENT STATE      AGE
pi-java21   ResourceReleased   28s
```

```
$ kubectl describe sparkapp pi-java21
Name:         pi-java21
Namespace:    default
Labels:       <none>
Annotations:  <none>
API Version:  spark.apache.org/v1alpha1
Kind:         SparkApplication
Metadata:
  Creation Timestamp:  2024-09-19T04:08:16Z
  Finalizers:
    sparkapplications.spark.apache.org/finalizer
  Generation:        2
  Resource Version:  95294
  UID:               2bc46e8d-6339-4867-9a28-6552c6c471e5
Spec:
  Application Tolerations:
    Application Timeout Config:
      Driver Ready Timeout Millis:            300000
      Driver Start Timeout Millis:            300000
      Executor Start Timeout Millis:          300000
      Force Termination Grace Period Millis:  300000
      Termination Requeue Period Millis:      2000
    Instance Config:
      Init Executors:        0
      Max Executors:         0
      Min Executors:         0
    Resource Retain Policy:  OnFailure
    Restart Config:
      Max Restart Attempts:    3
      Restart Backoff Millis:  30000
      Restart Policy:          Never
  Deployment Mode:             ClusterMode
  Driver Args:
  Jars:        local:///opt/spark/examples/jars/spark-examples.jar
  Main Class:  org.apache.spark.examples.SparkPi
  Runtime Versions:
    Scala Version:  2.13
    Spark Version:  4.0.0-preview1
  Spark Conf:
    spark.dynamicAllocation.enabled:                          true
    spark.dynamicAllocation.maxExecutors:                     3
    spark.dynamicAllocation.shuffleTracking.enabled:          true
    spark.kubernetes.authenticate.driver.serviceAccountName:  spark
    spark.kubernetes.container.image:                         apache/spark:4.0.0-preview1-java21-scala
    spark.log.structuredLogging.enabled:                      false
Status:
  Current Attempt Summary:
    Attempt Info:
      Id:  0
  Current State:
    Current State Summary:  ResourceReleased
    Last Transition Time:   2024-09-19T04:08:33.316041381Z
  State Transition History:
    0:
      Current State Summary:  Submitted
      Last Transition Time:   2024-09-19T04:08:16.584629470Z
      Message:                Spark application has been created on Kubernetes Cluster.
    1:
      Current State Summary:  DriverRequested
      Last Transition Time:   2024-09-19T04:08:17.269457304Z
      Message:                Requested driver from resource scheduler.
    2:
      Current State Summary:  DriverStarted
      Last Transition Time:   2024-09-19T04:08:17.809898304Z
      Message:                Driver has started running.
    3:
      Current State Summary:  DriverReady
      Last Transition Time:   2024-09-19T04:08:17.810393971Z
      Message:                Driver has reached ready state.
    4:
      Current State Summary:  RunningHealthy
      Last Transition Time:   2024-09-19T04:08:17.828526471Z
      Message:                Application is running healthy.
    5:
      Current State Summary:  Succeeded
      Last Transition Time:   2024-09-19T04:08:33.241514089Z
      Message:                Spark application completed successfully.
    6:
      Current State Summary:  ResourceReleased
      Last Transition Time:   2024-09-19T04:08:33.316041381Z
Events:                       <none>
```

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes #129 from dongjoon-hyun/SPARK-49714.

Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
dongjoon-hyun added a commit to apache/spark-kubernetes-operator that referenced this pull request Sep 19, 2024
@dongjoon-hyun
[SPARK-49715] Add Java 21-based SparkCluster example
b187764 
### What changes were proposed in this pull request?

This PR aims to add `Java 21`-based `SparkCluster` example.

### Why are the changes needed?

Apache Spark starts to publish Java 21-based image from Today (2024-09-19). This new example will illustrate how to use it.
- apache/spark-docker#69

### Does this PR introduce _any_ user-facing change?

No, this is a new example.

### How was this patch tested?

Manual review.

```
$ kubectl apply -f cluster-java21.yaml
```

```
$ kubectl get sparkcluster
NAME             CURRENT STATE    AGE
cluster-java21   RunningHealthy   9s
```

```
$ kubectl describe sparkcluster cluster-java21
Name:         cluster-java21
Namespace:    default
Labels:       <none>
Annotations:  <none>
API Version:  spark.apache.org/v1alpha1
Kind:         SparkCluster
Metadata:
  Creation Timestamp:  2024-09-19T04:25:30Z
  Finalizers:
    sparkclusters.spark.apache.org/finalizer
  Generation:        2
  Resource Version:  96663
  UID:               9421c957-380d-4a26-a266-379304bf83ee
Spec:
  Cluster Tolerations:
    Instance Config:
      Init Workers:  3
      Max Workers:   3
      Min Workers:   3
  Master Spec:
  Runtime Versions:
    Spark Version:  4.0.0-preview1
  Spark Conf:
    spark.kubernetes.container.image:  apache/spark:4.0.0-preview1-java21
    spark.master.rest.enabled:         true
    spark.master.rest.host:            0.0.0.0
    spark.master.ui.title:             Prod Spark Cluster (Java 21)
    spark.ui.reverseProxy:             true
  Worker Spec:
Status:
  Current Attempt Summary:
    Attempt Info:
      Id:  0
  Current State:
    Current State Summary:  RunningHealthy
    Last Transition Time:   2024-09-19T04:25:30.665095088Z
    Message:                Cluster has reached ready state.
  State Transition History:
    0:
      Current State Summary:  Submitted
      Last Transition Time:   2024-09-19T04:25:30.640072963Z
      Message:                Spark cluster has been submitted to Kubernetes Cluster.
    1:
      Current State Summary:  RunningHealthy
      Last Transition Time:   2024-09-19T04:25:30.665095088Z
      Message:                Cluster has reached ready state.
Events:                       <none>
```

```
$ k get pod
NAME                                         READY   STATUS    RESTARTS   AGE
cluster-java21-master-0                      1/1     Running   0          3m20s
cluster-java21-worker-0                      1/1     Running   0          3m20s
cluster-java21-worker-1                      1/1     Running   0          3m20s
cluster-java21-worker-2                      1/1     Running   0          3m20s
spark-kubernetes-operator-778b9bbdc6-fqks9   1/1     Running   0          20m
```

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes #130 from dongjoon-hyun/SPARK-49715.

Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
@dongjoon-hyun
Copy link
Member Author

For the record, Java 21 images are published successfully.

@dongjoon-hyun dongjoon-hyun deleted the SPARK-49703 branch September 19, 2024 05:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Yikun Yikun Yikun left review comments

@viirya viirya viirya approved these changes

@HyukjinKwon HyukjinKwon HyukjinKwon approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@dongjoon-hyun @viirya @Yikun @HyukjinKwon