Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Deploy all submodules for default sparkver in nightly [skip ci] #11516

Merged
merged 1 commit into from
Sep 27, 2024

Conversation

pxLi
Copy link
Collaborator

@pxLi pxLi commented Sep 27, 2024

We met issue started from 24.12, this is due to we stop deploying some nightly modules after #11301

e.g. spark-shell run with --packages

/home/jenkins/agent/workspace/jenkins-rapids_integration-dev-github-1093-3.2.0/jars/spark-3.2.0-bin-hadoop3.2/bin/spark-shell 
--master 'local-cluster[1,2,1024]' 
--conf spark.plugins=com.nvidia.spark.SQLPlugin 
--conf spark.deploy.maxExecutorRetries=0 
--packages com.nvidia:rapids-4-spark_2.12:24.12.0-SNAPSHOT 
--repositories https://urm.nvidia.com/artifactory/sw-spark-maven

which would fail to find jdk-profiles

Update the nightly script to deploy all submodules for default sparkver. Keep other spark shims build to deploy integration_tests module only

@pxLi pxLi added bug Something isn't working build Related to CI / CD or cleanly building labels Sep 27, 2024
@pxLi pxLi changed the title Deploy all submodules for default sparkver Deploy all submodules for default sparkver [skip ci] Sep 27, 2024
@pxLi pxLi requested a review from yinqingh September 27, 2024 01:34
@pxLi
Copy link
Collaborator Author

pxLi commented Sep 27, 2024

build

@pxLi pxLi changed the title Deploy all submodules for default sparkver [skip ci] Deploy all submodules for default sparkver in nightly [skip ci] Sep 27, 2024
@pxLi pxLi merged commit 9625e58 into NVIDIA:branch-24.10 Sep 27, 2024
45 of 46 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working build Related to CI / CD or cleanly building
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants