Skip to content

Commit

Permalink
[MINOR][CORE] Validate spark.deploy.defaultCores value during config …
Browse files Browse the repository at this point in the history
…setting

### What changes were proposed in this pull request?

This aims to move `spark.deploy.defaultCores` validation logic to config setting.

### Why are the changes needed?

In order to ensure the value range by early checking.

### Does this PR introduce _any_ user-facing change?

No. `Spark Master` will fail to start in both cases, *before* and *after*.

### How was this patch tested?

Manual review.

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes #43655 from dongjoon-hyun/spark.deploy.defaultCores.

Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
  • Loading branch information
dongjoon-hyun committed Nov 4, 2023
1 parent 4f56e38 commit 3363c2a
Show file tree
Hide file tree
Showing 2 changed files with 2 additions and 4 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ import java.util.concurrent.{ScheduledFuture, TimeUnit}
import scala.collection.mutable.{ArrayBuffer, HashMap, HashSet}
import scala.util.Random

import org.apache.spark.{SecurityManager, SparkConf, SparkException}
import org.apache.spark.{SecurityManager, SparkConf}
import org.apache.spark.deploy.{ApplicationDescription, DriverDescription, ExecutorState}
import org.apache.spark.deploy.DeployMessages._
import org.apache.spark.deploy.master.DriverState.DriverState
Expand Down Expand Up @@ -119,9 +119,6 @@ private[deploy] class Master(
// Default maxCores for applications that don't specify it (i.e. pass Int.MaxValue)
private val defaultCores = conf.get(DEFAULT_CORES)
val reverseProxy = conf.get(UI_REVERSE_PROXY)
if (defaultCores < 1) {
throw new SparkException(s"${DEFAULT_CORES.key} must be positive")
}
val historyServerUrl = conf.get(MASTER_UI_HISTORY_SERVER_URL)

// Alternative application submission gateway that is stable across Spark versions
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,7 @@ private[spark] object Deploy {
val DEFAULT_CORES = ConfigBuilder("spark.deploy.defaultCores")
.version("0.9.0")
.intConf
.checkValue(_ > 0, "spark.deploy.defaultCores must be positive.")
.createWithDefault(Int.MaxValue)

val MAX_DRIVERS = ConfigBuilder("spark.deploy.maxDrivers")
Expand Down

0 comments on commit 3363c2a

Please sign in to comment.