Skip to content

Commit

Permalink
[SPARK-50395][SQL] Fix malformed URI syntax in Windows
Browse files Browse the repository at this point in the history
### What changes were proposed in this pull request?

This PR fixes an issue that the Artifact Manager is using a malformed URI string for Class Dir in windows. The issue is caused by using a platform-specific `File.separator` instead of `/`: Windows's file separator is `\`, which results in a wrong URI string:

```
java.net.URISyntaxException: Illegal character in path at index 88:
spark://xxxx:57839/artifacts\bd3e1ffe-50d2-412c-8fe4-911ae160c251\classes\
```

This failure is captured by the scheduled Windows build on `master`, such as https://github.com/apache/spark/actions/runs/11958030827.

To fix this issue we just make sure that the separator is always `/` on all OSes.

### Why are the changes needed?

Fix a compilation failure in Windows.

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

Tested on my own fork with a modified Workflow that runs on PR: xupefei#1, https://github.com/xupefei/spark/actions/runs/11970330735/job/33372836765?pr=1

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes #48934 from xupefei/repl-class-uri-windows.

Authored-by: Paddy Xu <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
  • Loading branch information
xupefei authored and HyukjinKwon committed Nov 24, 2024
1 parent 3cdebbe commit 8887d53
Showing 1 changed file with 2 additions and 3 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -66,12 +66,11 @@ class ArtifactManager(session: SparkSession) extends Logging {
// The base directory/URI where all artifacts are stored for this `sessionUUID`.
protected[artifact] val (artifactPath, artifactURI): (Path, String) =
(ArtifactUtils.concatenatePaths(artifactRootPath, session.sessionUUID),
s"$artifactRootURI${File.separator}${session.sessionUUID}")
s"$artifactRootURI/${session.sessionUUID}")

// The base directory/URI where all class file artifacts are stored for this `sessionUUID`.
protected[artifact] val (classDir, replClassURI): (Path, String) =
(ArtifactUtils.concatenatePaths(artifactPath, "classes"),
s"$artifactURI${File.separator}classes${File.separator}")
(ArtifactUtils.concatenatePaths(artifactPath, "classes"), s"$artifactURI/classes/")

private lazy val alwaysApplyClassLoader =
session.conf.get(SQLConf.ARTIFACTS_SESSION_ISOLATION_ALWAYS_APPLY_CLASSLOADER.key).toBoolean
Expand Down

0 comments on commit 8887d53

Please sign in to comment.