Skip to content

Commit

Permalink
refine the pending FIXME in addFile function (#30)
Browse files Browse the repository at this point in the history
  • Loading branch information
xkx9431 authored Dec 28, 2023
1 parent ea6cdc4 commit 6dec498
Showing 1 changed file with 15 additions and 4 deletions.
19 changes: 15 additions & 4 deletions docs/SparkContext.md
Original file line number Diff line number Diff line change
Expand Up @@ -204,15 +204,26 @@ addFile(
path: String): Unit
```

`addFile` creates a Hadoop `Path` from the given `path`. For a no-schema path, `addFile` converts it to a canonical form.

`addFile` prints out the following WARN message to the logs and exits.
Firstly, `addFile` validate the schema of given `path`. For a no-schema path, `addFile` converts it to a canonical form. For a local schema path, `addFile` prints out the following WARN message to the logs and exits.

```text
File with 'local' scheme is not supported to add to file server, since it is already available on every node.
```
And for other schema path, `addFile` creates a Hadoop Path from the given path.

`addFile` Will validate the URL if the path is an HTTP, HTTPS or FTP URI.

`addFile` Will throw `SparkException` with below message if path is local directories but not in local mode.

`addFile`...FIXME
```text
addFile does not support local directories when not running local mode.
```

`addFile` Will throw `SparkException` with below message if path is directories but not turn on `recursive` flag.

```text
Added file $hadoopPath is a directory and recursive is not turned on.
```

In the end, `addFile` adds the file to the [addedFiles](#addedFiles) internal registry (with the current timestamp):

Expand Down

0 comments on commit 6dec498

Please sign in to comment.