mise-spark

Apache Spark plugin for the mise version manager.

Contents

• Dependencies
• Usage
• Contributing
• License

Dependencies

• bash, curl, tar, and POSIX utilities
• Java 8/11/17 - see Spark docs

Usage

Add plugin

mise plugins install spark https://github.com/benberryallwood/mise-spark

List and set Spark versions

This is currently slow, so may require increasing the fetch_remote_versions_timeout setting:

# As mise setting
mise settings set fetch_remote_versions_timeout 30s

# As environment variable
export MISE_FETCH_REMOTE_VERSIONS_TIMEOUT=30s

Apache provide various distributions, including/excluding Hadoop, supporting different Scala versions, etc.

Important

mise ls-remote spark lists all installable versions, some of which may not be installable on your machine

mise latest spark gives the short version, so by combining the two you can find a suitable version. Unless you know otherwise, you probably want the version including hadoop with no Scala version specified.

# Get all versions for the latest stable release
mise ls-remote spark | grep $(mise latest spark)

# Install specific version
mise install [email protected]

# Set a version globally (on your ~/.config/mise/config.toml file)
mise use --global [email protected]

# Now spark commands are available
spark-shell --help

Check mise docs for more instructions on how to install & manage versions.

Using provided scripts

If you wish to use Spark scripts provided in sbin/, such as Spark Connect, you can do so with mise where:

$(mise where spark)/sbin/start-connect-server.sh --help

Contributing

Contributions of any kind welcome! See the contributing guide.

Thanks goes to these contributors!

Thanks also to the asdf-spark author and contributors.

License

See LICENSE © Ben Berry-Allwood