Skip to content

alope107/findspark

 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

12 Commits
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Find spark

PySpark isn't on sys.path by default, but that doesn't mean it can't be used as a regular library. You can address this by either symlinking pyspark into your site-packages, or adding pyspark to sys.path at runtime. findspark does the latter.

To initialize PySpark, just call

import findspark
findspark.init()

import pyspark
sc = pyspark.SparkContext(appName="myAppName")

Without any arguments, the SPARK_HOME environmental variable will be used, and if that isn't set, other possible install locations will be checked. If you've installed spark with

brew install apache-spark

on OS X, the location /usr/local/opt/apache-spark/libexec will be searched.

Alternatively, you can specify a location with the spark_home argument.

findspark.init('/path/to/spark_home')

To verify the automatically detected location, call

findspark.find()

About

No description, website, or topics provided.

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Python 100.0%