Frameless is a proof-of-concept library for working with Spark using more expressive types. It consists of the following modules:
catsfor using Spark with catsdataframe(deprecated) for more strongly typedDataFramesdatasetfor more strongly typedDatasets
The Frameless project and contributors support the Typelevel Code of Conduct and want all its associated channels (e.g. GitHub, Gitter) to be a safe and friendly environment for contributing and learning.
Benefits of using TypedDataset compared to vanilla Dataset:
- Typesafe columns referencing and expressions
- Customizable, typecafe encoders
- Enhanced type signature for some built-in functions
Frameless is compiled against Scala 2.11.x.
Note that while Frameless is still getting off the ground, it is very possible that breaking changes will be made for at least the next few versions.
To use frameless add the following dependencies as needed:
resolvers += Resolver.sonatypeRepo("releases")
val framelessVersion = "0.1.0"
libraryDependencies ++= List(
"io.github.adelbertc" %% "frameless-cats" % framelessVersion,
"io.github.adelbertc" %% "frameless-dataset" % framelessVersion,
"io.github.adelbertc" %% "frameless-dataframe" % framelessVersion
)We require at least one sign-off (thumbs-up, +1, or similar) to merge pull requests. The current maintainers (people who can merge pull requests) are:
Code is provided under the Apache 2.0 license available at http://opensource.org/licenses/Apache-2.0, as well as in the LICENSE file. This is the same license used as Spark.