DataToolkit is a batteries-included family of packages for robustly managing data. The particular package(s) you want to use will depend on the project.
- Use DataToolkit for analysis projects and scripts
- Use DataToolkitBase when making a package that needs data
- Optionally, use DataToolkitDocumenter too to document the datasets
- Use DataToolkitCore when making a package extending DataToolkit, and possibly DataToolkitStore too.
For now, this set of packages around the beta stage of development. No major changes to the core functionality or structure are anticipated, but small expansions in the data-CLI functionality and set of transformers and plugins provided by DataToolkitCommon are expected prior to the 1.0 release, and larger changes may occur if there is good reason for them.
[[iris]]
uuid = "3f3d7714-22aa-4555-a950-78f43b74b81c"
description = "Fisher's famous Iris flower measurements"
[[iris.storage]]
driver = "web"
checksum = "k12:cfb9a6a302f58e5a9b0c815bb7e8efb4"
url = "https://raw.githubusercontent.com/scikit-learn/scikit-learn/1.0/sklearn/datasets/data/iris.csv"
[[iris.loader]]
driver = "csv"
args.header = ["sepal_length", "sepal_width", "petal_length", "petal_width", "species_class"]
args.skipto = 2- DataDeps.jl
- Downloading files on-demand. Essentially implements the
webstorage driver along with some of the machinery. - DataSets.jl
- An alternate take on declarative data representation. Focused on filling a gap with JuliaHub’s cloud compute offering; less versatile overall.
- RemoteFiles.jl
- Automatically re-downloading files on a schedule. Equivalent
to the
webstorage driver when using thelifetimeparameter of thestoreplugin.
- Announcement thread on Discourse
- JuliaCon 2023 slides and recording