-
Notifications
You must be signed in to change notification settings - Fork 6
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge branch 'main' into release/v2.0-beta-3
- Loading branch information
Showing
12 changed files
with
117 additions
and
7 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
# Backups |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,6 @@ | ||
# Configuration | ||
|
||
Each instance of Clowder can be configured at deployment time using the config file. The config file is located | ||
in `backend/app/config.py`. At run time, environmental variables passed into the container will override the defaults | ||
in the config file. When using Docker Compose, the environmental variables are set in the `docker-compose.yml` file or | ||
in `.env` file. In the case of Kubernetes, the environmental variables are set in the `values.yaml` file. |
Empty file.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,26 @@ | ||
# Overview | ||
|
||
Clowder relies heavily on [Docker](https://www.docker.com/) containers for both development and production. There are | ||
two options to deploy Clowder in production: [Docker Compose]((dockercompose.md)) and [Kubernetes Helm | ||
charts](https://helm.sh/). | ||
|
||
## Docker Compose | ||
|
||
The easiest way to get started is to clone the repository and run `docker compose up` in the main directory. This will | ||
start all the services and the web interface. Internally, docker compose will run | ||
[traeifk](https://traefik.io/traefik/) as a reverse proxy and make all services available on the host machine. | ||
|
||
``` | ||
docker compose up | ||
open http://localhost | ||
``` | ||
|
||
For more information on using docker compose see [Docker Compose](dockercompose.md). | ||
|
||
## Kubernetes Helm Charts | ||
|
||
While this works find for single node instances, if you want to scale the system, you can use the Helm charts to deploy | ||
the application on a Kubernetes cluster. The Helm charts are available in the `deployments/kubernetes/charts` directory. | ||
See *README.md* in that directory for more information. | ||
|
||
For more information on kubernetes and Helm see [Kubernetes](kubernetes.md). |
Empty file.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
# Webpack Lazy Loading |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,2 @@ | ||
# Precommits and Linting | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1 +1,60 @@ | ||
# ⚠ Under construction ⚠ | ||
# Getting Started | ||
|
||
This section is for end users interested in using Clowder as a data repository, data sharing | ||
platform, or to develop data pipelines. | ||
|
||
There are two main ways to interact with Clowder | ||
|
||
- web interface | ||
- web API | ||
|
||
Most users will use the web interface to upload, download, and manage data, metadata, analytics and data visualizations. | ||
Advanced users can use the web API to interact with Clowder programmatically through scripts, Jupyter notebooks, or | ||
other services. | ||
|
||
## Data Model | ||
|
||
The main resources in Clowder is the **dataset**. | ||
A Clowder dataset is a collection of files, folders, metadata and visualizations. | ||
Datasets can be shared and downloaded. | ||
When uploading new data, users can create a new dataset or add files to existing datasets. | ||
|
||
One unique aspect of Clowder is the ability to attach **metadata** to files and datasets. | ||
Metadata can be used to store any kind of information about the underlying data, such as provenance, data quality, or | ||
data processing steps. | ||
A dataset or a file can have multiple metadata documents attached to it. | ||
These metadata documents can be manually added by the owner of the file or dataset, or automatically generated by | ||
information extractors. | ||
If the dataset is shared with other users, they can also add metadata to the dataset. | ||
|
||
Unlike most other systems which store metadata as key value pairs, Clowder represents metadata as JSON for Linking | ||
Data, [JSON-LD](https://json-ld.org/). | ||
The advantage of using JSON-LD is that it affords the simplicity of JSON, but enforces semantic information about each | ||
field in the JSON document. | ||
JSON-LD can also be serialized to [RDF](https://en.wikipedia.org/wiki/Resource_Description_Framework), make metadata | ||
interoperable with other systems, including [knowledge graphs](https://en.wikipedia.org/wiki/Knowledge_graph). | ||
|
||
When a user manually adds metadata to a file or dataset, they will pick from a list of metadata fields defined by the | ||
system. | ||
The administrators of the Clowder instance can define the metadata fields that are available to users. | ||
These metadata fields can be simple text fields, dropdowns, or more complex fields like date pickers. | ||
Clowder, via information extractors, can also add metadata automatically to files and datasets. | ||
These metadata documents can be the results a ofa manual submission to an information extractor or the result of an | ||
automatic extraction triggered by the system. | ||
|
||
## Signing Up | ||
|
||
Each user must sign up for an account to use Clowder. By default, users can sign up for a local account on the specific | ||
Clowder instance. Some instances may also allow users to sign in using single sign-on, through OAuth provider like | ||
Google, GitHub, [CILogon](https://cilogon.org/), [Globus](https://www.globus.org/). | ||
|
||
Depending on the configuration of the instance, users might not be automatically enabled once they have created an | ||
account. | ||
In this case, the administrator of the instance will need to enable the account. | ||
|
||
## Browsing and Searching | ||
|
||
## Uploading Data | ||
|
||
## Sharing Data | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters