Merge branch 'main' into release/v2.0-beta-3

.github/workflows/codegen.yaml

@@ -42,7 +42,7 @@ jobs:
  tags: github-action
 
  - name: Start containers
- run: docker-compose -f "docker-compose.test.yml" up -d --build
+ run: docker compose -f "docker-compose.test.yml" up -d --build
 
  # setup node
  - name: Install node
@@ -73,4 +73,4 @@ jobs:
 
  - name: Stop containers
  if: always()
- run: docker-compose -f "docker-compose.test.yml" down
+ run: docker compose -f "docker-compose.test.yml" down

.github/workflows/pytest.yml

@@ -35,7 +35,7 @@ jobs:
  pipenv install --deploy --dev
 
  - name: Start env
- run: docker-compose -f ../docker-compose.dev.yml up -d
+ run: docker compose -f ../docker-compose.dev.yml up -d
 
  # wait for docker containers to come up before running tests
  - name: Sleep for 60 seconds

docs/docs/admins/backups.md

@@ -0,0 +1 @@
+# Backups

docs/docs/admins/config.md

@@ -0,0 +1,6 @@
+# Configuration
+
+Each instance of Clowder can be configured at deployment time using the config file. The config file is located
+in `backend/app/config.py`. At run time, environmental variables passed into the container will override the defaults
+in the config file. When using Docker Compose, the environmental variables are set in the `docker-compose.yml` file or
+in `.env` file. In the case of Kubernetes, the environmental variables are set in the `values.yaml` file.

docs/docs/admins/index.md

@@ -0,0 +1,26 @@
+# Overview
+
+Clowder relies heavily on [Docker](https://www.docker.com/) containers for both development and production. There are
+two options to deploy Clowder in production: [Docker Compose]((dockercompose.md)) and [Kubernetes Helm
+charts](https://helm.sh/).
+
+## Docker Compose
+
+The easiest way to get started is to clone the repository and run `docker compose up` in the main directory. This will
+start all the services and the web interface. Internally, docker compose will run
+[traeifk](https://traefik.io/traefik/) as a reverse proxy and make all services available on the host machine.
+
+```
+docker compose up
+open http://localhost
+```
+
+For more information on using docker compose see [Docker Compose](dockercompose.md).
+
+## Kubernetes Helm Charts
+
+While this works find for single node instances, if you want to scale the system, you can use the Helm charts to deploy
+the application on a Kubernetes cluster. The Helm charts are available in the `deployments/kubernetes/charts` directory.
+See *README.md* in that directory for more information.
+
+For more information on kubernetes and Helm see [Kubernetes](kubernetes.md).

docs/docs/devs/architecture.md

@@ -10,3 +10,9 @@ Lines show interactions between containers over the docker network.
  ![Clowder v2 architecture](../assets/images/architecture.jpg)
  <figcaption>Architecture</figcaption>
 </figure>
+I want to `slkdfjslkdjf` include a 
+
+- alkdjflskjdf
+- alskdjflksajfd
+ - [slkjdf](http://clowder.com)
+ - 

docs/docs/devs/frontend/lazyloading.md

@@ -0,0 +1 @@
+# Webpack Lazy Loading

docs/docs/devs/precommits.md

@@ -0,0 +1,2 @@
+# Precommits and Linting
+

docs/docs/users/index.md

@@ -1 +1,60 @@
-# ⚠ Under construction ⚠
+# Getting Started
+
+This section is for end users interested in using Clowder as a data repository, data sharing
+platform, or to develop data pipelines.
+
+There are two main ways to interact with Clowder
+
+- web interface
+- web API
+
+Most users will use the web interface to upload, download, and manage data, metadata, analytics and data visualizations.
+Advanced users can use the web API to interact with Clowder programmatically through scripts, Jupyter notebooks, or
+other services.
+
+## Data Model
+
+The main resources in Clowder is the **dataset**.
+A Clowder dataset is a collection of files, folders, metadata and visualizations.
+Datasets can be shared and downloaded.
+When uploading new data, users can create a new dataset or add files to existing datasets.
+
+One unique aspect of Clowder is the ability to attach **metadata** to files and datasets.
+Metadata can be used to store any kind of information about the underlying data, such as provenance, data quality, or
+data processing steps.
+A dataset or a file can have multiple metadata documents attached to it.
+These metadata documents can be manually added by the owner of the file or dataset, or automatically generated by
+information extractors.
+If the dataset is shared with other users, they can also add metadata to the dataset.
+
+Unlike most other systems which store metadata as key value pairs, Clowder represents metadata as JSON for Linking
+Data, [JSON-LD](https://json-ld.org/).
+The advantage of using JSON-LD is that it affords the simplicity of JSON, but enforces semantic information about each
+field in the JSON document.
+JSON-LD can also be serialized to [RDF](https://en.wikipedia.org/wiki/Resource_Description_Framework), make metadata
+interoperable with other systems, including [knowledge graphs](https://en.wikipedia.org/wiki/Knowledge_graph).
+
+When a user manually adds metadata to a file or dataset, they will pick from a list of metadata fields defined by the
+system.
+The administrators of the Clowder instance can define the metadata fields that are available to users.
+These metadata fields can be simple text fields, dropdowns, or more complex fields like date pickers.
+Clowder, via information extractors, can also add metadata automatically to files and datasets.
+These metadata documents can be the results a ofa manual submission to an information extractor or the result of an
+automatic extraction triggered by the system.
+
+## Signing Up
+
+Each user must sign up for an account to use Clowder. By default, users can sign up for a local account on the specific
+Clowder instance. Some instances may also allow users to sign in using single sign-on, through OAuth provider like
+Google, GitHub, [CILogon](https://cilogon.org/), [Globus](https://www.globus.org/).
+
+Depending on the configuration of the instance, users might not be automatically enabled once they have created an
+account.
+In this case, the administrator of the instance will need to enable the account.
+
+## Browsing and Searching
+
+## Uploading Data
+
+## Sharing Data
+

docs/mkdocs.yml

@@ -3,12 +3,14 @@ repo_url: https://github.com/clowder-framework/clowder2
 edit_uri: mkdocs/docs
 nav:
  - Home: index.md
- - User Guide:
- - users/index.md
+ - Users:
+ - Overview: users/index.md
  - Visualizations: users/visualizations.md
- - Developer Guide:
+ - Developers:
  - Getting started: devs/getstarted.md
  - Architecture: devs/architecture.md
+ - Frontend:
+ - devs/frontend/lazyloading.md
  - Keycloak: devs/keycloak.md
  - Metadata: devs/metadata.md
  - Mongo: devs/mongo-fastapi.md
@@ -18,6 +20,13 @@ nav:
  - Standalone Docker Deployment: devs/standalone-docker-deployment.md
  - Proposals:
  - Storage: devs/storage.md
+ - Admins:
+ - Overview: admins/index.md
+ - Deployment:
+ - 'Docker compose': admins/dockercompose.md
+ - admins/kubernetes.md
+ - admins/config.md
+ - admins/backups.md
  - About: about.md
 
 theme: