From 377177ebc6cf604b2e489621298506d511dcdf4d Mon Sep 17 00:00:00 2001 From: Luigi Marini Date: Tue, 13 Aug 2024 11:32:13 -0500 Subject: [PATCH 1/2] Docs updates (#1101) * Admin section with starting instructions on deploying. * Overall cleanup. Started new user section and a few other placeholder pages. --- docs/docs/admins/backups.md | 1 + docs/docs/admins/config.md | 6 +++ docs/docs/admins/dockercompose.md | 0 docs/docs/admins/index.md | 26 +++++++++++ docs/docs/admins/kubernetes.md | 0 docs/docs/devs/architecture.md | 6 +++ docs/docs/devs/frontend/lazyloading.md | 1 + docs/docs/devs/precommits.md | 2 + docs/docs/users/index.md | 61 +++++++++++++++++++++++++- docs/mkdocs.yml | 15 +++++-- 10 files changed, 114 insertions(+), 4 deletions(-) create mode 100644 docs/docs/admins/backups.md create mode 100644 docs/docs/admins/config.md create mode 100644 docs/docs/admins/dockercompose.md create mode 100644 docs/docs/admins/index.md create mode 100644 docs/docs/admins/kubernetes.md create mode 100644 docs/docs/devs/frontend/lazyloading.md create mode 100644 docs/docs/devs/precommits.md diff --git a/docs/docs/admins/backups.md b/docs/docs/admins/backups.md new file mode 100644 index 000000000..8c0c04ce2 --- /dev/null +++ b/docs/docs/admins/backups.md @@ -0,0 +1 @@ +# Backups \ No newline at end of file diff --git a/docs/docs/admins/config.md b/docs/docs/admins/config.md new file mode 100644 index 000000000..0a7b1bd35 --- /dev/null +++ b/docs/docs/admins/config.md @@ -0,0 +1,6 @@ +# Configuration + +Each instance of Clowder can be configured at deployment time using the config file. The config file is located +in `backend/app/config.py`. At run time, environmental variables passed into the container will override the defaults +in the config file. When using Docker Compose, the environmental variables are set in the `docker-compose.yml` file or +in `.env` file. In the case of Kubernetes, the environmental variables are set in the `values.yaml` file. \ No newline at end of file diff --git a/docs/docs/admins/dockercompose.md b/docs/docs/admins/dockercompose.md new file mode 100644 index 000000000..e69de29bb diff --git a/docs/docs/admins/index.md b/docs/docs/admins/index.md new file mode 100644 index 000000000..91df44876 --- /dev/null +++ b/docs/docs/admins/index.md @@ -0,0 +1,26 @@ +# Overview + +Clowder relies heavily on [Docker](https://www.docker.com/) containers for both development and production. There are +two options to deploy Clowder in production: [Docker Compose]((dockercompose.md)) and [Kubernetes Helm +charts](https://helm.sh/). + +## Docker Compose + +The easiest way to get started is to clone the repository and run `docker compose up` in the main directory. This will +start all the services and the web interface. Internally, docker compose will run +[traeifk](https://traefik.io/traefik/) as a reverse proxy and make all services available on the host machine. + +``` +docker compose up +open http://localhost +``` + +For more information on using docker compose see [Docker Compose](dockercompose.md). + +## Kubernetes Helm Charts + +While this works find for single node instances, if you want to scale the system, you can use the Helm charts to deploy +the application on a Kubernetes cluster. The Helm charts are available in the `deployments/kubernetes/charts` directory. +See *README.md* in that directory for more information. + +For more information on kubernetes and Helm see [Kubernetes](kubernetes.md). diff --git a/docs/docs/admins/kubernetes.md b/docs/docs/admins/kubernetes.md new file mode 100644 index 000000000..e69de29bb diff --git a/docs/docs/devs/architecture.md b/docs/docs/devs/architecture.md index b51f3838b..392b8fd8f 100644 --- a/docs/docs/devs/architecture.md +++ b/docs/docs/devs/architecture.md @@ -10,3 +10,9 @@ Lines show interactions between containers over the docker network. ![Clowder v2 architecture](../assets/images/architecture.jpg)
Architecture
+I want to `slkdfjslkdjf` include a + +- alkdjflskjdf +- alskdjflksajfd + - [slkjdf](http://clowder.com) + - diff --git a/docs/docs/devs/frontend/lazyloading.md b/docs/docs/devs/frontend/lazyloading.md new file mode 100644 index 000000000..5653e63fc --- /dev/null +++ b/docs/docs/devs/frontend/lazyloading.md @@ -0,0 +1 @@ +# Webpack Lazy Loading \ No newline at end of file diff --git a/docs/docs/devs/precommits.md b/docs/docs/devs/precommits.md new file mode 100644 index 000000000..5c559b681 --- /dev/null +++ b/docs/docs/devs/precommits.md @@ -0,0 +1,2 @@ +# Precommits and Linting + diff --git a/docs/docs/users/index.md b/docs/docs/users/index.md index 6f8d37813..21b025d4b 100644 --- a/docs/docs/users/index.md +++ b/docs/docs/users/index.md @@ -1 +1,60 @@ -# ⚠ Under construction ⚠ +# Getting Started + +This section is for end users interested in using Clowder as a data repository, data sharing +platform, or to develop data pipelines. + +There are two main ways to interact with Clowder + +- web interface +- web API + +Most users will use the web interface to upload, download, and manage data, metadata, analytics and data visualizations. +Advanced users can use the web API to interact with Clowder programmatically through scripts, Jupyter notebooks, or +other services. + +## Data Model + +The main resources in Clowder is the **dataset**. +A Clowder dataset is a collection of files, folders, metadata and visualizations. +Datasets can be shared and downloaded. +When uploading new data, users can create a new dataset or add files to existing datasets. + +One unique aspect of Clowder is the ability to attach **metadata** to files and datasets. +Metadata can be used to store any kind of information about the underlying data, such as provenance, data quality, or +data processing steps. +A dataset or a file can have multiple metadata documents attached to it. +These metadata documents can be manually added by the owner of the file or dataset, or automatically generated by +information extractors. +If the dataset is shared with other users, they can also add metadata to the dataset. + +Unlike most other systems which store metadata as key value pairs, Clowder represents metadata as JSON for Linking +Data, [JSON-LD](https://json-ld.org/). +The advantage of using JSON-LD is that it affords the simplicity of JSON, but enforces semantic information about each +field in the JSON document. +JSON-LD can also be serialized to [RDF](https://en.wikipedia.org/wiki/Resource_Description_Framework), make metadata +interoperable with other systems, including [knowledge graphs](https://en.wikipedia.org/wiki/Knowledge_graph). + +When a user manually adds metadata to a file or dataset, they will pick from a list of metadata fields defined by the +system. +The administrators of the Clowder instance can define the metadata fields that are available to users. +These metadata fields can be simple text fields, dropdowns, or more complex fields like date pickers. +Clowder, via information extractors, can also add metadata automatically to files and datasets. +These metadata documents can be the results a ofa manual submission to an information extractor or the result of an +automatic extraction triggered by the system. + +## Signing Up + +Each user must sign up for an account to use Clowder. By default, users can sign up for a local account on the specific +Clowder instance. Some instances may also allow users to sign in using single sign-on, through OAuth provider like +Google, GitHub, [CILogon](https://cilogon.org/), [Globus](https://www.globus.org/). + +Depending on the configuration of the instance, users might not be automatically enabled once they have created an +account. +In this case, the administrator of the instance will need to enable the account. + +## Browsing and Searching + +## Uploading Data + +## Sharing Data + diff --git a/docs/mkdocs.yml b/docs/mkdocs.yml index 486994ab1..9cabe4e1e 100644 --- a/docs/mkdocs.yml +++ b/docs/mkdocs.yml @@ -3,12 +3,14 @@ repo_url: https://github.com/clowder-framework/clowder2 edit_uri: mkdocs/docs nav: - Home: index.md - - User Guide: - - users/index.md + - Users: + - Overview: users/index.md - Visualizations: users/visualizations.md - - Developer Guide: + - Developers: - Getting started: devs/getstarted.md - Architecture: devs/architecture.md + - Frontend: + - devs/frontend/lazyloading.md - Keycloak: devs/keycloak.md - Metadata: devs/metadata.md - Mongo: devs/mongo-fastapi.md @@ -18,6 +20,13 @@ nav: - Standalone Docker Deployment: devs/standalone-docker-deployment.md - Proposals: - Storage: devs/storage.md + - Admins: + - Overview: admins/index.md + - Deployment: + - 'Docker compose': admins/dockercompose.md + - admins/kubernetes.md + - admins/config.md + - admins/backups.md - About: about.md theme: From d9fc7e47122d7642386ac26cb421c21bbb3e5a5a Mon Sep 17 00:00:00 2001 From: Todd Nicholson <40038535+tcnichol@users.noreply.github.com> Date: Tue, 13 Aug 2024 11:58:18 -0500 Subject: [PATCH 2/2] fix tests that are failing (#1167) * changing docker-compose to docker compose * changing docker-compose to docker compose --- .github/workflows/codegen.yaml | 4 ++-- .github/workflows/pytest.yml | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/.github/workflows/codegen.yaml b/.github/workflows/codegen.yaml index 8f5cfa968..419c01071 100644 --- a/.github/workflows/codegen.yaml +++ b/.github/workflows/codegen.yaml @@ -42,7 +42,7 @@ jobs: tags: github-action - name: Start containers - run: docker-compose -f "docker-compose.test.yml" up -d --build + run: docker compose -f "docker-compose.test.yml" up -d --build # setup node - name: Install node @@ -73,4 +73,4 @@ jobs: - name: Stop containers if: always() - run: docker-compose -f "docker-compose.test.yml" down + run: docker compose -f "docker-compose.test.yml" down diff --git a/.github/workflows/pytest.yml b/.github/workflows/pytest.yml index 4322a15c1..3dcdee2ae 100644 --- a/.github/workflows/pytest.yml +++ b/.github/workflows/pytest.yml @@ -35,7 +35,7 @@ jobs: pipenv install --deploy --dev - name: Start env - run: docker-compose -f ../docker-compose.dev.yml up -d + run: docker compose -f ../docker-compose.dev.yml up -d # wait for docker containers to come up before running tests - name: Sleep for 60 seconds