Merge remote-tracking branch 'origin/main' into 978-migrate-users

tcnichol · tcnichol · commit 58ada467e4f9 · 2024-08-14T12:35:38.000-05:00
diff --git a/.github/workflows/codegen.yaml b/.github/workflows/codegen.yaml
@@ -42,7 +42,7 @@ jobs:
           tags: github-action
 
       - name: Start containers
-        run: docker-compose -f "docker-compose.test.yml" up -d --build
+        run: docker compose -f "docker-compose.test.yml" up -d --build
 
       # setup node
       - name: Install node
@@ -73,4 +73,4 @@ jobs:
 
       - name: Stop containers
         if: always()
-        run: docker-compose -f "docker-compose.test.yml" down
+        run: docker compose -f "docker-compose.test.yml" down
diff --git a/.github/workflows/pytest.yml b/.github/workflows/pytest.yml
@@ -35,7 +35,7 @@ jobs:
           pipenv install --deploy --dev
 
       - name: Start env
-        run: docker-compose -f ../docker-compose.dev.yml up -d
+        run: docker compose -f ../docker-compose.dev.yml up -d
 
       # wait for docker containers to come up before running tests
       - name: Sleep for 60 seconds
diff --git a/docs/docs/admins/backups.md b/docs/docs/admins/backups.md
@@ -0,0 +1 @@
+# Backups
diff --git a/docs/docs/admins/config.md b/docs/docs/admins/config.md
@@ -0,0 +1,6 @@
+# Configuration
+
+Each instance of Clowder can be configured at deployment time using the config file. The config file is located
+in `backend/app/config.py`. At run time, environmental variables passed into the container will override the defaults
+in the config file. When using Docker Compose, the environmental variables are set in the `docker-compose.yml` file or
+in `.env` file. In the case of Kubernetes, the environmental variables are set in the `values.yaml` file.
diff --git a/docs/docs/admins/dockercompose.md b/docs/docs/admins/dockercompose.md
diff --git a/docs/docs/admins/index.md b/docs/docs/admins/index.md
@@ -0,0 +1,26 @@
+# Overview
+
+Clowder relies heavily on [Docker](https://www.docker.com/) containers for both development and production. There are
+two options to deploy Clowder in production: [Docker Compose]((dockercompose.md)) and [Kubernetes Helm
+charts](https://helm.sh/).
+
+## Docker Compose
+
+The easiest way to get started is to clone the repository and run `docker compose up` in the main directory. This will
+start all the services and the web interface. Internally, docker compose will run
+[traeifk](https://traefik.io/traefik/) as a reverse proxy and make all services available on the host machine.
+
+```
+docker compose up
+open http://localhost
+```
+
+For more information on using docker compose see [Docker Compose](dockercompose.md).
+
+## Kubernetes Helm Charts
+
+While this works find for single node instances, if you want to scale the system, you can use the Helm charts to deploy
+the application on a Kubernetes cluster. The Helm charts are available in the `deployments/kubernetes/charts` directory.
+See *README.md* in that directory for more information.
+
+For more information on kubernetes and Helm see [Kubernetes](kubernetes.md).
diff --git a/docs/docs/admins/kubernetes.md b/docs/docs/admins/kubernetes.md
diff --git a/docs/docs/devs/architecture.md b/docs/docs/devs/architecture.md
@@ -10,3 +10,9 @@ Lines show interactions between containers over the docker network.
   ![Clowder v2 architecture](../assets/images/architecture.jpg)
   <figcaption>Architecture</figcaption>
 </figure>
+I want to `slkdfjslkdjf` include a 
+
+- alkdjflskjdf
+- alskdjflksajfd
+  - [slkjdf](http://clowder.com)
+  - 
diff --git a/docs/docs/devs/frontend/lazyloading.md b/docs/docs/devs/frontend/lazyloading.md
@@ -0,0 +1 @@
+# Webpack Lazy Loading
diff --git a/docs/docs/devs/precommits.md b/docs/docs/devs/precommits.md
@@ -0,0 +1,2 @@
+# Precommits and Linting
+
diff --git a/docs/docs/users/index.md b/docs/docs/users/index.md
@@ -1 +1,60 @@
-# ⚠ Under construction ⚠
+# Getting Started
+
+This section is for end users interested in using Clowder as a data repository, data sharing
+platform, or to develop data pipelines.
+
+There are two main ways to interact with Clowder
+
+- web interface
+- web API
+
+Most users will use the web interface to upload, download, and manage data, metadata, analytics and data visualizations.
+Advanced users can use the web API to interact with Clowder programmatically through scripts, Jupyter notebooks, or
+other services.
+
+## Data Model
+
+The main resources in Clowder is the **dataset**.
+A Clowder dataset is a collection of files, folders, metadata and visualizations.
+Datasets can be shared and downloaded.
+When uploading new data, users can create a new dataset or add files to existing datasets.
+
+One unique aspect of Clowder is the ability to attach **metadata** to files and datasets.
+Metadata can be used to store any kind of information about the underlying data, such as provenance, data quality, or
+data processing steps.
+A dataset or a file can have multiple metadata documents attached to it.
+These metadata documents can be manually added by the owner of the file or dataset, or automatically generated by
+information extractors.
+If the dataset is shared with other users, they can also add metadata to the dataset.
+
+Unlike most other systems which store metadata as key value pairs, Clowder represents metadata as JSON for Linking
+Data, [JSON-LD](https://json-ld.org/).
+The advantage of using JSON-LD is that it affords the simplicity of JSON, but enforces semantic information about each
+field in the JSON document.
+JSON-LD can also be serialized to [RDF](https://en.wikipedia.org/wiki/Resource_Description_Framework), make metadata
+interoperable with other systems, including [knowledge graphs](https://en.wikipedia.org/wiki/Knowledge_graph).
+
+When a user manually adds metadata to a file or dataset, they will pick from a list of metadata fields defined by the
+system.
+The administrators of the Clowder instance can define the metadata fields that are available to users.
+These metadata fields can be simple text fields, dropdowns, or more complex fields like date pickers.
+Clowder, via information extractors, can also add metadata automatically to files and datasets.
+These metadata documents can be the results a ofa manual submission to an information extractor or the result of an
+automatic extraction triggered by the system.
+
+## Signing Up
+
+Each user must sign up for an account to use Clowder. By default, users can sign up for a local account on the specific
+Clowder instance. Some instances may also allow users to sign in using single sign-on, through OAuth provider like
+Google, GitHub, [CILogon](https://cilogon.org/), [Globus](https://www.globus.org/).
+
+Depending on the configuration of the instance, users might not be automatically enabled once they have created an
+account.
+In this case, the administrator of the instance will need to enable the account.
+
+## Browsing and Searching
+
+## Uploading Data
+
+## Sharing Data
+
diff --git a/docs/mkdocs.yml b/docs/mkdocs.yml
@@ -3,12 +3,14 @@ repo_url: https://github.com/clowder-framework/clowder2
 edit_uri: mkdocs/docs
 nav:
   - Home: index.md
-  - User Guide:
-      - users/index.md
+  - Users:
+      - Overview: users/index.md
       - Visualizations: users/visualizations.md
-  - Developer Guide:
+  - Developers:
       - Getting started: devs/getstarted.md
       - Architecture: devs/architecture.md
+      - Frontend:
+        - devs/frontend/lazyloading.md
       - Keycloak: devs/keycloak.md
       - Metadata: devs/metadata.md
       - Mongo: devs/mongo-fastapi.md
@@ -18,6 +20,13 @@ nav:
       - Standalone Docker Deployment: devs/standalone-docker-deployment.md
       - Proposals:
         - Storage: devs/storage.md
+  - Admins:
+      - Overview: admins/index.md
+      - Deployment:
+        - 'Docker compose': admins/dockercompose.md
+        - admins/kubernetes.md
+      - admins/config.md
+      - admins/backups.md
   - About: about.md
 
 theme:
