Merge pull request #1267 from devstream-io/docs-app-centric-docs-update

docs: update app-centric docs

Author: daniel-hutao

docs/core-concepts/config.md

@@ -1,120 +1,284 @@
-# DevStream Configuration
+# Config
 
-DevStream uses YAML files to describe your DevOps toolchain configuration.
+Now let's have a look at some config examples.
 
-## Main Config File
+TL;DR: [see the complete config file examle at the end of this doc](#7-putting-it-all-together).
 
-By default, `dtm` tries to use `./config.yaml` (under your current directory) as the main config.
+---
 
-The main config contains three sections:
+## 1 The Config File
 
-- `varFile`: the path/to/your/variables file
-- `toolFile`: the path/to/your/tools configuration file
-- `pluginDir`: the path/to/your/plugins directory, default: `~/.devstream/plugins`, or use `-d` flag to specify a directory
-- `state`: configuration of DevStream state. For example, 
+As mentioned in the overview section, the main config contains many sections, like:
 
-### Example Main Config File
+- `config`
+- `vars`
+- `tools`
+- `apps`
+- `pipelineTemplates`
 
-See the `config.yaml` example below:
+By default, `dtm` tries to use `./config.yaml` (under your working directory) as the main config.
+
+You can override the default value with `-f` or `--config-file`. Examples:
+
+```shell
+dtm apply -f path/to/your/config.yaml
+dtm apply --config-file path/to/your/config.yaml
+```
+
+---
+
+## 2 State Config
+
+The `config` section specifies where to store the DevStream state.
+
+### 2.1 Local
 
 ```yaml
-varFile: "./variables.yaml"
-toolFile: "./tools.yaml"
-appFile: "./apps.yaml"
-templateFile: "./templates.yaml"
-state:           # state config, backend can be "local", "s3", or "k8s"
-  backend: local
-  options:
-    stateFile: devstream.state
+config:
+  state:
+    backend: local
+    options:
+      stateFile: devstream.state
+```
+
+_Note: The `stateFile` under the `options` section is mandatory for the local backend._
+
+### 2.2 Kubernetes
+
+```yaml
+config:
+  state:
+    backend: k8s
+    options:
+      namespace: devstream # optional, default is "devstream", will be created if not exists
+      configmap: state     # optional, default is "state", will be created if not exists
+```
+
+### 2.3 S3
+
+```yaml
+config:
+  state:
+    backend: s3
+    options:
+      bucket: devstream-remote-state
+      region: ap-southeast-1
+      key: devstream.state
 ```
 
-### Variables File
+_Note: the `bucket`, `region`, and `key` under the `options` section are all mandatory fields for the s3 backend._
 
-The var file is a YAML file containing key-value pairs.
+---
 
-_At the moment, nested/composite values (for example, the value is a list/dictionary) are not supported yet._
+## 3 Variables
 
-See the `variables.yaml` example below:
+To not repeat yourself (Don't repeat yourself, DRY, see [here](https://en.wikipedia.org/wiki/Don%27t_repeat_yourself),) we can define some variables in a var file and use the vars in the config file.
+
+The vars section is a YAML dict containing key-value pairs. Example:
+
+Example:
 
 ```yaml
-githubUsername: daniel-hutao
-repoName: dtm-test-go
-defaultBranch: main
-dockerhubUsername: exploitht
+vars:
+  repoOwner: RepoOwner
+  repoTemplateBaseURL: github.com/devstream-io
+  imageRepoOwner: ImageRepoOwner
 ```
 
-### Tool File
+To use these variables in a config file, use the following syntax:
 
-Tool file contains a list of tools.
+```yaml
+[[ varNameHere ]]
+```
 
-The tool file contains:
+---
 
-- Only one section (at the moment), which is `tools`.
-- `tools` is a list of dictionaries.
-- Each dictionary defines a DevOps "tool" which is managed by a DevStream plugin
-- Each dictionary (tool) has the following mandatory fields:
-    - `name`: the name of the tool/plugin, string, without underscore
-    - `instanceID`: the id of this tool instance
-    - you can have duplicated `name` in one config file, and you can also have duplicated `instanceID` in one config file, but the `name + instanceID` combination must be unique in one config file
-- Each dictionary (tool) has an optional field which is `options`, which in turn is a dictionary containing parameters for that specific plugin. For plugins' parameters, see the "plugins" section of this document.
-- Each directory (tool) has an optional field which is `dependsOn`. Continue reading for detail about dependencies.
+## 4 Tools
 
-See the `tools.yaml` example down below:
+The tools section contains a list of tools.
 
 ```yaml
 tools:
 - name: repo-scaffolding
-  instanceID: golang-github
+  instanceID: myapp
   options:
     destinationRepo:
-      owner: [[ githubUsername ]]
-      org: ""
-      repo: [[ repoName ]]
-      branch: [[ defaultBranch ]]
+      owner: [[ githubUser ]]
+      repo: [[ app ]]
+      branch: main
       repoType: github
-    vars:
-      ImageRepo: "[[ dockerhubUsername ]]/[[ repoName ]]"
     sourceRepo:
       org: devstream-io
-      repo: dtm-scaffolding-golang
+      repo: dtm-scaffolding-python
       repoType: github
-- name: jira-github-integ
+    vars:
+      imageRepo: [[ dockerUser ]]/[[ app ]]
+- name: githubactions-python
   instanceID: default
-  dependsOn: [ "repo-scaffolding.golang-github" ]
+  dependsOn: [ repo-scaffolding.myapp ]
   options:
-    owner: [[ githubUsername ]]
-    repo: [[ repoName ]]
-    jiraBaseUrl: https://xxx.atlassian.net
-    jiraUserEmail: [email protected]
-    jiraProjectKey: zzz
+    owner: [[ githubUser ]]
+    repo:  [[ app ]]
+    language:
+      name: python
     branch: main
+    docker:
+      registry:
+        type: dockerhub
+        username: [[ dockerUser ]]
+        repository: [[ app ]]
+- name: helm-installer
+  instanceID: argocd
+- name: argocdapp
+  instanceID: default
+  dependsOn: [ "helm-installer.argocd", "githubactions-python.default" ]
+  options:
+    app:
+      name: [[ app ]]
+      namespace: argocd
+    destination:
+      server: https://kubernetes.default.svc
+      namespace: default
+    source:
+      valuefile: values.yaml
+      path: helm/[[ app ]]
+      repoURL: ${{repo-scaffolding.myapp.outputs.repoURL}}
 ```
 
-### State
+If you want tool A to be installed before tool B, you can let tool B depend on tool A.
 
-The `state` section specifies where to store DevStream state. As of now (v0.5.0), we only support local backend.
+The syntax for dependency is:
 
-From v0.6.0 on, we will support both "local" and "s3" backend store the DevStream state.
+```yaml
+dependsOn: [ "ToolName.ToolInstanceID" ]
+```
 
-Read the section [The State Section in the Main Config](./stateconfig.md) for more details.
+Since `dependsOn` is a list, a tool can have multiple dependencies:
 
-## Default Values
+```
+dependsOn: [ "ToolName1.ToolInstanceID1", "ToolName2.ToolInstanceID2", "..." ]
+```
 
-By default, `dtm` uses `config.yaml` as the main config file.
+---
 
-### Specifying a Main Config File Explicitly 
+## 5 Apps
 
-You can override the default value with `-f` or `--config-file`. Examples:
+The Apps section looks like the following:
 
-```shell
-dtm apply -f path/to/your/config.yaml
-dtm apply --config-file path/to/your/config.yaml
+```yaml
+apps:
+- name: service-A
+  spec:
+    language: python
+    framework: django
+  repo:
+    scmType: github
+    owner: devstream-io
+    org: devstream-io # either owner or org must exist
+    name: service-A   # defaults to "app.name"
+    url: github.com/devstream-io/repo-name   # optional. if exists, no need for the scm/owner/org/name sections
+    apiURL: gitlab.com/some/path/to/your/api # optional, in case of self-managed git
+  repoTemplate:   # optional. if repoTemplate isn't empty, create repo according to the template
+    scmType: github
+    owner: devstream-io
+    org: devstream-io # either owner or org must exist
+    name: dtm-scaffolding-golang
+    url: github.com/devstream-io/repo-name   # optional. if exists, no need for the scm/owner/org/name sections
+    vars:  # optional
+      foo: bar  # variables used for repoTemplate specifically
+  ci:
+  - type: template              # type template means it's a reference to the pipeline template. read the next section.
+    templateName: ci-pipeline-1
+    options: # optional, used to override defaults in the template
+    vars:    # optional, key/values to be passed to the template
+      key: value
+  cd:
+  - type: template              # type template means it's a reference to the pipeline template. read the next section.
+    templateName: cd-pipeline-1
+    options: # optional, used to override defaults in the template
+    vars:    # optional, key/values to be passed to the template
+      key: value
+```
+
+Since many key/values have default values, it's possible to use the following minimum config for the apps section:
+
+```yaml
+apps:
+- name: myapp1
+  spec:
+    language: python
+    framework: django
+  repo:
+    url: github.com/ironcore864/myapp1
+  repoTemplate:
+    url: github.com/devstream-io/dtm-scaffolding-python
+  ci:
+  - type: githubactions
+  cd:
+  - type: argocdapp
+```
+
+---
+
+## 6 Pipeline Templates
+
+You can define some pipeline templates in the pipelineTemplates section:
+
+```yaml
+pipelineTemplates:
+- name: ci-pipeline-1
+  type: githubactions # corresponds to a DevStream plugin
+  options:
+    branch: main      # optional
+    docker:
+      registry:
+        type: dockerhub
+        username: [[ dockerUser ]]
+        repository: [[ app ]]
+- name: cd-pipeline-1
+  type: argocdapp
+  options:
+    app:
+      namespace: argocd
+    destination:
+      server: https://kubernetes.default.svc
+      namespace: default
+    source:
+      valuefile: values.yaml
+      path: helm/[[ app ]]
+      repoURL: ${{repo-scaffolding.myapp.outputs.repoURL}}
 ```
 
-### No Defaults for varFile and toolFile
+Then, you can refer to these pipeline templates in the apps file.
+
+---
 
-For `varFile` and `toolFile`, no default values are provided.
+## 7 Putting It All Together
 
-If `varFile` isn't specified in the main config, `dtm` will not use any var files, even if there is already a file named `variables.yaml` under the current directory.
+Here's a complete example:
 
-Similarly, if `toolFile` isn't specified in the main config, `dtm` will throw an error, even if there is a `tools.yaml` file under the current directory.
+```yaml
+config:
+  state:
+  backend: local
+  options:
+    stateFile: devstream.state
+
+tools:
+- name: helm-installer
+  instanceID: argocd
+
+apps:
+- name: myapp1
+  spec:
+    language: python
+    framework: django
+  repo:
+    url: github.com/ironcore864/myapp1
+  repoTemplate:
+    url: github.com/devstream-io/dtm-scaffolding-python
+  ci:
+  - type: githubactions
+  cd:
+  - type: argocdapp
+```

docs/core-concepts/overview.md

@@ -29,14 +29,13 @@ The main config file, which defaults to `config.yaml` in the working directory,
 
 There are a few different configs, but please don't be overwhelmed because some are not mandatory, and [you can define all things within a single file](https://stackoverflow.com/questions/50788277/why-3-dashes-hyphen-in-yaml-file).
 
-Configurations in the main config:
-
-- `pluginDir`: the directory where DevStream plugins are stored, defaults to `~/.devstream/plugins`. You can also use the `-d' flag to specify a directory.
-- `state`: state-related settings. Read more [here](./state.md).
-- `varFile`: the location of the variable file used by DevStream. The variable file is a YAML of key/values, which can be referred to in the toolFile/appFile/templateFile.
-- `toolFile`: the location of the tool file used by DevStream. The tool file is a list of DevStream _Tools_, each containing its name, instanceID (unique identifier), and options. Read more [here](./tools-apps.md).
-- `appFile`: the location of the app file used by DevStream. The app file is a list of _Apps_, another DevStream concept, each corresponding to a microservice. Read more [here](./tools-apps.md).
-- `templateFile`: the location of the template file used by DevStream. The templateFile can be referred to by DevStream _Apps_. Read more [here](./tools-apps.md).
+Configurations in the main config contains multiple sections:
+
+- `config`: basic configuration of DevStream, at the moment mainly state-related settings. Read more [here](./state.md).
+- `vars`: variable definitions. Key/value pairs, which can be referred to in the tools/apps/pipelineTemplates sections.
+- `tools`: a list of DevStream _Tools_, each containing its name, instanceID (unique identifier), and options. Read more [here](./tools-apps.md).
+- `apps`: a list of _Apps_, another DevStream concept, each corresponding to a microservice. Read more [here](./tools-apps.md).
+- `pipelineTemplates`: a list of templates which can be referred to by DevStream _Apps_. Read more [here](./tools-apps.md).
 
 ---