|
| 1 | +# Run multiple applications with Dapr sidecars |
| 2 | + |
| 3 | +Author(s): Mukundan Sundararajan |
| 4 | + |
| 5 | +State: Ready for implementation |
| 6 | + |
| 7 | +Updated: 30th Nov 2022 |
| 8 | + |
| 9 | +## Overview |
| 10 | + |
| 11 | +This is a proposal for feature to be included in the dapr CLI which allows easy way to start multiple services that needs to be run in tandem along with their `daprd` sidecars in local self-hosted mode. |
| 12 | + |
| 13 | +## Background |
| 14 | + |
| 15 | +Currently to run multiple services along with `dapr sidecar` locally, users need to run multiple `dapr run` commands, keep track of all ports opened, the components folders that each service refers to, the config file each service refers to etc. |
| 16 | +There are also multiple other flags that can be used to tweak behavior of `dapr run` command eg: `--unix-domain-socket`, `--dapr-internal-grpc-port`, `--app-health-check-path` etc. |
| 17 | + |
| 18 | +This increases the complexity of using dapr in development, where users want to run multiple services in local mode and be able to partially/fully replicate the production sceanrio. |
| 19 | + |
| 20 | +In K8s mode this is alleviated through the use of helm/deployment YAML files. There is currently no such capability available for local self hosted mode. |
| 21 | + |
| 22 | +Asking a user to run multiple different `dapr run` commands each with different flags, increases the complxity for users onboarding onto Dapr. |
| 23 | + |
| 24 | +### Why dapr CLI? |
| 25 | + |
| 26 | +From the initial [proposal](https://github.com/dapr/community/issues/207), the solution was proposed as a seprate repo and CLI in itself. But later it was suggested to use dapr CLI itself to have a `compose` command in it instead of a having a separate CLI. |
| 27 | +The main reason for including it in the dapr CLI itself is that, users do not have to download and use a new CLI in addition to the dapr CLI. |
| 28 | + |
| 29 | +This feature is tightly coupled and opinionated on how `dapr` is going to be run locally, and having a separate CLI `dapr-compose` deciding on how `dapr` CLI should be used, is not a good pattern to start with. |
| 30 | + |
| 31 | +> Note: `daprd` is more generally used and considered a binary and not necessarily a CLI tool. So `dapr` CLI is not making use of another CLI but rather passing on configs for running a binary. |
| 32 | +
|
| 33 | +## Related Items |
| 34 | + |
| 35 | +### Related Proposals |
| 36 | +- https://github.com/dapr/community/issues/207 |
| 37 | +- https://github.com/dapr/cli/issues/1123 |
| 38 | + |
| 39 | +### Related Issues |
| 40 | + |
| 41 | +## Expectations and alternatives |
| 42 | + |
| 43 | +The scope of this proposal is to enhance the `run` CLI command which allowing users to define and run multiple services from a single run configuration file. |
| 44 | + |
| 45 | +This proposal specifically targets running in local environments and `slim` mode, where container engines are available. For running `daprd` container along with `app` container the solution is to use Kubernetes or docker-compose. |
| 46 | + |
| 47 | +For this proposal we will targetting runnning the applications and side cars as processes in the OS. |
| 48 | + |
| 49 | +> Note: All other commands in `dapr` CLI for self hosted mode are written to work with processes |
| 50 | +
|
| 51 | +## Requirements |
| 52 | + |
| 53 | +The main requirements for the command: |
| 54 | +- being able to configure multiple dapr apps from a single configuration file |
| 55 | +- users should be able to use normal `dapr` CLI commands for self hosted mode against any apps that are started through `dapr compose` |
| 56 | + |
| 57 | +Additional requirement for this feature is to come up with conventions on how to organize/run Dapr projects locally. |
| 58 | + |
| 59 | +## Proposed Structure for organizing Dapr projects locally |
| 60 | + |
| 61 | +Currently `dapr` CLI initializes in the home directory(user profile dir for windows) a folder called `.dapr` and the default configurations and resources to be used are stored there. |
| 62 | + |
| 63 | +Users developing different apps using dapr will have different resources/configurations that they use per application. Each time the user has to run the application with a particular config and resources directory value, they have to override the flag for `dapr run` command. |
| 64 | + |
| 65 | +Instead the following convention is proposed for loading the resources/config for an application. |
| 66 | +The command expects the following directory structure: |
| 67 | +``` |
| 68 | +.dapr/ |
| 69 | + |____ config.yaml |
| 70 | + | |
| 71 | + |____ resources/ |
| 72 | + | |
| 73 | + |____ statestore.yaml |
| 74 | + |____ pubsub.yaml |
| 75 | + |____ resiliency_conf.yaml |
| 76 | + |____ subscription.yaml |
| 77 | +``` |
| 78 | +In each app directory,there can be a `.dapr` folder, which contains a `resources` directory and a `config.yaml` file. If that directory is not present, the default locations is used which are `~/.dapr/resources/` and `~/.dapr/config.yaml` (`%USERPROFILE%` instead of `~` for windows). |
| 79 | + |
| 80 | +> Note: This change will be made in `dapr run` only when the newly introduced `-f` flag is used. See [below](#precedence-rules) for details on which folder content will take preceedence when a run configuration is given as input. |
| 81 | +
|
| 82 | +> Note: This change does not impact the `bin` folder where `dapr` CLI looks for the `daprd` and `dashboard` binaries. That will still remain the same `~/.dapr/bin/` (%USERPROFILE% for windows). |
| 83 | +
|
| 84 | +## Proposed Structure for run configuration file |
| 85 | + |
| 86 | +> Expected default file name is `dapr.yaml` |
| 87 | +
|
| 88 | +```yaml |
| 89 | +version: 1 |
| 90 | +common: |
| 91 | + resources_dir: ./app/components # any dapr resources to be shared across apps |
| 92 | + env: # any environment variable shared among apps |
| 93 | + - DEBUG: true |
| 94 | +apps: |
| 95 | + - app_id: webapp |
| 96 | + app_dir: ./webapp/ |
| 97 | + resources_dir: ./webapp/components # (optional) can be default by convention too, ignore if dir is not found. |
| 98 | + config_file: ./webapp/config.yaml # (optional) can be default by convention too, ignore if file is not found. |
| 99 | + app_protocol: HTTP |
| 100 | + app_port: 8080 |
| 101 | + app_health_check_path: "/healthz" # All _ converted to - for all properties defined under daprd section |
| 102 | + command: ["python3" "app.py"] |
| 103 | + - app_id: backend |
| 104 | + app_dir: ./backend/ |
| 105 | + app_protocol: GRPC |
| 106 | + app_port: 3000 |
| 107 | + unix_domain_socket: "/tmp/test-socket" |
| 108 | + env: |
| 109 | + - DEBUG: false |
| 110 | + command: ["./backend"] |
| 111 | +``` |
| 112 | +> Note: Running the dependencies for each app as contianers is out of scope for this discussion initially. We might consider that in the future. |
| 113 | +
|
| 114 | +- Each file contains a `common` object which contains `env`, `resources_dir` and `config_file` that can be used in common across all the apps defined in this YAML |
| 115 | +- There is an `apps` section that lists the different app configs. |
| 116 | +- Each app config has the following |
| 117 | + - `app_id` application ID (mandatory field). Passed to `daprd` as `--app-id`. |
| 118 | + - `app_dir` directory of the application (mandatory field). |
| 119 | + - `resources_dir` (optional) directory(ies) of all dapr resources (components, resiliency policies, subscription crds) (overrides common def). Passed to `daprd` as `--resources-dir`. |
| 120 | + - `config_file` (optional) the configuration file to be used for this app (overrides common def). Passed to `daprd` as `--config-file`. |
| 121 | + - `app_protocol` Application protocol, HTTP, gRPC defaults to HTTP. Passed to `daprd` as `--app-protocol`. |
| 122 | + - `app_port` port the app listens to if any. Passed to `daprd` as `--app-port`. |
| 123 | + - other dapr run parameters (mostly pass through flags to `daprd`) All properties must have `_` as separators which will be validated(so that no unknown flags are passed) and translated to `-` for cmd line arguments for `daprd`. |
| 124 | + - `command` ["exec" "arg1" "arg2"] format for application command |
| 125 | + - `env` which overrides or adds to common env var defined or the shell env var passed in when `dapr compose` is called |
| 126 | + |
| 127 | +The DAPR_HTTP_PORT and DAPR_GRPC_PORT will be passed in as extra environment variables to the application that is being run. Those flags for `daprd` can be overridden in the run configuration file above but that is optional as random ports will be assigned as needed. |
| 128 | + |
| 129 | +### Precedence rules |
| 130 | + |
| 131 | +For `env` field: |
| 132 | +> Note: In addition to the defined env fields the app also gets the `DAPR_HTTP_PORT` and `DAPR_GRPC_PORT` fields. |
| 133 | + |
| 134 | +- If no field is present, the environment variables of the current shell which executes the CLI command is passed to the app and dir. |
| 135 | +- if `env` field is present in the `common` section, in addition to the shell environment variables, the `env` map defined will be passed to all `apps` and `daprd` sidecars. |
| 136 | +- if `env` field is present only in a particular `apps` section, any shell environment variables, `env` maps from `common` section and the `env` map for the current app will be passed to both the `app` and `daprd`. |
| 137 | +- The more specific `env` key-value pairs will override the lesser specific ones i.e. `apps` section specific `env` key-value pairs will override the key-value pairs from the `common` section which will override the passed in shell environment variables. |
| 138 | + |
| 139 | + |
| 140 | +For each app in the `apps` section, the `resources_dir` and `config_file` values will be resolved in the following order: |
| 141 | + |
| 142 | +- If `reosurce_dir` and/or `config_file` fields are given for any `apps[i]` configuration use that value as is. |
| 143 | +- If not, check for `.dapr` folder in the **`apps[i].app_dir`** folder. If found, use the `.dapr/resources` folder for configuring the resources and `.dapr/config.yaml` for the `daprd` configuration file(argument `--config-file` in `daprd`). |
| 144 | +- If not, check if a `resources_dir` and/or `config_file` fields are defined in the `common` section of the compose configuration file. If so use those values for those fields. |
| 145 | +- If not, default to `~/.dapr/resources/` for `resources_dir` and `~/.dapr/config.yaml` for `config_file` values. |
| 146 | + |
| 147 | + |
| 148 | +## Proposed command format |
| 149 | + |
| 150 | +Given the run configuration file defined above, there should be a way to use the configuration file and run the differnt commands and `daprd` with the configuration given in the file. |
| 151 | + |
| 152 | +For this there will be a flag `-f, --file` that will be defined in the `dapr run` command. If the input path is a `file`, it expectes the file to have [structure defined above](#proposed-structure-for-run-configuration-file). |
| 153 | +If the path for the flag is a directory, then it expects the `dapr.yaml` file to be present in the directory with the same [structure defined above](#proposed-structure-for-run-configuration-file). |
| 154 | + |
| 155 | +### Interaction flow |
| 156 | + |
| 157 | +The interaction flow for the `dapr run -f <path>` is shown as below. |
| 158 | + |
| 159 | + |
| 160 | + |
| 161 | +> Note: app-id needs to be unique across all applications that have been run using `dapr run`. |
| 162 | +### Logging options |
| 163 | + |
| 164 | +Right `dapr run` executes as a foreground interactive process, and both the `daprd` logs and associated application logs are directly written to the STDOUT of the `dapr run` _process shell_ and it is not stored anywhere. |
| 165 | + |
| 166 | +Considering that executing `dapr run -f <path>` will run multiple applciations, routing all the logs to STDOUT for all applications and `daprd` processes will make the STDOUT completely chaotic and the user will be overwhelmed with log output. |
| 167 | + |
| 168 | +For example, consider two applications `order-proc` and `checkout` that are run on executing `dapr run -f <path>` and logs are routed to STDOUT: |
| 169 | + |
| 170 | +``` |
| 171 | +<order-proc> ==APP== waiting for daprd to start |
| 172 | +<order-proc> INFO[0000] enabled gRPC tracing middleware app_id=order-proc instance=Mukundans-MacBook-Pro.local scope=dapr.runtime.grpc.api type=log ver=1.9.3 |
| 173 | +<checkout> ==APP==pinging dapr API |
| 174 | +<checkout> INFO[0000] enabled gRPC tracing middleware app_id=checkout instance=Mukundans-MacBook-Pro.local scope=dapr.runtime.grpc.api type=log ver=1.9.3 |
| 175 | +<order-proc> INFO[0000] started daprd app_id=order-proc instance=Mukundans-MacBook-Pro.local scope=dapr.runtime.grpc.api type=log ver=1.9.3 |
| 176 | +<order-proc> ==APP== starting the application |
| 177 | +<order-proc> ==APP== processing request 1 |
| 178 | +<order-proc> ==APP== processing request 2 |
| 179 | +<checkout> INFO[0000] started daprd app_id=checkout instance=Mukundans-MacBook-Pro.local scope=dapr.runtime.grpc.api type=log ver=1.9.3 |
| 180 | +<checkout> ==APP== processing request 2 |
| 181 | +<order-proc> ==APP== processing request 3. request 3 calls dapr API |
| 182 | +<order-proc> INFO[0000] requst 3 is calls dapr API app_id=order-proc instance=Mukundans-MacBook-Pro.local scope=dapr.runtime.grpc.api type=log ver=1.9.3 |
| 183 | +<checkout> ==APP== processing request 3 |
| 184 | +
|
| 185 | +``` |
| 186 | + |
| 187 | +The logs will be as shown above, it will be chaotic to see what application is in which state and so on. |
| 188 | + |
| 189 | +Instead of writing to STDOUT, each application and associated `daprd` process will write the logs to `{application dir}/.dapr/logs/{app id}/app_{datetime}.log`, `{application dir}/.dapr/logs/{app id}/daprd_{datetime}.log`. |
| 190 | + |
| 191 | +## Feature lifecycle outline |
| 192 | + |
| 193 | +Compatibility with `dapr run` is expected to be maintained. But in certain cases there might be introduction of new behavior which might be _opt-in_ for running individual applications using `dapr run` whereas it might be _on by default_ when `dapr run -f <path>` is used. |
| 194 | + |
| 195 | +The expectation is for this feature to be refined and stabilized over a series of releases. |
| 196 | + |
| 197 | +### Recommendation for initial version |
| 198 | + |
| 199 | +- Initial implementation will only support Linux OS. |
| 200 | +- `dapr run -f <path>` will be an interactive process running completely for the lifecycle of applications, on exiting the process, all other spawned processes will also quit. |
| 201 | +- logs will be written to a predefined location. Users will need to manually tail the file. (Present in app-dir) |
| 202 | +- [optional] change the full `dapr run` command itself to honor [proposed organization for dapr projects](#proposed-structure-for-organizing-dapr-projects-locally). If not only when `-f` flag is used will the proposed organizing structure be used. |
| 203 | + |
| 204 | +### Changes for future releases |
| 205 | + |
| 206 | +- Extend support for Windows and macOS. |
| 207 | +- Extend `dapr run` to have a detached mode `-d, --detach` flag. This will also be honoroed when running multiple applications using the `-f` flag. |
| 208 | +- Add support for `dapr logs` to query from saved logs when the application is being run. |
| 209 | + |
| 210 | + |
| 211 | +## Completion checklist |
| 212 | +For initial version |
| 213 | +- [ ] Implement initail version of `dapr run -f <path>` feature |
| 214 | + - [ ] Add E2E tests for this feature |
| 215 | +- [ ] Add documenteation for this feature |
| 216 | + |
| 217 | +For later |
| 218 | +- [ ] Enahance `dapr run` command to have a detached mode |
| 219 | +- [ ] Enahance `dapr logs` command to track and output logs in self-hosted mode |
0 commit comments