Skip to content

Commit 215c6e0

Browse files
GabrielSallaGabrielSalla
committed
update how to run docs
1 parent 0938286 commit 215c6e0

File tree

2 files changed

+123
-66
lines changed

2 files changed

+123
-66
lines changed

‎docs/configuration.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,10 +3,10 @@ The basic configs are set through the `configs.yaml` file. This file is read whe
33

44
The monitors path is also defined in the `configs.yaml` file. By default, it's set to the `sample_monitors` folder, but it can be changed to another folder if desired. The `configs.yaml` file also have other configurations that can be adjusted.
55

6+
# Environment variables
67
> [!IMPORTANT]
78
> Check the documentation for the plugins that are being used to see if they have environment variables of their own.
89
9-
# Environment variables
1010
## `CONFIGS_FILE`
1111
The application will try to load the configs file through the path defined in the `CONFIGS_FILE` environment variable. If this variable is not defined, it'll look for the file in the root directory of the application.
1212

‎docs/how_to_run.md‎

Lines changed: 122 additions & 65 deletions
Original file line numberDiff line numberDiff line change
@@ -1,68 +1,125 @@
1-
# How to run
2-
## Development execution
3-
Development execution should be used when developing or testing the platform features. It's not intended to be used to develop monitors as it might set variables that might interfere with the monitors execution.
4-
5-
To run the application in development mode, use the following command.
6-
```shell
7-
make run-dev
8-
```
9-
10-
For testing purposes, sometimes it's useful to start a container and be able to execute multiple commands without needing to create a new container every time. This is useful when testing features, for example, as it's possible to run `pytest` for a selected list of tests. To start a container in `shell` mode, use the following command.
11-
```shell
12-
# Migrate the database
13-
# Only necessary when it's the first time executing or if there're new versions
14-
make migrate-dev
15-
16-
# Start the application container in shell mode
17-
make run-shell-dev
18-
```
19-
20-
To run tests, linter and type checking for the whole project, use the following commands, respectively.
21-
```shell
22-
make test-dev
23-
make linter
24-
make mypy
25-
```
26-
27-
## Local execution
28-
Local execution should be used when developing monitors and should not be used in production.
29-
30-
For local development, set the secrets in the `.env.secrets`, as specified in the [Configs and secrets](#configs-and-secrets) section.
31-
32-
When running the application locally, it's recommended to use the internal queue instead of the SQS queue for a faster and more fluid process, but it's possible to use the AWS mock or a real SQS queue.
33-
34-
To start the **controller** and a single **executor** (with concurrent internal executors) in the same container, run the following command.
35-
```shell
36-
# Only necessary when it's the first time executing or if there're new versions
37-
make migrate-local
38-
39-
# Start the application
40-
make run-local
41-
```
42-
43-
This will start the database and the application.
44-
45-
## Production deployment
46-
### Building the image
47-
The [Dockerfile](../Dockerfile) is a starting point for building the application image. This file implements the logic to install all the enabled plugins dependencies correctly.
48-
49-
```shell
50-
# Install the dependencies for the application and enabled plugins
51-
poetry install --no-root --only $(python ./tools/get_plugins_list.py)
52-
```
53-
54-
### Deploying the application
55-
In production deployment, it's recommended that the controller and executors to be deployed in different containers or pods (in case of a Kubernetes deployment). With this method a SQS queue is required to allow the controller communicate with the executors.
1+
# How to Run
2+
3+
## Development Execution
4+
Development execution should be used when developing or testing the platform features. **It is not intended for developing monitors**, as it might set variables that interfere with monitor execution.
5+
6+
1. Migrate the database to the latest version. This is only necessary when running for the first time or after updates.
7+
```shell
8+
make migrate-dev
9+
```
10+
2. Run the application in development mode.
11+
```shell
12+
make run-dev
13+
```
14+
15+
For testing purposes, it can be useful to start a container and execute multiple commands without creating a new container each time. This is helpful when testing features, such as running `pytest` for a selected list of tests.
16+
17+
1. Start the application container in shell mode.
18+
```shell
19+
make run-shell-dev
20+
```
21+
22+
The quality checks can also be run.
23+
1. Run the tests.
24+
```shell
25+
make test-dev
26+
```
27+
2. Run the linter (ruff).
28+
```shell
29+
make linter
30+
```
31+
3. Run the type checker (mypy).
32+
```shell
33+
make mypy
34+
```
35+
36+
## Local Execution - Single Container
37+
Local execution is recommended for developing monitors. It should not be used in production but provides a quick way to experiment with the application before committing to a more complex deployment.
38+
39+
**Pros**:
40+
- Quick to set up and run with the included `docker-compose` and config files.
41+
- Easy to create and modify monitors from the host machine.
42+
- No need for external services like a database or queues, as everything runs locally.
43+
44+
**Cons**:
45+
- Difficult to scale horizontally.
46+
- Creating and modifying monitors from outside the host can be cumbersome, as the client must connect to the controller.
47+
- Hard to monitor the application.
48+
- Running the database locally increases the risk of data loss.
49+
50+
When running the application locally, it is recommended to use the internal queue instead of the SQS queue for faster and smoother operation. However, it is also possible to use the AWS queue mock or a real SQS queue.
51+
52+
1. Set the secrets in the `.env.secrets` file, as specified in the [Configuration](./configuration.md) documentation.
53+
2. Migrate the database to the latest version. This is only necessary when running for the first time or after updates.
54+
```shell
55+
make migrate-local
56+
```
57+
3. Start the application. The local database will also be started.
58+
```shell
59+
make run-local
60+
```
61+
62+
## Local Execution - Multiple Containers
63+
For a more scalable deployment, it is recommended to use separate containers for the controller and executor. This approach allows running multiple executor containers, increasing concurrency when processing a high number of monitors. This setup can still run on a single machine without requiring external services.
64+
65+
**Pros**:
66+
- Quick to set up and run with the included `docker-compose` and config files.
67+
- Easy to create and modify monitors from the host machine.
68+
- Easy to scale horizontally by increasing the number of executor replicas.
69+
- No need for external services like a database or queues, as everything runs locally.
70+
71+
**Cons**:
72+
- A local container for the queue is required if not using an external queue. One is already included in the `docker-compose-scalable.yml` file.
73+
- Creating and modifying monitors from outside the host can be cumbersome, as the client must connect to the controller.
74+
- Hard to monitor the application.
75+
- Running the database locally increases the risk of data loss.
76+
77+
1. Set the `replicas` parameter in the `docker-compose-scalable.yml` file to the desired number of executors.
78+
2. Set the secrets in the `.env.secrets` file, as specified in the [Configuration](./configuration.md) documentation.
79+
3. Migrate the database to the latest version. This is only necessary when running for the first time or after updates.
80+
```shell
81+
make migrate-scalable
82+
```
83+
4. Start a container for the controller and executors.
84+
```shell
85+
make run-scalable
86+
```
87+
88+
## Production Deployment
89+
For production deployment, it is recommended to use a more complex setup with multiple containers or pods (in the case of a Kubernetes deployment) to allow for better resource management and scaling. This setup requires a database, a message queue, and the application itself.
90+
91+
**Pros**:
92+
- Allows horizontal scaling of executors by adding more containers or pods.
93+
- Persistent storage with an external database.
94+
- Enables monitoring the application with external tools.
95+
- Allows creating and modifying monitors from any client that can connect to the controller.
96+
97+
**Cons**:
98+
- More complex to set up and maintain.
99+
- Creating and modifying monitors can be more complicated, as a connection to the controller is required.
100+
- Requires an external database and message queue.
101+
102+
### Building the Image
103+
The [Dockerfile](../Dockerfile) is a starting point for building the application image. This file implements the logic to install all dependencies for the enabled plugins.
104+
105+
1. Install the dependencies for the application and enabled plugins.
106+
```shell
107+
poetry install --no-root --only $(python ./tools/get_plugins_list.py)
108+
```
109+
110+
### Deploying the Application
111+
In production deployment, it is recommended to deploy the controller and executors in separate containers or pods (in the case of a Kubernetes deployment). This method requires an external queue to allow communication between the controller and executors. A persistent database is also recommended to prevent data loss.
56112

57113
The files provided in the [Kubernetes template](../resources/kubernetes_template) directory can be used as a reference for a Kubernetes deployment.
58114

59-
All services must have the environment variables set as specified in the [Configs and secrets](#configs-and-secrets) section.
60-
61-
Controllers and executors can be started specifying them as parameters.
62-
```shell
63-
# Controller
64-
python3 src/main.py controller
65-
66-
# Executor
67-
python3 src/main.py executor
68-
```
115+
All services must have the environment variables set as specified in the [Configuration](./configuration.md) documentation.
116+
117+
Controllers and executors can be run by specifying them as parameters when starting the application:
118+
1. Run the controller.
119+
```shell
120+
python3 src/main.py controller
121+
```
122+
2. Run the executor.
123+
```shell
124+
python3 src/main.py executor
125+
```

0 commit comments

Comments
 (0)