Skip to content

Commit 2b1ce82

Browse files
artczartcz
committed
add some basic docs
1 parent ae9ffce commit 2b1ce82

File tree

3 files changed

+304
-4
lines changed

3 files changed

+304
-4
lines changed

‎README.md‎

Lines changed: 68 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -50,7 +50,16 @@ The bot's main goal is to make life easier for the conference teams by streamlin
5050
* Managed with `uv`. All Makefiles, Docker images, and related configs are set up to use `uv`.
5151

5252
* **CI/CD**
53-
* Built with GitHub Actions.
53+
* Built with GitHub Actions
54+
* Runs tests and linters on every push to pull requests
55+
* Automatically deploys to production when code is merged to `main`
56+
57+
## Documentation
58+
59+
Check the `docs/` directory for additional information:
60+
61+
* [Architecture Overview](docs/architecture.md) - Basic system design and integration workflow
62+
* [Deployment Guide](docs/deployment.md) - Server setup and deployment process
5463

5564
## Local Development
5665

@@ -71,13 +80,68 @@ $ pyenv install 3.12
7180

7281
## Contributing
7382

74-
...
83+
### Setting Up Development Environment
84+
85+
1. Clone the repository
86+
2. Install `uv` - all the dependencies and Makefile targets are using `uv`.
87+
3. Set up environment variables in `.env` file
88+
4. Run the database with `docker-compose up -d`
89+
5. Apply migrations with `make migrate`
90+
6. Run the development server with `make server`
91+
7. Run the bot with `make bot`
92+
8. Run the worker with `make worker`
93+
94+
You can check the Django admin at http://localhost:4672/admin/ to see webhooks and tasks.
95+
96+
### Adding a New Integration
97+
98+
1. Create a module in `intbot/core/integrations/`
99+
2. Define Pydantic models to organize the data
100+
3. Add a webhook endpoint in `core/endpoints/webhooks.py`
101+
4. Add security checks (signature verification)
102+
5. Update `process_webhook` in `core/tasks.py`
103+
6. If it will send Discord messages, add routing logic in `channel_router.py`
104+
7. Add the new URL in `urls.py`
105+
106+
### Testing
107+
108+
The project uses pytest with Django:
109+
- Run all tests with `make test`
110+
- Run single tests with `make test/k K=your_keyword`
111+
- Run fast tests with `make test/fast`
112+
- Check test coverage with `make test/cov`
113+
114+
When testing webhooks, make sure to test:
115+
- Security checks (signature verification)
116+
- Data parsing
117+
- Channel routing (if using Discord)
118+
- Message formatting (if using Discord)
119+
120+
### Code Quality
121+
122+
We use ruff and mypy to lint and format the project.
123+
Both of those are run on CI for every Pull Request.
124+
125+
- Use type hints for all functions and classes
126+
- Run `make format` before committing
127+
- Run `make lint` and `make type-check` to check for issues
128+
- Follow the same error handling patterns as existing code
75129

76130
## Operations
77131

78-
...
132+
TODO: Expand on this part :)
133+
134+
### Monitoring
135+
136+
- Currently not configured
137+
138+
### Debugging
139+
140+
- Logs can be viewed on the intbot_user with `make logs`
141+
- The Django admin interface is available at `/admin/`
79142

80143
## Deployment
81144

82-
...
145+
Currently deployed to a separate VPS, using ansible (for both provisioning and deployment).
83146

147+
See deployment doc for more details: [Deployment Guide](docs/deployment.md)

‎docs/architecture.md‎

Lines changed: 90 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,90 @@
1+
# Internal Bot Architecture
2+
3+
## Overview
4+
5+
The internal-bot helps EuroPython Society teams communicate better. Right now it connects GitHub and Zammad with Discord, but it's built to do more in the future.
6+
7+
The app has three main parts:
8+
1. **Django Web App**: Receives webhooks, provides an admin panel, and will support more features later
9+
2. **Discord Bot**: Sends messages to Discord and responds to commands
10+
3. **Background Worker**: Handles tasks in the background without blocking the web app
11+
12+
## System Architecture
13+
14+
```
15+
┌─────────────────┐ ┌──────────────────┐ ┌────────────────┐
16+
│ External │ │ │ │ │
17+
│ Services │────▶│ Django App │────▶│ Discord │
18+
│ (GitHub, │ │ (Webhook API) │ │ Channels │
19+
│ Zammad) │ │ │ │ │
20+
└─────────────────┘ └──────────────────┘ └────────────────┘
21+
│ ▲
22+
▼ │
23+
┌──────────────────┐
24+
│ │
25+
│ Database │
26+
│ (PostgreSQL) │
27+
│ │
28+
└──────────────────┘
29+
```
30+
31+
### Data Flow
32+
33+
1. External services send webhooks to our app
34+
2. We verify and save these webhooks to the database
35+
3. Our background worker processes these webhooks:
36+
- For some webhooks (like GitHub), we need to fetch more data to make them useful
37+
- We then turn the webhook data into a format we can use
38+
4. If a Discord message needs to be sent, the channel router picks the right channel
39+
5. Discord messages are saved to the database
40+
6. The Discord bot checks for new messages and sends them
41+
42+
### Using the Admin Panel
43+
44+
The Django Admin panel lets you:
45+
- See all webhooks and filter them by type or date
46+
- Look at the raw webhook data
47+
- Check if tasks worked or failed
48+
- Manually trigger processing for webhooks
49+
- View and manage Discord messages
50+
51+
## Key Components
52+
53+
### Models
54+
55+
- **Webhook**: Stores webhook data including source, event type, and content
56+
- **DiscordMessage**: Represents a message to be sent to Discord
57+
- **Task**: (django-tasks) Stores background task execution history
58+
59+
### Integrations
60+
61+
#### GitHub Integration
62+
- Receives webhooks at `/webhooks/github/`
63+
- Verifies signatures using HMAC-SHA256
64+
- Currently handles project item events and issues
65+
- Routes messages based on project ID or repository
66+
67+
**Why GitHub Needs Extra Steps:**
68+
1. **The Problem**: GitHub webhooks often just contain IDs, not useful information
69+
2. **Getting More Data**:
70+
- We need to ask GitHub's GraphQL API for details like item names and descriptions
71+
- Without this extra step, notifications would just say "Item 12345 was moved" instead of what actually happened
72+
3. **How It Works**:
73+
- First, we save the webhook and get the extra data from GitHub
74+
- Then, we process it into a readable message
75+
- Finally, we send it to the right Discord channel
76+
77+
This approach lets us send helpful messages with real information instead of just ID numbers.
78+
79+
#### Zammad Integration
80+
- Receives webhooks at `/webhooks/zammad/`
81+
- Verifies signatures using HMAC-SHA1
82+
- Processes ticket and article information
83+
- Figures out what happened (new ticket, reply to ticket)
84+
- Routes messages based on ticket group (billing, helpdesk)
85+
86+
### Channel Router
87+
- Decides which Discord channel should receive messages
88+
- Uses different routing rules for each source (GitHub, Zammad)
89+
- Channel mappings are set in configuration
90+
- Can be extended for new message sources in the future

‎docs/deployment.md‎

Lines changed: 146 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,146 @@
1+
# Deployment Guide
2+
3+
This document explains how to deploy the internal-bot to a production environment.
4+
5+
## Overview
6+
7+
The deployment process uses:
8+
- Docker for containerization
9+
- Docker Compose for container orchestration
10+
- Ansible for automation
11+
- Nginx for web server and SSL termination
12+
13+
The application is deployed as several containers:
14+
- Django web app (handles webhooks)
15+
- Discord bot (sends messages to Discord)
16+
- Background worker (processes tasks)
17+
- PostgreSQL database
18+
19+
## Prerequisites
20+
21+
- A server running Ubuntu
22+
- SSH access to the server
23+
- Domain name pointing to the server
24+
- uv installed on your local machine (it will automatically download and install ansible, if you run it from the `make deploy/*` targets.
25+
26+
## Deployment Process
27+
28+
The deployment is done in three stages using Ansible playbooks:
29+
30+
### 1. Server Setup
31+
32+
```bash
33+
make deploy/provision
34+
```
35+
36+
This runs the first two playbooks:
37+
- `01_setup.yml`: Sets up server, installs Docker, creates users (nginx_user and intbot_user)
38+
- `02_nginx.yml`: Configures Nginx with SSL certificates
39+
40+
### 2. Application Deployment
41+
42+
```bash
43+
make deploy/app
44+
```
45+
46+
This runs the `03_app.yml` playbook which:
47+
- Builds Docker images
48+
- Sets up environment variables
49+
- Creates Docker Compose configuration
50+
- Runs database migrations
51+
- Starts all services
52+
53+
## Ansible Templates and Separate User Environments
54+
55+
The deployment uses a separated approach with different users for different responsibilities:
56+
57+
### Separate Users and Docker Compose Files
58+
59+
1. **Nginx Environment** (managed by `nginx_user`):
60+
- Uses its own Docker Compose file generated from `docker-compose.nginx.yml.j2`
61+
- Handles SSL termination and proxying
62+
- Has access to port 80/443 for web traffic
63+
64+
2. **Application Environment** (managed by `intbot_user`):
65+
- Uses its own Docker Compose file generated from `docker-compose.app.yml.j2`
66+
- Runs Django app, Discord bot, worker, and database
67+
- Doesn't need direct public internet access
68+
69+
Both environments are connected via a shared Docker network called "shared_with_nginx_network". This architecture provides several benefits:
70+
- **Security**: Each component runs with minimal required permissions
71+
- **Access Control**: Different teams can have access to different parts (some only to app, some to both)
72+
- **Separation of Concerns**: Nginx configuration changes don't affect the application
73+
- **Maintenance**: Either component can be updated independently
74+
75+
### Custom Makefiles for Each Environment
76+
77+
Ansible generates two specialized Makefiles:
78+
79+
1. **Nginx Makefile** (`Makefile.nginx.j2`):
80+
- Focused on SSL certificate management
81+
- Key targets:
82+
- `certbot/init-staging`: Set up staging certificates (for testing)
83+
- `certbot/upgrade-to-prod`: Upgrade to production certificates
84+
- `certbot/renew`: Renew existing certificates
85+
- `certbot/force-reissue-PROD-certificate`: Force reissue production certificates
86+
87+
2. **Application Makefile** (`Makefile.app.j2`):
88+
- Focused on application management
89+
- All commands use the `prod/` prefix
90+
- Key targets:
91+
- `prod/migrate`: Run database migrations
92+
- `prod/shell`: Access Django shell
93+
- `prod/db_shell`: Access database shell
94+
- `prod/manage`: Run Django management commands
95+
- `logs`: View application logs
96+
97+
## Version Control
98+
99+
Deployments are tied to specific Git commits:
100+
101+
```bash
102+
# Deploy specific version
103+
make deploy/app V=abcd1234
104+
```
105+
106+
If no version is specified, the current Git commit hash is used.
107+
108+
## Environment Variables
109+
110+
The application needs these environment variables in `intbot.env`:
111+
112+
- `DJANGO_SECRET_KEY`: Secret key for Django
113+
- `DJANGO_ALLOWED_HOSTS`: Comma-separated list of allowed hosts
114+
- `DATABASE_URL`: PostgreSQL connection string
115+
- `DISCORD_BOT_TOKEN`: Discord bot authentication token
116+
- `DISCORD_GUILD_ID`: Discord server ID
117+
- `GITHUB_WEBHOOK_SECRET`: Secret for GitHub webhook verification
118+
- `GITHUB_TOKEN`: GitHub API token
119+
- `ZAMMAD_WEBHOOK_SECRET`: Secret for Zammad webhook verification
120+
121+
An example file is available at `deploy/templates/app/intbot.env.example`.
122+
123+
## Monitoring
124+
125+
- Logs can be viewed with `docker compose logs`
126+
- The Django admin interface is available at `/admin/`
127+
- Server monitoring should be set up separately (not included)
128+
129+
## Troubleshooting
130+
131+
Common issues:
132+
- **Webhook verification failures**: Check secret keys in environment variables
133+
- **Database connection errors**: Verify DATABASE_URL is correct
134+
- **Discord messages not being sent**: Check DISCORD_BOT_TOKEN and permissions
135+
136+
For more detailed logs:
137+
```bash
138+
docker compose logs -f bot
139+
docker compose logs -f web
140+
docker compose logs -f worker
141+
```
142+
143+
Or even simpler if you want to see all of them at once
144+
```bash
145+
make logs
146+
```

0 commit comments

Comments
 (0)