Skip to content

Commit a2b148b

Browse files
artczartcz
authored
Basic docs (#26)
1 parent 91c1bbe commit a2b148b

File tree

3 files changed

+357
-4
lines changed

3 files changed

+357
-4
lines changed

‎README.md

Lines changed: 82 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -50,7 +50,30 @@ The bot's main goal is to make life easier for the conference teams by streamlin
5050
* Managed with `uv`. All Makefiles, Docker images, and related configs are set up to use `uv`.
5151

5252
* **CI/CD**
53-
* Built with GitHub Actions.
53+
* Built with GitHub Actions
54+
* Runs tests and linters on every push to pull requests
55+
* Automatically deploys to production when code is merged to `main`
56+
57+
## Documentation
58+
59+
Check the `docs/` directory for additional information:
60+
61+
* [Architecture Overview](docs/architecture.md) - Basic system design and integration workflow
62+
* [Deployment Guide](docs/deployment.md) - Server setup and deployment process
63+
64+
## Project Structure
65+
66+
This project follows a simple, focused organization:
67+
68+
```
69+
deploy/ - Deployment configuration and Ansible playbooks
70+
docs/ - Documentation files
71+
intbot/ - Main application code
72+
├─ core/ - Core functionality (bot, integrations, endpoints)
73+
└─ tests/ - Test suite
74+
```
75+
76+
See the [Architecture Overview](docs/architecture.md) for more details.
5477

5578
## Local Development
5679

@@ -71,13 +94,68 @@ $ pyenv install 3.12
7194

7295
## Contributing
7396

74-
...
97+
### Setting Up Development Environment
98+
99+
1. Clone the repository
100+
2. Install `uv` - all the dependencies and Makefile targets are using `uv`.
101+
3. Set up environment variables in `.env` file
102+
4. Run the database with `docker-compose up -d`
103+
5. Apply migrations with `make migrate`
104+
6. Run the development server with `make server`
105+
7. Run the bot with `make bot`
106+
8. Run the worker with `make worker`
107+
108+
You can check the Django admin at http://localhost:4672/admin/ to see webhooks and tasks.
109+
110+
### Adding a New Integration
111+
112+
1. Create a module in `intbot/core/integrations/`
113+
2. Define Pydantic models to organize the data
114+
3. Add a webhook endpoint in `core/endpoints/webhooks.py`
115+
4. Add security checks (signature verification)
116+
5. Update `process_webhook` in `core/tasks.py`
117+
6. If it will send Discord messages, add routing logic in `channel_router.py`
118+
7. Add the new URL in `urls.py`
119+
120+
### Testing
121+
122+
The project uses pytest with Django:
123+
- Run all tests with `make test`
124+
- Run single tests with `make test/k K=your_keyword`
125+
- Run fast tests with `make test/fast`
126+
- Check test coverage with `make test/cov`
127+
128+
When testing webhooks, make sure to test:
129+
- Security checks (signature verification)
130+
- Data parsing
131+
- Channel routing (if using Discord)
132+
- Message formatting (if using Discord)
133+
134+
### Code Quality
135+
136+
We use ruff and mypy to lint and format the project.
137+
Both of those are run on CI for every Pull Request.
138+
139+
- Use type hints for all functions and classes
140+
- Run `make format` before committing
141+
- Run `make lint` and `make type-check` to check for issues
142+
- Follow the same error handling patterns as existing code
75143

76144
## Operations
77145

78-
...
146+
TODO: Expand on this part :)
147+
148+
### Monitoring
149+
150+
- Currently not configured
151+
152+
### Debugging
153+
154+
- Logs can be viewed on the intbot_user with `make logs`
155+
- The Django admin interface is available at `/admin/`
79156

80157
## Deployment
81158

82-
...
159+
Currently deployed to a separate VPS, using ansible (for both provisioning and deployment).
83160

161+
See deployment doc for more details: [Deployment Guide](docs/deployment.md)

‎docs/architecture.md

Lines changed: 129 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,129 @@
1+
# Internal Bot Architecture
2+
3+
## Overview
4+
5+
The internal-bot helps EuroPython Society teams communicate better. Right now it connects GitHub and Zammad with Discord, but it's built to do more in the future.
6+
7+
The app has three main parts:
8+
1. **Django Web App**: Receives webhooks, provides an admin panel, and will support more features later
9+
2. **Discord Bot**: Sends messages to Discord and responds to commands
10+
3. **Background Worker**: Handles tasks in the background without blocking the web app
11+
12+
Besides handling webhooks, the bot can also:
13+
- Respond to user commands (like `!ping`)
14+
- Answer questions in channels and threads
15+
- React to messages and reactions from users
16+
17+
The bot code and supported commands can be found in `intbot/core/bot/main.py`.
18+
19+
## Project Structure
20+
21+
The codebase is organized as follows:
22+
23+
```
24+
deploy/ # Deployment configuration
25+
├── playbooks/ # Ansible playbooks
26+
└── templates/ # Templates for Docker Compose and Makefiles
27+
28+
docs/ # Project documentation
29+
├── architecture.md # System design and structure
30+
└── deployment.md # Deployment guide
31+
32+
intbot/
33+
├── core/ # Main Django app with all the business logic
34+
│ ├── bot/ # Discord bot implementation
35+
│ ├── endpoints/ # API endpoints for webhooks
36+
│ ├── integrations/ # Integration modules (GitHub, Zammad)
37+
│ ├── management/ # Django management commands
38+
│ └── models.py # Database models
39+
├── intbot/ # Django project settings
40+
│ ├── settings.py # Configuration
41+
│ └── urls.py # URL routing
42+
└── tests/ # Test suite (mirrors the application structure)
43+
```
44+
45+
This structure was chosen because:
46+
- It's simple and has a single `core` app instead of many small apps
47+
- It clearly separates the integration logic from the bot logic
48+
- Tests mirror the application structure, making them easy to find
49+
- It supports multiple entry points (web server, bot, worker) from one codebase
50+
51+
## System Architecture
52+
53+
```
54+
┌─────────────────┐ ┌──────────────────┐ ┌────────────────┐
55+
│ External │ │ │ │ │
56+
│ Services │────▶│ Django App │────▶│ Discord │
57+
│ (GitHub, │ │ (Webhook API) │ │ Channels │
58+
│ Zammad) │ │ │ │ │
59+
└─────────────────┘ └──────────────────┘ └────────────────┘
60+
│ ▲
61+
▼ │
62+
┌──────────────────┐
63+
│ │
64+
│ Database │
65+
│ (PostgreSQL) │
66+
│ │
67+
└──────────────────┘
68+
```
69+
70+
### Data Flow
71+
72+
1. External services send webhooks to our app
73+
2. We verify and save these webhooks to the database
74+
3. Our background worker processes these webhooks:
75+
- For some webhooks (like GitHub), we need to fetch more data to make them useful
76+
- We then turn the webhook data into a format we can use
77+
4. If a Discord message needs to be sent, the channel router picks the right channel
78+
5. Discord messages are saved to the database
79+
6. The Discord bot checks for new messages and sends them
80+
81+
### Using the Admin Panel
82+
83+
The Django Admin panel lets you:
84+
- See all webhooks and filter them by type or date
85+
- Look at the raw webhook data
86+
- Check if tasks worked or failed
87+
- Manually trigger processing for webhooks
88+
- View and manage Discord messages
89+
90+
## Key Components
91+
92+
### Models
93+
94+
- **Webhook**: Stores webhook data including source, event type, and content
95+
- **DiscordMessage**: Represents a message to be sent to Discord
96+
- **Task**: (django-tasks) Stores background task execution history
97+
98+
### Integrations
99+
100+
#### GitHub Integration
101+
- Receives webhooks at `/webhooks/github/`
102+
- Verifies signatures using HMAC-SHA256
103+
- Currently handles project item events and issues
104+
- Routes messages based on project ID or repository
105+
106+
**Why GitHub Needs Extra Steps:**
107+
1. **The Problem**: GitHub webhooks often just contain IDs, not useful information
108+
2. **Getting More Data**:
109+
- We need to ask GitHub's GraphQL API for details like item names and descriptions
110+
- Without this extra step, notifications would just say "Item 12345 was moved" instead of what actually happened
111+
3. **How It Works**:
112+
- First, we save the webhook and get the extra data from GitHub
113+
- Then, we process it into a readable message
114+
- Finally, we send it to the right Discord channel
115+
116+
This approach lets us send helpful messages with real information instead of just ID numbers.
117+
118+
#### Zammad Integration
119+
- Receives webhooks at `/webhooks/zammad/`
120+
- Verifies signatures using HMAC-SHA1
121+
- Processes ticket and article information
122+
- Figures out what happened (new ticket, reply to ticket)
123+
- Routes messages based on ticket group (billing, helpdesk)
124+
125+
### Channel Router
126+
- Decides which Discord channel should receive messages
127+
- Uses different routing rules for each source (GitHub, Zammad)
128+
- Channel mappings are set in configuration
129+
- Can be extended for new message sources in the future

‎docs/deployment.md

Lines changed: 146 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,146 @@
1+
# Deployment Guide
2+
3+
This document explains how to deploy the internal-bot to a production environment.
4+
5+
## Overview
6+
7+
The deployment process uses:
8+
- Docker for containerization
9+
- Docker Compose for container orchestration
10+
- Ansible for automation
11+
- Nginx for web server and SSL termination
12+
13+
The application is deployed as several containers:
14+
- Django web app (handles webhooks)
15+
- Discord bot (sends messages to Discord)
16+
- Background worker (processes tasks)
17+
- PostgreSQL database
18+
19+
## Prerequisites
20+
21+
- A server running Ubuntu
22+
- SSH access to the server
23+
- Domain name pointing to the server
24+
- uv installed on your local machine (it will automatically download and install ansible, if you run it from the `make deploy/*` targets.
25+
26+
## Deployment Process
27+
28+
The deployment is done in three stages using Ansible playbooks:
29+
30+
### 1. Server Setup
31+
32+
```bash
33+
make deploy/provision
34+
```
35+
36+
This runs the first two playbooks:
37+
- `01_setup.yml`: Sets up server, installs Docker, creates users (nginx_user and intbot_user)
38+
- `02_nginx.yml`: Configures Nginx with SSL certificates
39+
40+
### 2. Application Deployment
41+
42+
```bash
43+
make deploy/app
44+
```
45+
46+
This runs the `03_app.yml` playbook which:
47+
- Builds Docker images
48+
- Sets up environment variables
49+
- Creates Docker Compose configuration
50+
- Runs database migrations
51+
- Starts all services
52+
53+
## Ansible Templates and Separate User Environments
54+
55+
The deployment uses a separated approach with different users for different responsibilities:
56+
57+
### Separate Users and Docker Compose Files
58+
59+
1. **Nginx Environment** (managed by `nginx_user`):
60+
- Uses its own Docker Compose file generated from `docker-compose.nginx.yml.j2`
61+
- Handles SSL termination and proxying
62+
- Has access to port 80/443 for web traffic
63+
64+
2. **Application Environment** (managed by `intbot_user`):
65+
- Uses its own Docker Compose file generated from `docker-compose.app.yml.j2`
66+
- Runs Django app, Discord bot, worker, and database
67+
- Doesn't need direct public internet access
68+
69+
Both environments are connected via a shared Docker network called "shared_with_nginx_network". This architecture provides several benefits:
70+
- **Security**: Each component runs with minimal required permissions
71+
- **Access Control**: Different teams can have access to different parts (some only to app, some to both)
72+
- **Separation of Concerns**: Nginx configuration changes don't affect the application
73+
- **Maintenance**: Either component can be updated independently
74+
75+
### Custom Makefiles for Each Environment
76+
77+
Ansible generates two specialized Makefiles:
78+
79+
1. **Nginx Makefile** (`Makefile.nginx.j2`):
80+
- Focused on SSL certificate management
81+
- Key targets:
82+
- `certbot/init-staging`: Set up staging certificates (for testing)
83+
- `certbot/upgrade-to-prod`: Upgrade to production certificates
84+
- `certbot/renew`: Renew existing certificates
85+
- `certbot/force-reissue-PROD-certificate`: Force reissue production certificates
86+
87+
2. **Application Makefile** (`Makefile.app.j2`):
88+
- Focused on application management
89+
- All commands use the `prod/` prefix
90+
- Key targets:
91+
- `prod/migrate`: Run database migrations
92+
- `prod/shell`: Access Django shell
93+
- `prod/db_shell`: Access database shell
94+
- `prod/manage`: Run Django management commands
95+
- `logs`: View application logs
96+
97+
## Version Control
98+
99+
Deployments are tied to specific Git commits:
100+
101+
```bash
102+
# Deploy specific version
103+
make deploy/app V=abcd1234
104+
```
105+
106+
If no version is specified, the current Git commit hash is used.
107+
108+
## Environment Variables
109+
110+
The application needs these environment variables in `intbot.env`:
111+
112+
- `DJANGO_SECRET_KEY`: Secret key for Django
113+
- `DJANGO_ALLOWED_HOSTS`: Comma-separated list of allowed hosts
114+
- `DATABASE_URL`: PostgreSQL connection string
115+
- `DISCORD_BOT_TOKEN`: Discord bot authentication token
116+
- `DISCORD_GUILD_ID`: Discord server ID
117+
- `GITHUB_WEBHOOK_SECRET`: Secret for GitHub webhook verification
118+
- `GITHUB_TOKEN`: GitHub API token
119+
- `ZAMMAD_WEBHOOK_SECRET`: Secret for Zammad webhook verification
120+
121+
An example file is available at `deploy/templates/app/intbot.env.example`.
122+
123+
## Monitoring
124+
125+
- Logs can be viewed with `docker compose logs`
126+
- The Django admin interface is available at `/admin/`
127+
- Server monitoring should be set up separately (not included)
128+
129+
## Troubleshooting
130+
131+
Common issues:
132+
- **Webhook verification failures**: Check secret keys in environment variables
133+
- **Database connection errors**: Verify DATABASE_URL is correct
134+
- **Discord messages not being sent**: Check DISCORD_BOT_TOKEN and permissions
135+
136+
For more detailed logs:
137+
```bash
138+
docker compose logs -f bot
139+
docker compose logs -f web
140+
docker compose logs -f worker
141+
```
142+
143+
Or even simpler if you want to see all of them at once
144+
```bash
145+
make logs
146+
```

0 commit comments

Comments
 (0)