Skip to content

Commit 1c0e8ae

Browse files
laurentftechlaurentftech
committed
chore: cleanup project and add comprehensive CI/CD
## 🧹 Cleanup - Removed 66 temporary files (cache, coverage, etc.) - Deleted obsolete meilisearchcrawler/ directory - Updated .gitignore with test/coverage patterns - Added typesense_data/ to gitignore ## 🚀 CI/CD - Added GitHub Actions workflows for tests and Docker builds - Multi-architecture Docker builds (amd64 + arm64) - Auto-publish to GitHub Container Registry - Added CI/CD badges to README ## 📝 Documentation - Added MIT License - Created CI/CD documentation - Created cleanup and setup guides - Added workflow README ## 🛠️ Tools - Created cleanup script (scripts/cleanup.sh) - Created release script (scripts/release.sh) - Added Makefile for common tasks - Created .dockerignore for optimized builds ## 🐛 Fixes - Fixed API search: score normalization (Typesense → 0-1) - Fixed API search: port configuration (8082) - Fixed API search: language filter ("all" option) - Fixed embeddings: corrected service URLs - Added embedding/reranking env vars to docker-compose ## ✨ Features - Dashboard: Collection recreation UI with cache cleanup - Dashboard: "all" language option for API Monitor - Reranking: Now fully functional (0.3-1ms) - Embeddings: Service connected (384D vectors) ## 📊 Tests - 67/70 tests passing (95.7%) - Coverage reporting configured - Linting and type checking ready
1 parent 7386e05 commit 1c0e8ae

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

52 files changed

+9356
-0
lines changed

‎.dockerignore‎

Lines changed: 69 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,69 @@
1+
# Git
2+
.git
3+
.gitignore
4+
.gitattributes
5+
.github
6+
7+
# Python
8+
__pycache__
9+
*.py[cod]
10+
*$py.class
11+
*.so
12+
.Python
13+
*.egg-info
14+
dist
15+
build
16+
.venv
17+
venv
18+
env
19+
ENV
20+
21+
# Testing
22+
.pytest_cache
23+
.coverage
24+
coverage.xml
25+
htmlcov
26+
.tox
27+
.mypy_cache
28+
.ruff_cache
29+
tests
30+
31+
# IDEs
32+
.vscode
33+
.idea
34+
*.swp
35+
*.swo
36+
*~
37+
38+
# OS
39+
.DS_Store
40+
Thumbs.db
41+
42+
# Data and logs
43+
data/*
44+
!data/.gitkeep
45+
typesense_data/*
46+
models_cache/*
47+
*.log
48+
*.db
49+
*.db-journal
50+
51+
# Docker
52+
docker-compose.override.yml
53+
.dockerignore
54+
Dockerfile
55+
56+
# Documentation
57+
README.md
58+
README-fr.md
59+
docs/*
60+
*.md
61+
62+
# Environment
63+
.env
64+
.env.*
65+
!.env.example
66+
67+
# Misc
68+
.coverage
69+
*.orig

‎.github/workflows/README.md‎

Lines changed: 154 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,154 @@
1+
# GitHub Actions Workflows
2+
3+
Ce répertoire contient les workflows GitHub Actions pour le projet KidSearch.
4+
5+
## Workflows disponibles
6+
7+
### 1. Tests (`tests.yml`)
8+
9+
**Déclenché par:**
10+
- Push sur `main` ou `develop`
11+
- Pull requests vers `main` ou `develop`
12+
- Manuellement via workflow_dispatch
13+
14+
**Actions:**
15+
- Run tests sur Python 3.10, 3.11 et 3.12
16+
- Vérification du code avec `ruff` (linter)
17+
- Vérification de types avec `mypy`
18+
- Génération de rapports de couverture
19+
- Upload des résultats vers Codecov
20+
21+
**Badge de statut:**
22+
```markdown
23+
![Tests](https://github.com/VOTRE_USERNAME/MeilisearchCrawler/actions/workflows/tests.yml/badge.svg)
24+
```
25+
26+
### 2. Docker Build and Publish (`docker-build.yml`)
27+
28+
**Déclenché par:**
29+
- Push sur `main` ou `develop`
30+
- Tags `v*` (ex: `v1.0.0`)
31+
- Pull requests
32+
- Manuellement via workflow_dispatch
33+
34+
**Actions:**
35+
1. **Test Job:**
36+
- Run tous les tests
37+
- Upload rapport de couverture
38+
39+
2. **Build and Push Job:**
40+
- Build l'image Docker multi-architecture (amd64, arm64)
41+
- Push vers GitHub Container Registry (ghcr.io)
42+
- Génération automatique de tags:
43+
- `latest` (branche main uniquement)
44+
- `main`, `develop` (selon la branche)
45+
- `v1.0.0`, `v1.0`, `v1` (pour les tags semver)
46+
- `main-abc123` (SHA du commit)
47+
48+
**Badge de statut:**
49+
```markdown
50+
![Docker](https://github.com/VOTRE_USERNAME/MeilisearchCrawler/actions/workflows/docker-build.yml/badge.svg)
51+
```
52+
53+
## Configuration requise
54+
55+
### Secrets GitHub
56+
57+
Aucun secret requis pour les fonctionnalités de base. Les secrets suivants sont optionnels:
58+
59+
- `CODECOV_TOKEN`: Pour uploader les rapports de couverture vers Codecov (optionnel)
60+
61+
### Permissions
62+
63+
Le workflow Docker nécessite les permissions suivantes (automatiquement configurées):
64+
- `contents: read` - Lire le code
65+
- `packages: write` - Publier sur GitHub Container Registry
66+
67+
## Utilisation
68+
69+
### Publier une nouvelle version
70+
71+
1. **Via Tag Git:**
72+
```bash
73+
git tag -a v1.0.0 -m "Release v1.0.0"
74+
git push origin v1.0.0
75+
```
76+
77+
Cela va créer les images:
78+
- `ghcr.io/VOTRE_USERNAME/meilisearchcrawler:v1.0.0`
79+
- `ghcr.io/VOTRE_USERNAME/meilisearchcrawler:v1.0`
80+
- `ghcr.io/VOTRE_USERNAME/meilisearchcrawler:v1`
81+
- `ghcr.io/VOTRE_USERNAME/meilisearchcrawler:latest` (si sur main)
82+
83+
2. **Build manuel:**
84+
- Aller sur Actions → Docker Build and Publish
85+
- Cliquer sur "Run workflow"
86+
- Sélectionner la branche
87+
- Cliquer sur "Run workflow"
88+
89+
### Utiliser l'image publiée
90+
91+
```yaml
92+
# docker-compose.yml
93+
services:
94+
kidsearch-all:
95+
image: ghcr.io/VOTRE_USERNAME/meilisearchcrawler:latest
96+
# ... reste de la configuration
97+
```
98+
99+
Ou avec une version spécifique:
100+
```yaml
101+
services:
102+
kidsearch-all:
103+
image: ghcr.io/VOTRE_USERNAME/meilisearchcrawler:v1.0.0
104+
```
105+
106+
### Pull l'image
107+
108+
```bash
109+
# Dernière version
110+
docker pull ghcr.io/VOTRE_USERNAME/meilisearchcrawler:latest
111+
112+
# Version spécifique
113+
docker pull ghcr.io/VOTRE_USERNAME/meilisearchcrawler:v1.0.0
114+
```
115+
116+
## Cache Docker
117+
118+
Le workflow utilise GitHub Actions cache pour accélérer les builds:
119+
- Cache des layers Docker entre les builds
120+
- Réutilisation des dépendances pip installées
121+
122+
## Multi-architecture
123+
124+
Les images sont buildées pour:
125+
- `linux/amd64` (Intel/AMD 64-bit)
126+
- `linux/arm64` (ARM 64-bit, ex: Apple Silicon, Raspberry Pi 4)
127+
128+
Docker sélectionne automatiquement la bonne architecture lors du pull.
129+
130+
## Troubleshooting
131+
132+
### Les tests échouent
133+
134+
1. Vérifier les logs du workflow
135+
2. Exécuter les tests localement:
136+
```bash
137+
python -m pytest tests/ -v
138+
```
139+
140+
### L'image Docker ne se build pas
141+
142+
1. Vérifier que le Dockerfile est valide
143+
2. Tester le build localement:
144+
```bash
145+
docker build -t test .
146+
```
147+
148+
### Impossible de pull l'image
149+
150+
1. Vérifier que le package est public (Settings → Packages)
151+
2. S'authentifier si le package est privé:
152+
```bash
153+
echo $GITHUB_TOKEN | docker login ghcr.io -u USERNAME --password-stdin
154+
```

‎.github/workflows/docker-build.yml‎

Lines changed: 112 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,112 @@
1+
name: Docker Build and Publish
2+
3+
on:
4+
push:
5+
branches:
6+
- main
7+
- develop
8+
tags:
9+
- 'v*'
10+
pull_request:
11+
branches:
12+
- main
13+
- develop
14+
workflow_dispatch:
15+
16+
env:
17+
REGISTRY: ghcr.io
18+
IMAGE_NAME: ${{ github.repository }}
19+
20+
jobs:
21+
test:
22+
name: Run Tests
23+
runs-on: ubuntu-latest
24+
25+
steps:
26+
- name: Checkout code
27+
uses: actions/checkout@v4
28+
29+
- name: Set up Python
30+
uses: actions/setup-python@v5
31+
with:
32+
python-version: '3.11'
33+
cache: 'pip'
34+
35+
- name: Install dependencies
36+
run: |
37+
python -m pip install --upgrade pip
38+
pip install -r requirements.txt
39+
pip install -r tests/requirements-test.txt
40+
41+
- name: Run tests with pytest
42+
run: |
43+
python -m pytest tests/ -v --cov=kidsearch --cov-report=xml --cov-report=term
44+
continue-on-error: true
45+
46+
- name: Upload coverage reports
47+
uses: codecov/codecov-action@v4
48+
if: github.event_name == 'push'
49+
with:
50+
file: ./coverage.xml
51+
flags: unittests
52+
name: codecov-umbrella
53+
env:
54+
CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
55+
continue-on-error: true
56+
57+
build-and-push:
58+
name: Build and Push Docker Image
59+
runs-on: ubuntu-latest
60+
needs: test
61+
permissions:
62+
contents: read
63+
packages: write
64+
65+
steps:
66+
- name: Checkout code
67+
uses: actions/checkout@v4
68+
69+
- name: Set up Docker Buildx
70+
uses: docker/setup-buildx-action@v3
71+
72+
- name: Log in to GitHub Container Registry
73+
if: github.event_name != 'pull_request'
74+
uses: docker/login-action@v3
75+
with:
76+
registry: ${{ env.REGISTRY }}
77+
username: ${{ github.actor }}
78+
password: ${{ secrets.GITHUB_TOKEN }}
79+
80+
- name: Extract metadata (tags, labels)
81+
id: meta
82+
uses: docker/metadata-action@v5
83+
with:
84+
images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
85+
tags: |
86+
type=ref,event=branch
87+
type=ref,event=pr
88+
type=semver,pattern={{version}}
89+
type=semver,pattern={{major}}.{{minor}}
90+
type=semver,pattern={{major}}
91+
type=sha,prefix={{branch}}-
92+
type=raw,value=latest,enable={{is_default_branch}}
93+
94+
- name: Build and push Docker image
95+
uses: docker/build-push-action@v5
96+
with:
97+
context: .
98+
file: ./Dockerfile
99+
push: ${{ github.event_name != 'pull_request' }}
100+
tags: ${{ steps.meta.outputs.tags }}
101+
labels: ${{ steps.meta.outputs.labels }}
102+
cache-from: type=gha
103+
cache-to: type=gha,mode=max
104+
platforms: linux/amd64,linux/arm64
105+
106+
- name: Generate artifact attestation
107+
if: github.event_name != 'pull_request'
108+
uses: actions/attest-build-provenance@v1
109+
with:
110+
subject-name: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
111+
subject-digest: ${{ steps.meta.outputs.digest }}
112+
push-to-registry: true

0 commit comments

Comments
 (0)