|
5 | 5 | <img src="assets/logo-light.svg" alt="ICICLE Insights Logo" width="700"/> |
6 | 6 | </picture> |
7 | 7 |
|
8 | | - <p><strong>A high-performance C++23 HTTP server for collecting and storing metrics about ICICLE project components</strong></p> |
| 8 | + <p><strong>A C++23 HTTP server for collecting and storing metrics about ICICLE project components</strong></p> |
9 | 9 |
|
10 | 10 | <p> |
11 | | - <a href="#features">Features</a> • |
| 11 | + <a href="#overview">Overview</a> • |
12 | 12 | <a href="#quick-start">Quick Start</a> • |
13 | | - <a href="#api-endpoints">API</a> • |
| 13 | + <a href="#api">API</a> • |
| 14 | + <a href="#deployment">Deployment</a> • |
14 | 15 | <a href="#documentation">Documentation</a> |
15 | 16 | </p> |
16 | 17 | </div> |
|
19 | 20 |
|
20 | 21 | ## Overview |
21 | 22 |
|
22 | | -ICICLE Insights tracks repositories, accounts, and platform-level statistics across various platforms including Git hosting services (GitHub, GitLab) and container registries (DockerHub, Quay.io). |
| 23 | +ICICLE Insights collects and stores metrics for the [ICICLE](https://icicle.osu.edu/) research project — tracking repositories and accounts across Git hosting platforms (GitHub) and container registries. |
23 | 24 |
|
24 | | -The system monitors metrics across the ICICLE research project ecosystem: |
| 25 | +Metrics tracked include: **stars, forks, clones, views, watchers, followers**. |
25 | 26 |
|
26 | | -- **Git Platforms**: stars, forks, clones, views, watchers, followers |
27 | | -- **Container Registries**: pulls, stars |
28 | | - |
29 | | -The project tracks ICICLE components across different research thrusts: |
30 | | -- AI4CI (AI for Cyberinfrastructure) |
31 | | -- FoundationAI |
32 | | -- CI4AI (Cyberinfrastructure for AI) |
33 | | -- Software |
34 | | -- Use-inspired domains (Digital Agriculture, Smart Foodsheds, Animal Ecology) |
35 | | - |
36 | | -## Features |
37 | | - |
38 | | -- **Modern C++23** implementation with `std::expected` for error handling |
39 | | -- **Async I/O** using ASIO for high performance |
40 | | -- **Background task scheduler** for periodic metric collection from Git platforms |
41 | | -- **Type-safe database operations** with PostgreSQL via libpqxx |
42 | | -- **RESTful API** with JSON serialization using glaze |
43 | | -- **Generic CRUD operations** using templates and traits |
44 | | -- **Non-blocking architecture** with thread pool for background tasks |
45 | | -- **Docker support** for easy deployment |
| 27 | +The server exposes a REST API for querying stored data and runs a background sync task every two weeks to pull fresh statistics from platform APIs. |
46 | 28 |
|
47 | 29 | ## Prerequisites |
48 | 30 |
|
49 | 31 | - C++23 compiler (GCC 13+, Clang 17+, or Apple Clang 15+) |
50 | | -- [Conan 2.x](https://conan.io/) - Package manager |
| 32 | +- [Conan 2.x](https://conan.io/) |
51 | 33 | - [CMake](https://cmake.org/) 3.25+ |
52 | | -- [Ninja](https://ninja-build.org/) - Build system |
53 | | -- [just](https://just.systems/) - Command runner |
54 | | -- PostgreSQL - Database |
| 34 | +- [Ninja](https://ninja-build.org/) |
| 35 | +- [just](https://just.systems/) |
| 36 | +- PostgreSQL |
55 | 37 |
|
56 | 38 | ## Quick Start |
57 | 39 |
|
58 | | -### 1. Clone the repository |
59 | | - |
60 | 40 | ```bash |
61 | 41 | git clone https://github.com/guzman109/icicle-insights.git |
62 | 42 | cd icicle-insights |
63 | 43 | ``` |
64 | 44 |
|
65 | | -### 2. Set up environment variables |
66 | | - |
67 | 45 | Create a `.env` file: |
68 | 46 |
|
69 | 47 | ```bash |
70 | 48 | # Required |
71 | 49 | DATABASE_URL=postgresql://user:password@localhost:5432/icicle_insights |
72 | 50 | GITHUB_TOKEN=your_github_token |
73 | | -TAPIS_TOKEN=your_tapis_token |
74 | 51 |
|
75 | 52 | # Optional |
76 | | -HOST=127.0.0.1 # Server host (default: 127.0.0.1) |
77 | | -PORT=3000 # Server port (default: 3000) |
78 | | -LOG_LEVEL=info # Log level: trace, debug, info, warn, error (default: info) |
| 53 | +HOST=127.0.0.1 # default: 127.0.0.1 |
| 54 | +PORT=3000 # default: 3000 |
| 55 | +LOG_LEVEL=info # trace | debug | info | warn | error |
| 56 | +SSL_CERT_FILE= # path to CA bundle (macOS: /opt/homebrew/etc/ca-certificates/cert.pem) |
79 | 57 | ``` |
80 | 58 |
|
81 | | -### 3. Build and run |
| 59 | +Build and run: |
82 | 60 |
|
83 | 61 | ```bash |
84 | | -# Install dependencies, configure, build, and run |
85 | | -just full-build |
86 | | -just run |
87 | | - |
88 | | -# Or individually: |
89 | | -just deps # Install dependencies with Conan |
90 | | -just setup # Configure CMake |
91 | | -just build # Build the project |
92 | | -just run # Run the server |
| 62 | +just full-build # install deps, configure, build |
| 63 | +just run # start the server on http://localhost:3000 |
93 | 64 | ``` |
94 | 65 |
|
95 | | -The server will start on `http://localhost:3000`. |
96 | | - |
97 | | -The background task scheduler will automatically run every 2 weeks to update Git platform metrics. See [docs/async-task-patterns.md](docs/async-task-patterns.md) for details on the async architecture. |
98 | | - |
99 | | -## API Endpoints |
| 66 | +## API |
100 | 67 |
|
101 | 68 | ### Core |
102 | 69 |
|
103 | | -- `GET /health` - Health check endpoint (verifies database connectivity) |
104 | | -- `GET /routes` - Lists all available API endpoints |
| 70 | +| Method | Path | Description | |
| 71 | +|--------|------|-------------| |
| 72 | +| `GET` | `/health` | Database connectivity check | |
| 73 | +| `GET` | `/routes` | List all registered routes | |
105 | 74 |
|
106 | | -### Platforms |
| 75 | +### GitHub Accounts |
107 | 76 |
|
108 | | -- `GET /api/git/platforms` - Get all platforms |
109 | | -- `POST /api/git/platforms` - Create a platform |
110 | | -- `GET /api/git/platforms/:id` - Get platform by ID |
111 | | -- `PATCH /api/git/platforms/:id` - Update platform |
112 | | -- `DELETE /api/git/platforms/:id` - Delete platform |
| 77 | +| Method | Path | Description | |
| 78 | +|--------|------|-------------| |
| 79 | +| `GET` | `/api/github/accounts` | List all accounts | |
| 80 | +| `POST` | `/api/github/accounts` | Create an account | |
| 81 | +| `GET` | `/api/github/accounts/:id` | Get account by ID | |
| 82 | +| `DELETE` | `/api/github/accounts/:id` | Delete account | |
113 | 83 |
|
114 | | -### Accounts |
| 84 | +### GitHub Repositories |
115 | 85 |
|
116 | | -- `GET /api/git/accounts` - Get all accounts |
117 | | -- `POST /api/git/accounts` - Create an account |
118 | | -- `GET /api/git/accounts/:id` - Get account by ID |
119 | | -- `PATCH /api/git/accounts/:id` - Update account |
120 | | -- `DELETE /api/git/accounts/:id` - Delete account |
| 86 | +| Method | Path | Description | |
| 87 | +|--------|------|-------------| |
| 88 | +| `GET` | `/api/github/repos` | List all repositories | |
| 89 | +| `POST` | `/api/github/repos` | Create a repository | |
| 90 | +| `GET` | `/api/github/repos/:id` | Get repository by ID | |
| 91 | +| `PATCH` | `/api/github/repos/:id` | Update repository | |
| 92 | +| `DELETE` | `/api/github/repos/:id` | Delete repository | |
121 | 93 |
|
122 | | -### Repositories |
| 94 | +## Deployment |
123 | 95 |
|
124 | | -- `GET /api/git/repos` - Get all repositories |
125 | | -- `POST /api/git/repos` - Create a repository |
126 | | -- `GET /api/git/repos/:id` - Get repository by ID |
127 | | -- `PATCH /api/git/repos/:id` - Update repository |
128 | | -- `DELETE /api/git/repos/:id` - Delete repository |
| 96 | +The CI/CD pipeline (GitHub Actions) builds the binary on `ubuntu-24.04`, packages it into a Docker image, and pushes to GHCR. |
129 | 97 |
|
130 | | -## Development |
| 98 | +```bash |
| 99 | +# Pull and run the latest image |
| 100 | +docker pull ghcr.io/icicle-ai/insights:latest |
| 101 | +docker run -p 3000:3000 --env-file .env ghcr.io/icicle-ai/insights:latest |
| 102 | +``` |
131 | 103 |
|
132 | | -### Project Structure |
| 104 | +To publish a release, push a version tag: |
133 | 105 |
|
| 106 | +```bash |
| 107 | +git tag v1.0.0 && git push origin v1.0.0 |
134 | 108 | ``` |
135 | | -. |
136 | | -├── include/ # Header files |
137 | | -│ ├── core/ # Foundation utilities |
138 | | -│ ├── db/ # Database layer |
139 | | -│ ├── git/ # Git platform models, routing, and tasks |
140 | | -│ │ ├── models.hpp # Data models |
141 | | -│ │ ├── router.hpp # HTTP route handlers |
142 | | -│ │ ├── tasks.hpp # Background task pipeline |
143 | | -│ │ └── scheduler.hpp # Periodic task scheduler |
144 | | -│ ├── containers/ # Container registry models |
145 | | -│ └── server/ # HTTP server |
146 | | -├── src/ # Implementation files |
147 | | -│ ├── git/ # Git route handlers and task implementations |
148 | | -│ ├── server/ # Server implementation |
149 | | -│ └── insights.cpp # Entry point |
150 | | -├── docs/ # Documentation |
151 | | -├── tests/ # Test files (HTTP client tests) |
152 | | -└── data/ # Component data and fixtures |
| 109 | + |
| 110 | +This triggers the full pipeline: build → Docker image → GitHub Release with binary tarball. |
| 111 | + |
| 112 | +### Local Docker build |
| 113 | + |
| 114 | +```bash |
| 115 | +just docker-build # build image locally |
| 116 | +just docker-run # run with .env |
153 | 117 | ``` |
154 | 118 |
|
| 119 | +## Development |
| 120 | + |
155 | 121 | ### Common Commands |
156 | 122 |
|
157 | 123 | ```bash |
158 | | -just build # Build the project |
159 | | -just run # Run the server |
160 | | -just clean-build # Clean and rebuild from scratch |
161 | | -just test # Run tests (if configured) |
| 124 | +just deps # Install Conan dependencies |
| 125 | +just setup # Configure CMake |
| 126 | +just build # Compile |
| 127 | +just run # Run the server |
| 128 | +just full-build # deps + setup + build |
| 129 | +just clean-build # wipe build dir and rebuild |
| 130 | + |
| 131 | +just act-linux # Test the linux CI job locally with act |
| 132 | +just act-build # Test the docker CI job locally with act |
162 | 133 | ``` |
163 | 134 |
|
164 | | -### Code Style |
| 135 | +### Project Structure |
165 | 136 |
|
166 | | -The project follows LLVM naming conventions: |
167 | | -- **Types**: `PascalCase` (e.g., `Platform`, `HttpStatus`) |
168 | | -- **Variables**: `PascalCase` (e.g., `Database`, `Config`) |
169 | | -- **Functions**: `lowerCamelCase` (e.g., `registerRoutes()`, `initServer()`) |
170 | | -- **Enumerators**: `PascalCase` (e.g., `Ok`, `BadRequest`) |
| 137 | +``` |
| 138 | +. |
| 139 | +├── include/insights/ |
| 140 | +│ ├── core/ # Config, error handling, logging, scheduler, HTTP status |
| 141 | +│ ├── db/ # Database layer (Database struct, DbTraits, DbEntity concept) |
| 142 | +│ ├── github/ # GitHub models, routes, and sync tasks |
| 143 | +│ └── server/ # HTTP server setup and middleware |
| 144 | +├── src/ |
| 145 | +│ ├── core/ # Health check and route listing endpoints |
| 146 | +│ ├── github/ # Route handlers and sync task implementation |
| 147 | +│ └── insights.cpp # Entry point |
| 148 | +├── docs/ # Developer documentation |
| 149 | +├── .github/ |
| 150 | +│ └── workflows/ |
| 151 | +│ └── build.yaml # CI/CD: build → Docker → release |
| 152 | +├── Dockerfile # Runtime image (ubuntu:24.04) |
| 153 | +├── conanfile.txt # Conan dependencies |
| 154 | +├── CMakeLists.txt # Build configuration |
| 155 | +└── justfile # Build command runner |
| 156 | +``` |
171 | 157 |
|
172 | | -Code formatting is enforced via `.clang-format` and static analysis via `.clang-tidy`. |
| 158 | +### Code Style |
173 | 159 |
|
174 | | -## Documentation |
| 160 | +LLVM naming conventions throughout: |
175 | 161 |
|
176 | | -- [Architecture Guide](docs/architecture.md) - Detailed architecture and design decisions |
177 | | -- [Async Task Patterns](docs/async-task-patterns.md) - Background task scheduling and async patterns |
178 | | -- [CLAUDE.md](CLAUDE.md) - Project instructions for AI assistants and build system details |
| 162 | +| Construct | Convention | Example | |
| 163 | +|-----------|-----------|---------| |
| 164 | +| Types | `PascalCase` | `Platform`, `HttpStatus` | |
| 165 | +| Variables | `PascalCase` | `Database`, `Config` | |
| 166 | +| Functions | `lowerCamelCase` | `registerRoutes()`, `syncStats()` | |
| 167 | +| Enumerators | `PascalCase` | `Ok`, `BadRequest` | |
| 168 | +| Members | `PascalCase` | `.Id`, `.Name`, `.AccountId` | |
179 | 169 |
|
180 | 170 | ## Dependencies |
181 | 171 |
|
182 | 172 | | Package | Version | Purpose | |
183 | 173 | |---------|---------|---------| |
184 | | -| asio | 1.36.0 | Async I/O and networking | |
185 | | -| openssl | 3.6.0 | TLS/SSL support | |
186 | | -| glaze | 7.0.0 | JSON serialization and HTTP routing | |
187 | | -| libpq | 17.7 | PostgreSQL C client | |
| 174 | +| asio | 1.36.0 | Async I/O, timers, thread pool | |
| 175 | +| openssl | — | TLS for outbound HTTP requests | |
| 176 | +| glaze | main | HTTP server, router, JSON | |
188 | 177 | | libpqxx | 7.10.5 | PostgreSQL C++ client | |
189 | | -| spdlog | 1.17.0 | Logging | |
| 178 | +| spdlog | 1.17.0 | Structured logging | |
190 | 179 |
|
191 | | -## Docker |
192 | | - |
193 | | -Build and run with Docker: |
| 180 | +## Documentation |
194 | 181 |
|
195 | | -```bash |
196 | | -docker build -t icicle-insights . |
197 | | -docker run -p 3000:3000 --env-file .env icicle-insights |
198 | | -``` |
| 182 | +- [Architecture](docs/architecture.md) — module structure, core patterns, data model, design decisions |
| 183 | +- [Developer Guide](docs/README.md) — how to add routes, tasks, loggers; debugging tips |
199 | 184 |
|
200 | 185 | ## License |
201 | 186 |
|
202 | | -This project is licensed under the GNU General Public License v3.0 - see the [LICENSE](LICENSE) file for details. |
203 | | - |
204 | | -## Contributing |
205 | | - |
206 | | -[Add contribution guidelines here] |
| 187 | +GNU General Public License v3.0 — see [LICENSE](LICENSE). |
207 | 188 |
|
208 | 189 | ## Acknowledgments |
209 | 190 |
|
210 | | -This project is part of the [ICICLE (Intelligent Cyberinfrastructure with Computational Learning in the Environment)](https://icicle.osu.edu/) initiative. |
| 191 | +Part of the [ICICLE (Intelligent Cyberinfrastructure with Computational Learning in the Environment)](https://icicle.osu.edu/) initiative. |
0 commit comments