11---
22description: Whenever touching the backend
3- globs: *backend*
3+ globs: *backend*,*.go*
4+ alwaysApply: false
45---
5- # Backend Architecture Overview
6-
7- ## Core Components
8-
9- ### Runner
10- - Main orchestrator that manages module lifecycle
11- - Initializes storage (S3) and database (ClickHouse)
12- - Registers and starts all enabled modules
13- - Handles graceful shutdown via signal handlers
14-
15- ### StateManager
16- - Generic key-value store for module/processor state
17- - Persists state to S3 as JSON
18- - Each module/processor manages its own state format
19- - Periodic flushing to S3 with configurable interval
20- - No type constraints - values must be JSON-serializable
21-
22- ### Module System
23- 1. Module
24- - Top-level component that groups related processors
25- - Has its own configuration section
26- - Manages processor lifecycles
27- - Example modules: beacon_chain_timings, xatu_public_contributors
28-
29- 2. Processor
30- - Handles specific data processing tasks
31- - Manages its own state under its name
32- - Updates state to prevent reprocessing
33-
34- ## Adding New Functionality
35-
36- ### Creating a New Module
37- 1. Create module directory: `backend/lab/modules/your_module_name/`
38- 2. Create files:
39- - `__init__.py` - Exports module class
40- - `module.py` - Module implementation
41- - `config.py` - Module configuration
42- - `models.py` - Data models
43- - `processors/` - Directory for processors
44-
45- 3. Module Configuration (config.py):
46- ```python
47- from pydantic import BaseModel, Field
48-
49- class YourModuleConfig(BaseModel):
50- enabled: bool = Field(default=True)
51- networks: List[str] = Field(default=["mainnet"])
52- # Add your module-specific configuration here
53- ```
54-
55- 4. Module Implementation (module.py):
56- ```python
57- from lab.core.module import Module, ModuleContext
58-
59- class YourModule(Module):
60- def __init__(self, ctx: ModuleContext):
61- super().__init__(ctx)
62- self._processors = {
63- "processor_name": YourProcessor(ctx)
64- }
65- self._tasks: Dict[str, asyncio.Task] = {}
66-
67- @property
68- def name(self) -> str:
69- return "your_module_name"
70-
71- async def start(self) -> None:
72- for name, processor in self._processors.items():
73- self._tasks[name] = asyncio.create_task(
74- self._run_processor(name, processor)
75- )
76-
77- async def stop(self) -> None:
78- await super().stop()
79- for task in self._tasks.values():
80- task.cancel()
81- ```
82-
83- ### Creating a New Processor
84- 1. Create processor file: `processors/your_processor.py`
85- 2. Implement processor:
86- ```python
87- from .base import BaseProcessor
88-
89- class YourProcessor(BaseProcessor):
90- def __init__(self, ctx: ModuleContext):
91- super().__init__(ctx, "processor_name")
92-
93- async def process(self) -> None:
94- if not await self.should_process():
95- return
96-
97- # Your processing logic here
98- await self.update_last_processed()
99- ```
100-
101- ### State Management
102- - Each processor gets its own state key in the state store
103- - Basic state format:
104- ```json
105- {
106- "last_processed": 0 // Unix timestamp
107- }
108- ```
109- - State is automatically initialized if not exists
110- - Use `should_process()` to check processing needs
111- - Always update state after successful processing
112-
113- ### Best Practices
114- 1. Error Handling
115- - Catch and log exceptions at processor level
116- - Continue processing on error
117- - Use descriptive error messages
118-
119- 2. Logging
120- - Use structured logging with context
121- - Log at appropriate levels (debug/info/warning/error)
122- - Include relevant metrics (counts, durations)
123-
124- 3. State Management
125- - Keep state minimal and JSON-serializable
126- - Update state after successful processing
127- - Validate state format on load
128-
129- 4. Performance
130- - Implement efficient database queries
131- - Process only what's needed using state checks
132-
133- 5. Configuration
134- - Use type hints and validation
135- - Provide sensible defaults
136- - Document configuration options
137-
138- ## Example Module Structure
139- ```
140- backend/lab/modules/your_module/
141- ├── __init__.py
142- ├── config.py
143- ├── models.py
144- ├── module.py
145- └── processors/
146- ├── __init__.py
147- ├── base.py
148- └── your_processor.py
149- ```
6+ # Golang Best Pracices
7+
8+ When developing the Go codebase, you must adhere to industry standard/best practices for backend and Golang.
9+
10+ ## Libraries
11+ Use the following libraries:
12+ - sirupsen/logrus for logging
13+
14+ ## Structure
15+ - Only use interfaces when absolutely required, or if they're beneficial for testing purposes. Structs should have a clearly defined
16+ seperation of concerns, and be small and testable.
17+ - When you create a new struct, interface, or feature, you should create tests in an equivalent side file.
18+ - E.g. if you create 'store.go', also create 'store_test.go'
19+ - It is VERY important to not stutter in your package and structure naming. For example:
20+ - 'service/store/store.go.Store' - BAD
21+ - 'service/store.go.Store' - GOOD
22+ - NEVER create packages that hold abstract contents. Definitions should live close to their related structs.
23+ - 'package/config/store.go' - DOGSHIT
24+ - 'package/store/{store.go, config.go}' - GOOD
25+ - 'utils/' - DOGSHIT. NEVER do this.
26+ - 'types/' - DOGSHIT. NEVER do this.
0 commit comments