Skip to content

Commit 614446c

Browse files
vincentbergmanclaude
vincentbergman
and
claude
committed
feat: separate agent and simunet logs with module-based filtering
Implement log separation to write agent and simunet logs to different files, preventing log duplication in test environments where both services run in the same process. Changes: - Add CLAUDE.md documentation for future Claude Code instances - Enhance configure_logman() with module_filter and exclude_filter parameters - Support multiple log handlers with independent filters - Add _logger_initialized flag to prevent handler removal on subsequent calls - Use logger.patch() in MockSSHDevice to force correct module identification - Configure separate log files in agent.yml and simunet.yml - Update test conftest.py to configure both handlers with proper filters Log routing: - logs/agent.log: agent, plugin, client, and other non-server modules - logs/simunet.log: only netdriver.server.* modules Technical details: - Loguru's record.name is inferred from call stack, not logger definition - logger.patch() modifies record to set explicit module name - exclude_filter prioritized over module_filter for correct separation - Test environment handles both services in single process without duplication 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>
1 parent 6dc61b2 commit 614446c

File tree

8 files changed

+314
-25
lines changed

8 files changed

+314
-25
lines changed

‎CLAUDE.md‎

Lines changed: 184 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,184 @@
1+
# CLAUDE.md
2+
3+
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
4+
5+
## Project Overview
6+
7+
NetDriver is a network device automation framework built on AsyncSSH that provides HTTP RESTful APIs for CLI command execution on network devices. It's organized as a monorepo with a Polylith architecture, consisting of:
8+
9+
- **netdriver-agent**: FastAPI-based REST API service for device connectivity testing and command execution
10+
- **netdriver-simunet**: SSH server simulation for testing network device terminals
11+
12+
## Key Architecture Patterns
13+
14+
### Polylith Architecture
15+
16+
The project uses a Polylith-inspired structure with `bases/` (applications) and `components/` (shared libraries):
17+
18+
```Text
19+
bases/netdriver/
20+
├── agent/ # REST API application
21+
└── simunet/ # Simulation network application
22+
23+
components/netdriver/
24+
├── client/ # SSH client with async session management
25+
├── exception/ # Centralized error handling and error codes
26+
├── log/ # Logging utilities (Loguru-based)
27+
├── plugin/ # Plugin system core and engine
28+
├── plugins/ # Device-specific plugins (Cisco, Huawei, Juniper, etc.)
29+
├── server/ # SSH server for simulated devices
30+
├── textfsm/ # Enhanced TextFSM for output parsing
31+
└── utils/ # Utility functions
32+
```
33+
34+
### Session Management
35+
36+
The `SessionPool` (in `components/netdriver/client/pool.py`) is a singleton that:
37+
38+
- Maintains persistent SSH sessions with devices (identified by `protocol:username@ip:port`)
39+
- Automatically monitors session health and removes closed/expired/idle sessions
40+
- Implements a command queue per session to prevent concurrent configuration conflicts
41+
- Uses asyncio locks to ensure thread-safe session operations
42+
43+
### Plugin System
44+
45+
The plugin engine (`components/netdriver/plugin/engine.py`) dynamically loads device plugins at startup:
46+
47+
- Plugins are organized by vendor directory under `components/netdriver/plugins/`
48+
- Each plugin inherits from `Base` (in `components/netdriver/plugins/base.py`) and implements device-specific behavior
49+
- Plugins define mode patterns (LOGIN, ENABLE, CONFIG), error patterns, and command handling
50+
- Plugin resolution: `vendor/model/version` → `vendor/model/base` → `vendor/base/base`
51+
52+
### Device Modes and State Management
53+
54+
Sessions track device state including:
55+
56+
- **Mode**: LOGIN, ENABLE, or CONFIG (defined in `client/mode.py`)
57+
- **Vsys**: Virtual system context (for multi-context devices like firewalls)
58+
- Mode switching is handled automatically by the base plugin via `switch_mode()`
59+
60+
## Commands
61+
62+
### Development Environment
63+
64+
```bash
65+
# Install dependencies
66+
poetry install
67+
68+
# Install Poetry plugins (if not already installed)
69+
poetry self add poetry-multiproject-plugin
70+
poetry self add poetry-polylith-plugin
71+
```
72+
73+
### Running Services
74+
75+
```bash
76+
# Start agent service (REST API on http://localhost:8000)
77+
poetry run agent
78+
79+
# Start simulation network service (SSH servers on configured ports)
80+
poetry run simunet
81+
```
82+
83+
### Testing
84+
85+
```bash
86+
# Run all tests
87+
poetry run pytest
88+
89+
# Run unit tests only
90+
poetry run pytest -m unit
91+
92+
# Run integration tests only
93+
poetry run pytest -m integration
94+
95+
# Run specific test file
96+
poetry run pytest tests/bases/netdriver/agent/test_cisco_nexus.py
97+
```
98+
99+
### Configuration
100+
101+
Configuration files in `config/`:
102+
103+
- `config/agent/agent.yml` - Agent service settings (logging, session timeouts, SSH parameters, profiles)
104+
- Logs are written to `logs/netdriver_agent.log`
105+
- `config/simunet/simunet.yml` - Simulated device definitions and logging settings
106+
- Logs are written to `logs/netdriver_simunet.log`
107+
108+
## Development Guidelines
109+
110+
### Adding a New Device Plugin
111+
112+
1. Create vendor directory under `components/netdriver/plugins/` if it doesn't exist
113+
2. Create plugin file named `{vendor}_{model}.py` (e.g., `cisco_nexus.py`)
114+
3. Inherit from vendor base class or `Base` plugin
115+
4. Define `PluginInfo` with vendor, model, version, and description
116+
5. Implement required abstract methods:
117+
- `get_mode_prompt_patterns()` - Regex patterns for each mode's prompt
118+
- `get_more_pattern()` - Pattern for pagination prompts
119+
- `get_union_pattern()` - Combined pattern for all prompts
120+
- `get_error_patterns()` - Patterns that indicate command errors
121+
- `get_ignore_error_patterns()` - Error patterns to ignore
122+
123+
Example:
124+
125+
```python
126+
from netdriver.plugin.plugin_info import PluginInfo
127+
from netdriver.plugins.cisco import CiscoBase
128+
129+
class CiscoNexus(CiscoBase):
130+
info = PluginInfo(
131+
vendor="cisco",
132+
model="nexus",
133+
version="base",
134+
description="Cisco Nexus Plugin"
135+
)
136+
```
137+
138+
### Testing Plugins
139+
140+
Integration tests are in `tests/bases/netdriver/agent/` and typically:
141+
142+
1. Start the simunet service with test fixtures in `conftest.py`
143+
2. Use httpx client to make API calls to the agent
144+
3. Verify command execution and error handling
145+
146+
### Error Handling
147+
148+
All custom exceptions are in `components/netdriver/exception/errors.py` and inherit from `BaseError`:
149+
150+
- Include HTTP status code and error code
151+
- For command execution errors, include output
152+
- Examples: `LoginFailed`, `PluginNotFound`, `ExecCmdTimeout`, `EnableFailed`
153+
154+
### Dependency Injection
155+
156+
The agent uses `dependency-injector` (see `bases/netdriver/agent/containers.py`) to wire:
157+
158+
- Configuration providers
159+
- Request handlers
160+
- The container is wired to API modules in `main.py`
161+
162+
### Logging
163+
164+
Uses Loguru configured via `netdriver.log.logman`:
165+
166+
- Correlation ID middleware tracks requests (agent only)
167+
- Log levels configurable in respective config files
168+
- Log files are separated by service:
169+
- Agent: `logs/agent.log` (excludes `netdriver.server` modules in test environment)
170+
- Simunet: `logs/simunet.log` (only `netdriver.server` modules)
171+
- Intercepts uvicorn logs for unified output
172+
- Log rotation: 1 day, retention: 60 days
173+
- Module filtering: Uses `logger.patch()` in `netdriver.server.device` to ensure correct module identification
174+
- In test environment: Both handlers are configured to prevent log duplication
175+
176+
## Important Notes
177+
178+
- Python 3.12+ required
179+
- Uses Poetry for dependency management
180+
- All SSH operations are async (AsyncSSH-based)
181+
- Session keys format: `{protocol}:{username}@{ip}:{port}`
182+
- The agent runs with auto-reload enabled by default (suitable for development)
183+
- Simulated devices use the plugin system to emulate vendor-specific behavior
184+
- Configuration profiles support device-specific settings by vendor/model/version or IP address

‎bases/netdriver/agent/main.py‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,8 @@
1919

2020

2121
logman.configure_logman(level=container.config.logging.level(),
22-
intercept_loggers=container.config.logging.intercept_loggers())
22+
intercept_loggers=container.config.logging.intercept_loggers(),
23+
log_file=container.config.logging.log_file())
2324
log = logman.logger
2425
container.wire(modules=[
2526
rest.v1.api,

‎bases/netdriver/simunet/main.py‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,9 @@
1111
from netdriver.simunet.containers import container
1212

1313

14+
logman.configure_logman(level=container.config.logging.level(),
15+
intercept_loggers=container.config.logging.intercept_loggers(),
16+
log_file=container.config.logging.log_file())
1417
log = logman.logger
1518
app = FastAPI()
1619

‎components/netdriver/log/logman.py‎

Lines changed: 63 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -72,26 +72,78 @@ def emit(self, record: logging.LogRecord) -> None:
7272
logger.opt(depth=depth, exception=record.exc_info).log(level, record.getMessage())
7373

7474

75-
def configure_logman(level: str = "INFO", intercept_loggers: List[str] = []):
76-
""" Config loguru """
75+
# Track if logger has been initialized
76+
_logger_initialized = False
77+
78+
def configure_logman(level: str = "INFO", intercept_loggers: List[str] = [], log_file: str = "logs/netdriver_agent.log",
79+
module_filter: str = None, exclude_filter: str = None):
80+
""" Config loguru
81+
82+
Args:
83+
level: Log level (INFO, DEBUG, TRACE)
84+
intercept_loggers: List of logger names to intercept
85+
log_file: Path to log file
86+
module_filter: Module name pattern to filter (e.g., "netdriver.agent", "netdriver.server")
87+
If None, all logs are written to the file
88+
exclude_filter: Module name pattern to exclude (e.g., "netdriver.server")
89+
Logs from modules starting with this pattern will be excluded
90+
91+
Note: This function can be called multiple times to add multiple log files with different filters.
92+
On first call, it removes all existing handlers and sets up base configuration.
93+
On subsequent calls, it only adds new handlers without removing existing ones.
94+
"""
95+
global _logger_initialized
96+
97+
# Only remove handlers on first call
98+
if not _logger_initialized:
99+
logger.remove()
100+
_logger_initialized = True
77101

78-
logger.remove()
79102
intercept_handler = InterceptHandler()
80103

81104
_all_loggers = logging.root.manager.loggerDict
82105
for name, _logger in _all_loggers.items():
83106
if name in intercept_loggers:
84-
_logger.handlers = [intercept_handler]
107+
if intercept_handler not in _logger.handlers:
108+
_logger.handlers = [intercept_handler]
109+
110+
# Create filter function based on module_filter and exclude_filter
111+
def log_filter(record):
112+
# First check exclude filter
113+
if exclude_filter and record["name"].startswith(exclude_filter):
114+
return False
115+
# Then check include filter
116+
if module_filter is None:
117+
return True
118+
# Filter based on module name
119+
return record["name"].startswith(module_filter)
120+
121+
# Add log file handler with optional module filter
122+
handler_id = logger.add(
123+
log_file,
124+
rotation="1 days",
125+
retention="60 days",
126+
colorize=False,
127+
enqueue=True,
128+
backtrace=True,
129+
diagnose=True,
130+
level=level,
131+
format=format_record,
132+
filter=log_filter
133+
)
134+
135+
# Configure standard logging only once
136+
if not logging.root.handlers or all(isinstance(h, InterceptHandler) for h in logging.root.handlers):
137+
# because the logging dose not support TRACE level, we use DEBUG instead
138+
if level == "TRACE":
139+
logging.basicConfig(handlers=[intercept_handler], level="DEBUG", force=True)
140+
else:
141+
logging.basicConfig(handlers=[intercept_handler], level=level, force=True)
85142

86-
logger.add("logs/netdriver_agent.log", rotation="1 days", retention="60 days", colorize=False,
87-
enqueue=True, backtrace=True, diagnose=True, level=level, format=format_record)
88-
# because the logging dose not support TRACE level, we use DEBUG instead
89-
if level == "TRACE":
90-
logging.basicConfig(handlers=[intercept_handler], level="DEBUG", force=True)
91-
else:
92-
logging.basicConfig(handlers=[intercept_handler], level=level, force=True)
93143
logger.configure(patcher=set_log_extras)
94144

145+
return handler_id
146+
95147

96148
def create_session_logger(session_key: str, level: str = "INFO"):
97149
"""Create a logger for a specific session."""

‎components/netdriver/server/device.py‎

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,10 +11,18 @@
1111
from netdriver.server.user_repo import UserRepo
1212

1313

14+
# Module-level logger with patched name for correct filtering
15+
def _patch_record(record):
16+
"""Patch the record to ensure it's identified as netdriver.server"""
17+
record["name"] = __name__ # __name__ will be "netdriver.server.device"
18+
19+
log = logman.logger.patch(_patch_record)
20+
21+
1422
class MockSSHDevice(asyncssh.SSHServer):
1523
""" Create mock SSH-device """
1624
_server: asyncssh.SSHAcceptor
17-
_logger = logman.logger
25+
_logger = log
1826
_handlers = List[CommandHandler]
1927

2028
vendor: str

‎config/agent/agent.yml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,8 @@ api:
66
logging:
77
# INFO / DEBUG / TRACE
88
level: INFO
9+
# Log file path for agent
10+
log_file: logs/agent.log
911
intercept_loggers:
1012
- uvicorn.access
1113
- uvicorn.error

‎config/simunet/simunet.yml‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,12 @@
1+
logging:
2+
# INFO / DEBUG / TRACE
3+
level: INFO
4+
# Log file path for simunet
5+
log_file: logs/simunet.log
6+
intercept_loggers:
7+
- uvicorn.access
8+
- uvicorn.error
9+
110
devices:
211
- vendor: cisco
312
model: nexus

0 commit comments

Comments
 (0)