Issues/364 Add Log File Support to MCP Gateway (#643)

* added log file with rotation

Signed-off-by: kumar-tiger <[email protected]>

* Fix logging service integration issues

- Add python-json-logger dependency to pyproject.toml
- Fix circular import by removing LoggingService from retry_manager.py
- Fix logging service lazy handler initialization
- Clean up unused logging imports
- All tests passing with 80% coverage

Co-Authored-By: Claude <[email protected]>
Signed-off-by: Mihai Criveti <[email protected]>

* Update documentation for logging service features

- Enhanced README.md logging section with comprehensive details
- Updated docs/manage/logging.md with file rotation and dual-format info
- Revised ADR-005 to reflect current implementation status
- Added log file management, debug mode, and troubleshooting guides
- Documented all new configuration options and usage examples

Co-Authored-By: Claude <[email protected]>
Signed-off-by: Mihai Criveti <[email protected]>

* Rebase and minor fixes

Signed-off-by: Mihai Criveti <[email protected]>

* Rebase and minor fixes

Signed-off-by: Mihai Criveti <[email protected]>

* Rebase and expand logging to other python files, make logging optional

Signed-off-by: Mihai Criveti <[email protected]>

* Configurable rotation

Signed-off-by: Mihai Criveti <[email protected]>

* Update .env.exaple

Signed-off-by: Mihai Criveti <[email protected]>

* Fix dual logging

Signed-off-by: Mihai Criveti <[email protected]>

* Update test coverage

Signed-off-by: Mihai Criveti <[email protected]>

---------

Signed-off-by: kumar-tiger <[email protected]>
Signed-off-by: Mihai Criveti <[email protected]>
Co-authored-by: Mihai Criveti <[email protected]>
Co-authored-by: Claude <[email protected]>

Author: 3 people

.env.example

@@ -171,8 +171,26 @@ LOG_LEVEL=INFO
 # Log format: json or text
 LOG_FORMAT=json
 
-# Optional path to write logs to file (leave empty to log to stdout)
-#LOG_FILE=./logs/gateway.log
+# Enable file logging (default: false - logs to stdout/stderr only)
+LOG_TO_FILE=false
+
+# Log filename (when file logging enabled)
+#LOG_FILE=gateway.log
+
+# Log directory (when file logging enabled)
+#LOG_FOLDER=./logs
+
+# File write mode: a+ (append) or w (overwrite)
+#LOG_FILEMODE=a+
+
+# Enable log rotation (default: false)
+#LOG_ROTATION_ENABLED=false
+
+# Max file size before rotation (MB)
+#LOG_MAX_SIZE_MB=1
+
+# Number of backup files to keep
+#LOG_BACKUP_COUNT=5
 
 #####################################
 # Transport Configuration

CLAUDE.md

@@ -139,8 +139,22 @@ MCPGATEWAY_ADMIN_API_ENABLED=true
 MCPGATEWAY_ENABLE_MDNS_DISCOVERY=true
 MCPGATEWAY_ENABLE_FEDERATION=true
 
-# Development
+# Logging (Dual Output Support)
 LOG_LEVEL=INFO
+LOG_TO_FILE=false  # Enable file logging (default: stdout/stderr only)
+LOG_ROTATION_ENABLED=false  # Enable log rotation when file logging is enabled
+LOG_MAX_SIZE_MB=1  # Max file size before rotation (MB)
+LOG_BACKUP_COUNT=5  # Number of backup files to keep
+LOG_FILE=mcpgateway.log  # Log filename (when file logging enabled)
+LOG_FOLDER=logs  # Log directory (when file logging enabled)
+
+# Dual Logging Features:
+# - Console logs: Human-readable text format (always enabled)
+# - File logs: Structured JSON format (when LOG_TO_FILE=true)
+# - All logs appear in both outputs: application, services, HTTP access logs
+# - File rotation: Configurable size-based rotation with backup retention
+
+# Development
 RELOAD=true  # For development hot-reload
 ```
 

README.md

@@ -1016,11 +1016,53 @@ You can get started by copying the provided [.env.example](.env.example) to `.en
 
 ### Logging
 
-| Setting      | Description       | Default | Options            |
-| ------------ | ----------------- | ------- | ------------------ |
-| `LOG_LEVEL`  | Minimum log level | `INFO`  | `DEBUG`...`CRITICAL` |
-| `LOG_FORMAT` | Log format        | `json`  | `json`, `text`     |
-| `LOG_FILE`   | Log output file   | (none)  | path or empty      |
+MCP Gateway provides flexible logging with **stdout/stderr output by default** and **optional file-based logging**. When file logging is enabled, it provides JSON formatting for structured logs and text formatting for console output.
+
+| Setting                 | Description                        | Default           | Options                    |
+| ----------------------- | ---------------------------------- | ----------------- | -------------------------- |
+| `LOG_LEVEL`             | Minimum log level                  | `INFO`            | `DEBUG`...`CRITICAL`       |
+| `LOG_FORMAT`            | Console log format                 | `json`            | `json`, `text`             |
+| `LOG_TO_FILE`           | **Enable file logging**            | **`false`**       | **`true`, `false`**        |
+| `LOG_FILE`              | Log filename (when enabled)        | `null`            | `mcpgateway.log`           |
+| `LOG_FOLDER`            | Directory for log files            | `null`            | `logs`, `/var/log/gateway` |
+| `LOG_FILEMODE`          | File write mode                    | `a+`              | `a+` (append), `w` (overwrite)|
+| `LOG_ROTATION_ENABLED`  | **Enable log file rotation**       | **`false`**       | **`true`, `false`**        |
+| `LOG_MAX_SIZE_MB`       | Max file size before rotation (MB) | `1`               | Any positive integer       |
+| `LOG_BACKUP_COUNT`      | Number of backup files to keep     | `5`               | Any non-negative integer   |
+
+**Logging Behavior:**
+- **Default**: Logs only to **stdout/stderr** with human-readable text format
+- **File Logging**: When `LOG_TO_FILE=true`, logs to **both** file (JSON format) and console (text format)
+- **Log Rotation**: When `LOG_ROTATION_ENABLED=true`, files rotate at `LOG_MAX_SIZE_MB` with `LOG_BACKUP_COUNT` backup files (e.g., `.log.1`, `.log.2`)
+- **Directory Creation**: Log folder is automatically created if it doesn't exist
+- **Centralized Service**: All modules use the unified `LoggingService` for consistent formatting
+
+**Example Configurations:**
+
+```bash
+# Default: stdout/stderr only (recommended for containers)
+LOG_LEVEL=INFO
+# No additional config needed - logs to stdout/stderr
+
+# Optional: Enable file logging (no rotation)
+LOG_TO_FILE=true
+LOG_FOLDER=/var/log/mcpgateway
+LOG_FILE=gateway.log
+LOG_FILEMODE=a+
+
+# Optional: Enable file logging with rotation
+LOG_TO_FILE=true
+LOG_ROTATION_ENABLED=true
+LOG_MAX_SIZE_MB=10
+LOG_BACKUP_COUNT=3
+LOG_FOLDER=/var/log/mcpgateway
+LOG_FILE=gateway.log
+```
+
+**Default Behavior:**
+- Logs are written **only to stdout/stderr** in human-readable text format
+- File logging is **disabled by default** (no files created)
+- Set `LOG_TO_FILE=true` to enable optional file logging with JSON format
 
 ### Transport
 

docs/docs/architecture/adr/005-structured-json-logging.md

@@ -1,7 +1,7 @@
 # ADR-0005: Structured JSON Logging
 
-- *Status:* Accepted
-- *Date:* 2025-02-21
+- *Status:* Implemented
+- *Date:* 2025-01-09
 - *Deciders:* Core Engineering Team
 
 ## Context
@@ -14,27 +14,38 @@ The gateway must emit logs that:
 
 Our configuration supports:
 
-- `LOG_FORMAT`: `json` or `plain`
-- `LOG_LEVEL`: standard Python levels
-- `LOG_FILE`: optional log file destination
+- `LOG_FORMAT`: `json` or `text` (console format only)
+- `LOG_LEVEL`: standard Python levels (`DEBUG`, `INFO`, `WARNING`, `ERROR`, `CRITICAL`)
+- `LOG_TO_FILE`: enable file logging (default: `false` - stdout/stderr only)
+- `LOG_FILE`: log filename when file logging is enabled (default: `null`)
+- `LOG_FOLDER`: directory for log files when enabled (default: `null`)
+- `LOG_FILEMODE`: file write mode (default: `a+` for append)
+- `LOG_ROTATION_ENABLED`: enable automatic log rotation (default: `false`)
+- `LOG_MAX_SIZE_MB`: maximum file size before rotation in MB (default: `1`)
+- `LOG_BACKUP_COUNT`: number of backup files to keep (default: `5`)
 
-Logs are initialized at startup via `LoggingService`.
+Logs are initialized at startup via centralized `LoggingService`. By default, logs go only to stdout/stderr. File logging with dual-format output is optional via `LOG_TO_FILE=true`.
 
 ## Decision
 
-Use the Python standard `logging` module with:
+Use the Python standard `logging` module with centralized `LoggingService`:
 
-- A **custom JSON formatter** for structured logs (e.g. `{"level": "INFO", "msg": ..., "request_id": ...}`)
-- **Plain text output** when `LOG_FORMAT=plain`
-- Per-request context via filters or middleware
-- Global setup at app startup to avoid late binding issues
+- **JSON formatter** for file logs using `python-json-logger` library
+- **Text formatter** for console logs for human readability
+- **Dual output**: JSON to files, text to console
+- **Optional rotating file handler** for automatic log management (configurable)
+- **Centralized service** integrated across all 22+ modules
+- Global setup at app startup with lazy handler initialization
 
 ## Consequences
 
-- 📋 Easily parsed logs suitable for production observability pipelines
-- ⚙️ Compatible with `stdout`, file, or syslog targets
-- 🧪 Local development uses plain logs for readability
-- 🧱 Minimal dependency footprint (no third-party logging libraries)
+- 📋 **Structured JSON logs** suitable for production observability pipelines (ELK, Datadog, etc.)
+- ⚙️ **Dual format support**: JSON files for machines, text console for humans
+- 🔄 **Optional log rotation** prevents disk space issues when enabled
+- 🧪 **Development-friendly** with human-readable console output
+- 📁 **Organized storage** with configurable log directories and retention
+- 🧱 **Minimal dependencies**: Uses standard library + `python-json-logger`
+- 🎯 **Consistent logging** across all application modules
 
 ## Alternatives Considered
 
@@ -47,4 +58,15 @@ Use the Python standard `logging` module with:
 
 ## Status
 
-Structured logging is implemented in `LoggingService`, configurable via environment variables.
+**✅ Implemented** - Structured logging is fully implemented in `LoggingService` with:
+
+- Centralized logging service integrated across all modules
+- Dual-format output (JSON to files, text to console)
+- HTTP access log capture (uvicorn.access and uvicorn.error loggers)
+- Optional log rotation with configurable size limits and retention
+- Environment variable configuration support
+- Production-ready with proper error handling and lazy initialization
+
+**Files Modified**: 22 modules updated to use `LoggingService`
+**Dependencies Added**: `python-json-logger>=2.0.0`
+**Configuration**: Via `LOG_LEVEL`, `LOG_FORMAT`, `LOG_TO_FILE`, `LOG_FILE`, `LOG_FOLDER`, `LOG_FILEMODE`, `LOG_ROTATION_ENABLED`, `LOG_MAX_SIZE_MB`, `LOG_BACKUP_COUNT`

docs/docs/faq/index.md

@@ -66,7 +66,7 @@
 ???+ example "🪛 What are some advanced environment variables I can configure?"
     - Basic: `HOST`, `PORT`, `APP_ROOT_PATH`
     - Auth: `AUTH_REQUIRED`, `BASIC_AUTH_*`, `JWT_SECRET_KEY`
-    - Logging: `LOG_LEVEL`, `LOG_FORMAT`, `LOG_FILE`
+    - Logging: `LOG_LEVEL`, `LOG_FORMAT`, `LOG_TO_FILE`, `LOG_FILE`, `LOG_FOLDER`, `LOG_ROTATION_ENABLED`, `LOG_MAX_SIZE_MB`, `LOG_BACKUP_COUNT`
     - Transport: `TRANSPORT_TYPE`, `WEBSOCKET_PING_INTERVAL`, `SSE_RETRY_TIMEOUT`
     - Tools: `TOOL_TIMEOUT`, `MAX_TOOL_RETRIES`, `TOOL_RATE_LIMIT`, `TOOL_CONCURRENT_LIMIT`
     - Federation: `FEDERATION_ENABLED`, `FEDERATION_PEERS`, `FEDERATION_SYNC_INTERVAL`

docs/docs/manage/.pages

@@ -2,6 +2,7 @@ nav:
   - index.md
   - backup.md
   - logging.md
+  - logging-examples.md
   - upgrade.md
   - tuning.md
   - securing.md