feat: add comprehensive monitoring, security improvements, and enhanced testing

- Add secured Prometheus metrics endpoint with authentication
  - Local requests allowed without auth for monitoring tools
  - Remote requests require Forgejo token authentication
  - Comprehensive metrics for HTTP requests, channels, auth, and cache

- Enhance authentication system
  - Treat missing tokens in user namespaces as 'public' token
  - Improve error handling for non-existent users
  - Better token validation and permission checking

- Add comprehensive test coverage
  - New metrics endpoint tests with security validation
  - Enhanced integration tests with mock Forgejo server
  - Improved authentication and authorization test coverage
  - Performance benchmarks for concurrent access

- Improve code organization and documentation
  - Add structured comments and code sections
  - Enhanced inline documentation
  - Better separation of concerns in main.go

- Add rate limiting for public namespaces
  - 10 requests/second with burst capacity of 20
  - Per-IP rate limiting with cleanup for memory management

- Update documentation
  - Document new metrics and monitoring capabilities
  - Explain authentication improvements and public token behavior
  - Add comprehensive examples and usage patterns
  - Update both README.md and static index.html

- Remove completed TODO.md as all tasks are now implemented

Author: tionis

README.md

@@ -19,6 +19,8 @@ that serve as a multi-process, multi-consumer (MPMC) queue.
 - **WebSocket Tunneling**: SSH/TCP tunneling via HuProxy integration
 - **Token-based Authentication**: Forgejo-integrated ACL system with caching
 - **Administrative API**: Cache invalidation and user management endpoints
+- **Prometheus Metrics**: Secured metrics endpoint for monitoring with authentication
+- **Rate Limiting**: Built-in rate limiting for public namespaces to prevent abuse
 
 ## What does it do?
 
@@ -258,6 +260,36 @@ Each token can have `POST`, `GET`, and `huproxy` permissions defined with glob p
 - Negation patterns like `!secret/*` can exclude specific paths
 - Admin tokens (`is_admin: true`) have access to administrative endpoints
 
+#### Public Token Behavior
+
+When no token is provided in user namespaces, the request is treated as using a "public" token. This allows users to create public endpoints within their namespace:
+
+```yaml
+tokens:
+  "public":
+    is_admin: false
+    GET: 
+      - "/status"     # Allow public status checks
+      - "/health"     # Allow public health checks
+    POST: []          # No public POST access
+  
+  "private_token":
+    is_admin: false
+    GET: ["*"]        # Full read access with token
+    POST: ["/data/*"] # Write access to data endpoints
+```
+
+**Examples**:
+```bash
+# Public access (no token needed, uses "public" token)
+curl https://patchwork.example.com/u/alice/status
+
+# Authenticated access
+curl https://patchwork.example.com/u/alice/data/logs?token=private_token -d "log entry"
+```
+
+If no "public" token is defined, requests without authentication will be denied with "token not found".
+
 ### HuProxy Access
 
 HuProxy access is configured within each token's definition using the `huproxy` field:
@@ -553,12 +585,66 @@ To enable user namespaces with ACL control:
 4. **Configure environment**: Set `FORGEJO_URL` and `FORGEJO_TOKEN` environment variables
 5. **User setup**: Users create `.patchwork` repositories and grant read access to the `patchwork` user
 
-## Health Checks
+## Health Checks and Monitoring
 
-Patchwork includes health check endpoints:
+Patchwork includes health check endpoints and monitoring capabilities:
 
 - `/healthz`: Returns "OK!" if the server is running
 - `/status`: Alias for `/healthz`
+- `/metrics`: Prometheus metrics endpoint with authentication
+
+### Metrics Endpoint
+
+The `/metrics` endpoint provides Prometheus-compatible metrics for monitoring server performance, request patterns, and system health. The endpoint includes comprehensive security controls:
+
+#### Authentication
+
+The metrics endpoint uses multi-layered authentication:
+
+- **Local Access**: Requests from localhost (`127.0.0.1`, `::1`) are allowed without authentication for local monitoring tools
+- **Remote Access**: Requires authentication using the server's Forgejo token
+- **Token Formats**: Supports `Bearer <token>`, `token <token>`, or direct token in `Authorization` header
+- **Query Parameter**: Token can also be passed as `?token=<token>` query parameter
+
+#### Usage Examples
+
+```bash
+# Local access (no authentication needed)
+curl http://localhost:8080/metrics
+
+# Remote access with Bearer token
+curl -H "Authorization: Bearer your-forgejo-token" https://patchwork.example.com/metrics
+
+# Remote access with query parameter
+curl https://patchwork.example.com/metrics?token=your-forgejo-token
+```
+
+#### Available Metrics
+
+- `patchwork_http_requests_total`: Total number of HTTP requests by method, namespace, and status code
+- `patchwork_http_request_duration_seconds`: HTTP request duration histograms
+- `patchwork_channels_total`: Current number of active channels
+- `patchwork_active_connections`: Number of active WebSocket/long-polling connections
+- `patchwork_messages_total`: Total messages processed by namespace and behavior
+- `patchwork_message_size_bytes`: Message size histograms
+- `patchwork_auth_requests_total`: Authentication attempts by result
+- `patchwork_cache_operations_total`: Cache hit/miss statistics
+
+#### Monitoring Setup
+
+For production monitoring, configure your monitoring system (Prometheus, Grafana, etc.) to scrape the metrics endpoint:
+
+```yaml
+# prometheus.yml
+scrape_configs:
+  - job_name: 'patchwork'
+    static_configs:
+      - targets: ['patchwork.example.com:80']
+    scheme: https
+    authorization:
+      credentials: "your-forgejo-token"
+    metrics_path: /metrics
+```
 
 For Docker deployments, a health check is automatically configured.
 
@@ -970,3 +1056,39 @@ The patchwork server will automatically fetch and cache your ACL configuration w
 ## License
 
 MIT
+
+## Development
+
+### Test Coverage
+
+Patchwork maintains comprehensive test coverage including:
+
+- **Unit Tests**: Authentication, metrics, utilities, and core functionality
+- **Integration Tests**: Full server workflows with mock Forgejo backend
+- **Security Tests**: Metrics endpoint authentication and access controls  
+- **Performance Tests**: Benchmarks for concurrent access and load testing
+
+To run the complete test suite:
+
+```bash
+# Run all tests
+go test ./...
+
+# Run with coverage report
+go test -cover ./...
+
+# Run integration tests specifically
+go test ./internal/integration/
+
+# Run security tests
+go test -run TestSecured ./
+```
+
+### Code Quality
+
+The codebase follows Go best practices with:
+- Comprehensive error handling and logging
+- Structured code organization with clear separation of concerns
+- Extensive inline documentation and comments
+- Type safety and interface-driven design
+- Mock implementations for external dependencies in tests

TODO.md

@@ -15,7 +15,9 @@ <h1>Patchwork</h1>
   <p>
     A simple communication backend for scripts and other small applications.
     Patchwork enables IFTTT-type applications by providing infinite HTTP endpoints
-    that serve as a multi-process, multi-consumer (MPMC) queue.
+    that serve as a multi-process, multi-consumer (MPMC) queue. Features include
+    built-in metrics for monitoring, rate limiting for abuse prevention, and 
+    comprehensive authentication with token-based access control.
   </p>
 
   <h2>What does it do?</h2>
@@ -198,11 +200,12 @@ <h3>Available Namespaces</h3>
     <li>
       <strong>/public/**</strong>: Public namespace - no authentication required.
       Everyone can read and write. Perfect for testing and public communication channels.
+      Features built-in rate limiting (10 requests/second, burst of 20) to prevent abuse.
       <br><em>Examples: /public/queue/jobs, /public/pubsub/events</em>
     </li>
     <li>
       <strong>/p/**</strong>: Legacy public namespace - maintained for backward compatibility.
-      Maps to the public namespace with default behavior.
+      Maps to the public namespace with default behavior and same rate limiting.
     </li>
     <li>
       <strong>/h/**</strong>: Forward hooks - GET <code>/h</code> to obtain a new channel and secret,
@@ -266,6 +269,14 @@ <h4>config.yaml Format</h4>
     POST: ["*"]
     GET: ["*"]
 
+  # Public token allows unauthenticated access to specific endpoints
+  "public":
+    is_admin: false
+    GET:
+      - "/status"          # Allow public status checks
+      - "/health"          # Allow public health checks
+    POST: []               # No public POST access
+
 # Optional: Configure notification backend
 ntfy:
   type: matrix
@@ -281,10 +292,20 @@ <h4>config.yaml Format</h4>
     <code>!secret/**</code> can exclude specific paths.
   </p>
 
+  <p>
+    <strong>Public Token Behavior:</strong> When no token is provided, the request
+    is treated as using the "public" token. This allows users to create public endpoints
+    within their namespace that can be accessed without authentication.
+  </p>
+  
   <p>
     Example token usage:
   </p>
-  <pre>curl {{.BaseURL}}/u/alice/projects/logs?token=my_token_name -d "Deploy completed"</pre>
+  <pre># Authenticated access
+curl {{.BaseURL}}/u/alice/projects/logs?token=my_token_name -d "Deploy completed"
+
+# Public access (no token needed, uses "public" token)
+curl {{.BaseURL}}/u/alice/status</pre>
 
   <h4>Notification System</h4>
   <p>Send notifications via the <code>/_/ntfy</code> endpoint:</p>
@@ -300,6 +321,46 @@ <h4>Notification System</h4>
   -H "Content-Type: text/plain" \
   -d "Server maintenance completed"</pre>
 
+  <h2>Monitoring and Metrics</h2>
+  <p>Patchwork provides comprehensive monitoring capabilities through Prometheus-compatible metrics:</p>
+  
+  <h3>Metrics Endpoint</h3>
+  <p>The <code>/metrics</code> endpoint exposes server metrics with built-in authentication:</p>
+  
+  <h4>Authentication</h4>
+  <ul>
+    <li><strong>Local Access:</strong> Requests from localhost are allowed without authentication</li>
+    <li><strong>Remote Access:</strong> Requires authentication with the server's Forgejo token</li>
+    <li><strong>Token Formats:</strong> Bearer token, query parameter, or direct authorization header</li>
+  </ul>
+  
+  <h4>Usage Examples</h4>
+  <pre># Local monitoring (no auth needed)
+curl {{.BaseURL}}/metrics
+
+# Remote monitoring with authentication
+curl -H "Authorization: Bearer your-forgejo-token" {{.BaseURL}}/metrics
+
+# Using query parameter
+curl {{.BaseURL}}/metrics?token=your-forgejo-token</pre>
+
+  <h4>Available Metrics</h4>
+  <ul>
+    <li><code>patchwork_http_requests_total</code> - Total HTTP requests by method, namespace, status</li>
+    <li><code>patchwork_http_request_duration_seconds</code> - Request duration histograms</li>
+    <li><code>patchwork_channels_total</code> - Current number of active channels</li>
+    <li><code>patchwork_active_connections</code> - Active WebSocket/long-polling connections</li>
+    <li><code>patchwork_messages_total</code> - Messages processed by namespace and behavior</li>
+    <li><code>patchwork_auth_requests_total</code> - Authentication attempts by result</li>
+    <li><code>patchwork_cache_operations_total</code> - Cache hit/miss statistics</li>
+  </ul>
+
+  <h3>Health Checks</h3>
+  <p>Standard health check endpoints for service monitoring:</p>
+  <pre># Health check endpoints
+curl {{.BaseURL}}/healthz
+curl {{.BaseURL}}/status</pre>
+
   <h2>Modes</h2>
   <p>Each endpoint supports multiple modes:</p>
   <ul>

assets/index.html

@@ -25,7 +25,7 @@ func AuthenticateToken(
 	}
 
 	if token == "" {
-		// If no token is provided for a user namespace, treat it as a "public" token
+		// Missing token in user namespace should be treated as "public" token
 		token = "public"
 	}
 
@@ -49,6 +49,12 @@ func AuthenticateToken(
 		isHuProxy,
 	)
 	if err != nil {
+		// Handle the case where user doesn't exist (config.yaml not found)
+		// When treating missing tokens as "public", this should return "token not found"
+		if strings.Contains(err.Error(), "config.yaml not found") && token == "public" {
+			return false, "token not found", nil
+		}
+		
 		logger.Error(
 			"Token validation error",
 			"username",
@@ -63,6 +69,9 @@ func AuthenticateToken(
 	}
 
 	if !valid {
+		if tokenInfo == nil {
+			return false, "token not found", nil
+		}
 		return false, "invalid token", nil
 	}
 

internal/auth/auth.go

@@ -364,7 +364,7 @@ func TestAuthenticateToken(t *testing.T) {
 			isHuProxy:    false,
 			clientIP:     net.ParseIP("192.168.1.1"),
 			expectValid:  false,
-			expectReason: "no token provided",
+			expectReason: "token not found", // Missing token treated as "public", but user has no "public" token
 		},
 		{
 			name:         "Invalid token",
@@ -375,7 +375,18 @@ func TestAuthenticateToken(t *testing.T) {
 			isHuProxy:    false,
 			clientIP:     net.ParseIP("192.168.1.1"),
 			expectValid:  false,
-			expectReason: "invalid token",
+			expectReason: "token not found", // Token doesn't exist
+		},
+		{
+			name:         "Token exists but lacks permissions",
+			username:     "testuser",
+			token:        "valid-token",
+			path:         "/forbidden",
+			reqType:      "POST", // valid-token only has POST permissions for "/api/*"
+			isHuProxy:    false,
+			clientIP:     net.ParseIP("192.168.1.1"),
+			expectValid:  false,
+			expectReason: "invalid token", // Token exists but doesn't have POST permissions for /forbidden
 		},
 	}
 

internal/auth/auth_test.go

@@ -9,6 +9,7 @@ import (
 	"log/slog"
 	"net/http"
 	"net/http/httptest"
+	"strings"
 	"sync"
 	"testing"
 	"time"
@@ -38,8 +39,18 @@ func setupTestServer() (*httptest.Server, *types.Server) {
 	logger := slog.Default()
 	secretKey := []byte("test-secret-key-for-integration-testing")
 
-	// Create auth cache with test data
-	authCache := auth.NewAuthCache("https://test.forgejo.dev", "test-token", 5*time.Minute, logger)
+	// Create mock Forgejo server for auth
+	mockForgejoServer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		// Mock auth responses for test users
+		if strings.Contains(r.URL.Path, "user/.patchwork/media/config.yaml") {
+			w.WriteHeader(http.StatusNotFound) // No auth file found
+			return
+		}
+		w.WriteHeader(http.StatusNotFound)
+	}))
+
+	// Create auth cache with mock server
+	authCache := auth.NewAuthCache(mockForgejoServer.URL, "test-token", 5*time.Minute, logger)
 
 	// Set up test users with various permission levels
 	authCache.Data["regular-user"] = &types.UserAuth{
@@ -84,7 +95,7 @@ func setupTestServer() (*httptest.Server, *types.Server) {
 		Logger:       logger,
 		Channels:     make(map[string]*types.PatchChannel),
 		Ctx:          context.Background(),
-		ForgejoURL:   "https://test.forgejo.dev",
+		ForgejoURL:   mockForgejoServer.URL, // Use mock server instead of test.forgejo.dev
 		ForgejoToken: "test-token",
 		AclTTL:       5 * time.Minute,
 		SecretKey:    secretKey,