Introduce embedded Dendrite server

contrib/dendrite-demo-embedded/README.md

@@ -0,0 +1,204 @@
+# Dendrite Embedded
+
+A package for embedding a Matrix homeserver into your Go applications.
+
+## Overview
+
+The `embedded` package allows you to integrate a fully functional Matrix homeserver within your Go application. This is useful for applications that need to provide Matrix-based communications without requiring a separate server deployment.
+
+## Usage
+
+# Dendrite Embedded
+
+A package for embedding a Matrix homeserver into your Go applications.
+
+## Overview
+
+The `embedded` package allows you to integrate a fully functional Matrix homeserver within your Go application. This is useful for applications that need to provide Matrix-based communications without requiring a separate server deployment, or for creating specialized Matrix servers with custom networking layers (e.g., Tor, I2P).
+
+## Usage
+
+### Basic Example
+
+```go
+package main
+
+import (
+    "context"
+    "crypto/ed25519"
+    "crypto/rand"
+    "log"
+    "net"
+    "time"
+
+    "github.com/element-hq/dendrite/contrib/dendrite-demo-embedded"
+)
+
+func main() {
+    // Generate server keys
+    _, privateKey, _ := ed25519.GenerateKey(rand.Reader)
+
+    // Configure the server
+    config := embedded.DefaultConfig()
+    config.ServerName = "localhost"
+    config.KeyID = "ed25519:1"
+    config.PrivateKey = privateKey
+    config.DatabasePath = "./dendrite.db"
+    config.MediaStorePath = "./media_store"
+    config.JetStreamPath = "./jetstream"
+
+    // Create server
+    server, err := embedded.NewServer(config)
+    if err != nil {
+        log.Fatalf("Failed to create server: %v", err)
+    }
+
+    // Set up the listener
+    listener, err := net.Listen("tcp", "0.0.0.0:8080")
+    if err != nil {
+        log.Fatalf("Failed to create listener: %v", err)
+    }
+
+    // Start the server
+    if err := server.Start(context.Background(), listener); err != nil {
+        log.Fatalf("Failed to start server: %v", err)
+    }
+
+    // Wait for shutdown signal
+    <-server.GetProcessContext().WaitForShutdown()
+
+    // Stop gracefully
+    ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
+    defer cancel()
+    server.Stop(ctx)
+}
+```
+
+### Using with Custom Network Layer
+
+The embedded library is designed to work with any `net.Listener` implementation, making it easy to integrate with custom networking layers:
+
+```go
+// Example: Using with a custom listener (e.g., Tor)
+listener := createCustomListener() // Returns net.Listener
+
+server, err := embedded.NewServer(config)
+if err != nil {
+    log.Fatal(err)
+}
+
+if err := server.Start(context.Background(), listener); err != nil {
+    log.Fatal(err)
+}
+```
+
+### Using with Existing Dendrite Configuration
+
+If you already have a Dendrite configuration file or object, you can use it directly:
+
+```go
+import (
+    "github.com/element-hq/dendrite/setup"
+    "github.com/element-hq/dendrite/setup/config"
+)
+
+// Parse existing config
+dendriteConfig := setup.ParseFlags(true)
+
+// Use it with embedded server
+config := embedded.ServerConfig{
+    RawDendriteConfig: dendriteConfig,
+}
+
+server, err := embedded.NewServer(config)
+// ... rest of setup
+```
+
+## Configuration
+
+The `ServerConfig` struct allows you to configure various aspects of the embedded server:
+
+### Basic Identity
+- `ServerName`: The Matrix server name (e.g., "example.com")
+- `KeyID`: The key ID for signing (e.g., "ed25519:auto")
+- `PrivateKey`: The ed25519 private key for the server
+
+### Storage Paths
+- `DatabasePath`: Path to the SQLite database file
+- `MediaStorePath`: Path to store uploaded media files
+- `JetStreamPath`: Path to store JetStream/NATS data
+
+### HTTP Client
+- `HTTPClient`: Custom HTTP client for outbound requests (useful for routing through Tor/I2P)
+
+### Feature Flags
+- `DisableFederation`: Disable federation with other Matrix servers
+- `EnableMetrics`: Enable Prometheus metrics endpoint
+- `MetricsUsername`/`MetricsPassword`: Basic auth for metrics endpoint
+
+### Performance Settings
+- `CacheMaxSize`: Maximum cache size in bytes (default: 64MB)
+- `CacheMaxAge`: Maximum age for cached items (default: 1 hour)
+
+### Advanced
+- `RateLimitYAMLPath`: Path to custom rate limiting configuration
+- `RawDendriteConfig`: Use a complete Dendrite configuration object
+
+## Features
+
+- Full Matrix API support (Client-Server and Server-Server)
+- Optional federation support
+- Metrics and profiling capabilities
+- Configurable rate limiting
+- SQLite database backend
+- Media storage and serving
+- MSC (Matrix Spec Change) support
+- Admin endpoints
+
+## Examples
+
+See the following implementations for real-world examples:
+
+- **Tor**: `contrib/dendrite-demo-tor` - Matrix server over Tor onion services
+- **I2P**: `contrib/dendrite-demo-i2p` - Matrix server over I2P
+
+## Architecture
+
+The embedded library provides a clean separation between:
+
+1. **Core Server Logic**: Handled by the embedded package
+2. **Transport Layer**: Provided by the application (via `net.Listener`)
+3. **HTTP Client**: Configurable for custom routing (e.g., through anonymity networks)
+
+This architecture allows the same server code to work with any network transport that implements the standard Go `net.Listener` interface.
+
+## Lifecycle Management
+
+The embedded server provides proper lifecycle management:
+
+1. **Initialization**: `NewServer()` creates the server but doesn't start it
+2. **Startup**: `Start()` begins serving on the provided listener
+3. **Runtime**: Server runs until shutdown is requested
+4. **Shutdown**: `Stop()` gracefully shuts down all components
+
+The process context returned by `GetProcessContext()` can be used to coordinate shutdown signals across your application.
+
+## Thread Safety
+
+The embedded server is thread-safe and uses internal locking to prevent concurrent start/stop operations. Multiple goroutines can safely call `Start()` and `Stop()`## Configuration
+
+The `ServerConfig` struct allows you to configure various aspects of the embedded server:
+
+- Server identity (name, keys)
+- Storage paths
+- Feature flags (federation, metrics, etc.)
+- Performance settings
+
+Use `DefaultConfig()` as a starting point and customize as needed.
+
+## Features
+
+- Full Matrix API support
+- Optional federation support
+- Metrics and profiling capabilities
+- Configurable rate limiting

contrib/dendrite-demo-embedded/config.go

@@ -0,0 +1,182 @@
+package embedded
+
+import (
+	"crypto/ed25519"
+	"fmt"
+	"net/http"
+	"time"
+
+	"github.com/element-hq/dendrite/setup/config"
+	"github.com/matrix-org/gomatrixserverlib"
+	"github.com/matrix-org/gomatrixserverlib/spec"
+)
+
+// ServerConfig contains configuration for the embedded server
+type ServerConfig struct {
+	// Basic server identity
+	ServerName string
+	KeyID      string
+	PrivateKey ed25519.PrivateKey
+
+	// Storage paths
+	DatabasePath   string
+	MediaStorePath string
+	JetStreamPath  string
+
+	// HTTP client configuration
+	HTTPClient *http.Client
+
+	// Feature flags
+	DisableFederation bool
+	EnableMetrics     bool
+	MetricsUsername   string
+	MetricsPassword   string
+
+	// Cache configuration
+	CacheMaxSize int64
+	CacheMaxAge  time.Duration
+
+	// Rate limiting
+	RateLimitYAMLPath string
+
+	// Custom config options
+	RawDendriteConfig *config.Dendrite
+}
+
+// DefaultConfig returns a configuration with sensible defaults for an embedded server
+func DefaultConfig() ServerConfig {
+	return ServerConfig{
+		ServerName:        "localhost",
+		KeyID:             "ed25519:auto",
+		DatabasePath:      "./dendrite.db",
+		MediaStorePath:    "./media_store",
+		JetStreamPath:     "./jetstream",
+		DisableFederation: true,
+		EnableMetrics:     false,
+		CacheMaxSize:      64 * 1024 * 1024, // 64 MB
+		CacheMaxAge:       time.Hour,
+		HTTPClient:        http.DefaultClient,
+	}
+}
+
+// toDendriteConfig converts the ServerConfig to a Dendrite config
+func (c *ServerConfig) toDendriteConfig() (*config.Dendrite, error) {
+	// If a raw config was provided, use that as the base
+	if c.RawDendriteConfig != nil {
+		return c.RawDendriteConfig, nil
+	}
+
+	// Create a new base config
+	cfg := &config.Dendrite{}
+	err := SetDefaults(cfg)
+	if err != nil {
+		return nil, fmt.Errorf("failed to set config defaults: %w", err)
+	}
+
+	// Set basic identity configuration
+	cfg.Global.ServerName = spec.ServerName(c.ServerName)
+	cfg.Global.PrivateKey = c.PrivateKey
+	cfg.Global.KeyID = gomatrixserverlib.KeyID(c.KeyID)
+
+	// Set storage paths
+	cfg.Global.DatabaseOptions.ConnectionString = config.DataSource("file:" + c.DatabasePath)
+	cfg.MediaAPI.BasePath = config.Path(c.MediaStorePath)
+	cfg.Global.JetStream.StoragePath = config.Path(c.JetStreamPath)
+
+	// Configure caching
+	cfg.Global.Cache.EstimatedMaxSize = config.DataUnit(c.CacheMaxSize)
+	cfg.Global.Cache.MaxAge = c.CacheMaxAge
+
+	// Configure federation
+	cfg.Global.DisableFederation = c.DisableFederation
+
+	// Set up metrics
+	if c.EnableMetrics {
+		cfg.Global.Metrics.Enabled = true
+		if c.MetricsUsername != "" && c.MetricsPassword != "" {
+			cfg.Global.Metrics.BasicAuth = struct {
+				Username string `yaml:"username"`
+				Password string `yaml:"password"`
+			}{
+				Username: c.MetricsUsername,
+				Password: c.MetricsPassword,
+			}
+		}
+	}
+
+	// Configure rate limiting
+	if c.RateLimitYAMLPath != "" {
+		cfg.ClientAPI.RateLimiting.Enabled = true
+		// Use custom rate limiting file if provided
+		// Note: This assumes the Dendrite config structure supports setting this
+	}
+
+	// Enable registration by default for embedded servers
+	cfg.ClientAPI.OpenRegistrationWithoutVerificationEnabled = true
+	cfg.ClientAPI.RegistrationDisabled = false
+
+	return cfg, nil
+}
+
+// SetDefaults populates a Dendrite config with sensible default values
+func SetDefaults(cfg *config.Dendrite) error {
+	// Create a new config with default values if nil
+	if cfg == nil {
+		return fmt.Errorf("cannot set defaults on nil config")
+	}
+
+	// Global defaults
+	if cfg.Global.ServerName == "" {
+		cfg.Global.ServerName = "localhost"
+	}
+	if cfg.Global.DatabaseOptions.ConnectionString == "" {
+		cfg.Global.DatabaseOptions.ConnectionString = "file:dendrite.db"
+	}
+	if cfg.Global.JetStream.StoragePath == "" {
+		cfg.Global.JetStream.StoragePath = config.Path("jetstream")
+	}
+
+	// Cache defaults
+	if cfg.Global.Cache.EstimatedMaxSize == 0 {
+		cfg.Global.Cache.EstimatedMaxSize = config.DataUnit(64 * 1024 * 1024) // 64 MB
+	}
+	if cfg.Global.Cache.MaxAge == 0 {
+		cfg.Global.Cache.MaxAge = time.Hour
+	}
+
+	// Media API defaults
+	if cfg.MediaAPI.BasePath == "" {
+		cfg.MediaAPI.BasePath = config.Path("media_store")
+	}
+
+	// Client API defaults
+	cfg.ClientAPI.OpenRegistrationWithoutVerificationEnabled = true
+	cfg.ClientAPI.RegistrationDisabled = false
+
+	// Room server defaults
+	if cfg.RoomServer.Database.ConnectionString == "" {
+		cfg.RoomServer.Database.ConnectionString = cfg.Global.DatabaseOptions.ConnectionString
+	}
+
+	// Federation API defaults
+	if cfg.FederationAPI.Database.ConnectionString == "" {
+		cfg.FederationAPI.Database.ConnectionString = cfg.Global.DatabaseOptions.ConnectionString
+	}
+
+	// Sync API defaults
+	if cfg.SyncAPI.Database.ConnectionString == "" {
+		cfg.SyncAPI.Database.ConnectionString = cfg.Global.DatabaseOptions.ConnectionString
+	}
+
+	// Media API db defaults
+	if cfg.MediaAPI.Database.ConnectionString == "" {
+		cfg.MediaAPI.Database.ConnectionString = cfg.Global.DatabaseOptions.ConnectionString
+	}
+
+	// User API db defaults
+	if cfg.UserAPI.AccountDatabase.ConnectionString == "" {
+		cfg.UserAPI.AccountDatabase.ConnectionString = cfg.Global.DatabaseOptions.ConnectionString
+	}
+
+	return nil
+}