diff --git a/MULTI_UNDERLAY_PROTOCOL.md b/MULTI_UNDERLAY_PROTOCOL.md new file mode 100644 index 0000000..24c0f88 --- /dev/null +++ b/MULTI_UNDERLAY_PROTOCOL.md @@ -0,0 +1,259 @@ +# Multi-Underlay Protocol Documentation + +## Overview + +The multi-underlay protocol is a channel-layer feature that allows a single logical channel to maintain multiple concurrent underlying transport connections (underlays). This provides resilience, load distribution, and the ability to use different transport types simultaneously. + +## Core Concepts + +### Multi-Channel +A `MultiChannel` represents a single logical communication channel that can contain multiple underlying transport connections. It implements the standard `Channel` interface while internally managing multiple `Underlay` instances. + +### Underlay +An `Underlay` represents a single underlying transport connection (TCP, WebSocket, UDP, etc.). Multiple underlays can be grouped together under a single multi-channel using a shared group secret. + +## Connection ID +A short ID, used to identify which logical channel the connection belongs to. + +### Group Secret +A secret generated on the server side when the first connection for a channel is made. It must be presented back to the server when making additional connections. As it is a UUID, it is nearly impossible to guess. Because it's +generated on first connection, it also changes if a server loses all connection to a channel. Since clients may not lose all connections simultaenously, this also serves as a signal that the client must start afresh, as +the old group secret won't be recognized and new connections will be rejected. + +## Connection Setup Flow + +### Initial Connection (First Underlay) + +1. **Dialer Initiation** + - Creates first underlay connection to listener + - Sends Hello message with required headers + +2. **Header Requirements for First Connection** + ```go + headers := Headers{ + TypeHeader: []byte("default"), // Underlay type identifier + IsGroupedHeader: {1}, // Mark as grouped connection + IsFirstGroupConnection: {1}, // Mark as first in group + ConnectionIdHeader: []byte(connectionId), // Unique group identifier + } + ``` + +3. **Listener Processing** + - Receives Hello message + - Validates `IsFirstGroupConnection` header + - Generates group secret if not provided + - Creates new `MultiChannel` instance + - Responds with acknowledgment containing group secret + +4. **Response Headers** + ```go + response.Headers[GroupSecretHeader] = groupSecret // Generated secret + response.Headers[IsGroupedHeader] = {1} // Confirm grouped + response.Headers[TypeHeader] = underlayType // Echo type + ``` + +### Additional Underlays + +1. **Subsequent Connections** + - Use existing group's connection ID and secret + - Must provide correct group secret for validation + +2. **Header Requirements for Additional Connections** + ```go + headers := Headers{ + TypeHeader: []byte(underlayType), // Type (e.g., "priority") + ConnectionIdHeader: []byte(groupId), // Same group ID + GroupSecretHeader: groupSecret, // Shared secret + IsGroupedHeader: {1}, // Mark as grouped + // Note: IsFirstGroupConnection NOT set + } + ``` + +3. **Validation Process** + - Listener checks if group exists by connection ID + - Validates group secret matches + - Accepts underlay into existing multi-channel + - Rejects if secret doesn't match or channel is closed + +## Required Headers + +### Core Headers (message.go:41-52) + +| Header | Value | Description | Required When | +|--------|-------|-------------|---------------| +| `ConnectionIdHeader` (0) | string | Unique identifier for the channel group | All grouped connections | +| `TypeHeader` (7) | string | Type of underlay (e.g., "default", "priority") | All connections | +| `IsGroupedHeader` (9) | boolean | Indicates this is a grouped connection | All grouped connections | +| `GroupSecretHeader` (10) | []byte | Cryptographic group identifier | All connections after first | +| `IsFirstGroupConnection` (11) | boolean | Marks the first connection in a group | First connection only | + +### Header Usage by Connection Type + +#### First Connection +- `ConnectionIdHeader`: Generated or provided connection ID +- `TypeHeader`: Underlay type (typically "default") +- `IsGroupedHeader`: Must be `true` +- `IsFirstGroupConnection`: Must be `true` +- `GroupSecretHeader`: Optional (generated if not provided) + +#### Subsequent Connections +- `ConnectionIdHeader`: Same as first connection +- `TypeHeader`: Underlay type (can be different, e.g., "priority") +- `IsGroupedHeader`: Must be `true` +- `GroupSecretHeader`: Must match group secret +- `IsFirstGroupConnection`: Must NOT be set + +## Protocol Flow Diagrams + +### First Connection Setup +``` +Dialer Listener + | | + |-- Hello (IsFirstGroupConnection) ----->| + | |- Generate GroupSecret + | |- Create MultiChannel + |<-- Result (GroupSecret) ---------------| + | | + |-- Start Message Exchange ------------->| +``` + +### Additional Connection Setup +``` +Dialer Listener + | | + |-- Hello (GroupSecret) ---------------->| + | |- Validate GroupSecret + | |- Add to existing MultiChannel + |<-- Result (Confirmed) -----------------| + | | + |-- Start Message Exchange ------------->| +``` + +## Implementation Details + +### MultiChannel Creation (multi.go:95-141) +```go +// Configuration for creating a new multi-channel +type MultiChannelConfig struct { + LogicalName string + Options *Options + UnderlayHandler UnderlayHandler + BindHandler BindHandler + Underlay Underlay // First underlay +} +``` + +### Underlay Acceptance (multi.go:143-171) +```go +func (mc *multiChannelImpl) AcceptUnderlay(underlay Underlay) error { + // Validate group secret + groupSecret := underlay.Headers()[GroupSecretHeader] + if !bytes.Equal(groupSecret, mc.groupSecret) { + return fmt.Errorf("incorrect group secret") + } + + // Add to underlay collection + mc.underlays.Append(underlay) + mc.startMultiplex(underlay) + + return nil +} +``` + +### Listener Processing (multi_listener.go:34-85) +```go +func (ml *MultiListener) AcceptUnderlay(underlay Underlay) { + isGrouped, _ := Headers(underlay.Headers()).GetBoolHeader(IsGroupedHeader) + + if !isGrouped { + // Handle non-grouped connection + return + } + + chId := underlay.ConnectionId() + + if existingChannel, exists := ml.channels[chId]; exists { + // Add to existing multi-channel + existingChannel.AcceptUnderlay(underlay) + } else { + // Create new multi-channel if first connection + isFirst, _ := Headers(underlay.Headers()).GetBoolHeader(IsFirstGroupConnection) + if !isFirst { + // Reject - not first connection but no existing channel + underlay.Close() + return + } + + // Create new multi-channel + mc, err := ml.multiChannelFactory(underlay, closeCallback) + if err == nil { + ml.channels[chId] = mc + } + } +} +``` + +## Security Considerations + +1. **Group Secret Validation**: All connections must provide the correct group secret +2. **First Connection Protection**: Only connections marked with `IsFirstGroupConnection` can create new groups +3. **Connection Limits**: Implementations may limit the number of underlays per group +4. **Timeout Handling**: Connections have deadlines during the handshake process + +## Error Conditions + +### Invalid Group Secret +- **Condition**: Provided group secret doesn't match existing group +- **Action**: Close underlay connection immediately +- **Error**: "incorrect group secret" + +### No Existing Group for Non-First Connection +- **Condition**: Connection lacks `IsFirstGroupConnection` but no group exists +- **Action**: Close underlay connection +- **Log**: "no existing channel found for underlay, but isFirstGroupConnection not set" + +### Channel Already Closed +- **Condition**: Attempting to add underlay to closed multi-channel +- **Action**: Close underlay connection +- **Error**: "multi-channel is closed" + +## Testing + +The test suite in `multi_test.go` demonstrates: +- Creating multi-channel with initial underlay +- Adding additional underlays of different types +- Message routing across multiple underlays +- Priority handling with different underlay types +- Connection failure and recovery scenarios + +## Underlay Types and Constraints + +### Underlay Constraints (multi.go:547-643) +```go +type UnderlayConstraints struct { + types map[string]underlayConstraint + applyInProgress atomic.Bool + lastDial concurrenz.AtomicValue[time.Time] +} + +type underlayConstraint struct { + numDesired int // Target number of connections + minAllowed int // Minimum required connections +} +``` + +Example constraint configuration: +```go +constraints.AddConstraint("default", 2, 1) // Want 2, need at least 1 +constraints.AddConstraint("priority", 1, 0) // Want 1, can have 0 +``` + +## Message Routing + +Messages are distributed across underlays based on: +1. **Priority**: Priority messages use priority underlays when available +2. **Load Balancing**: Default messages distributed across available underlays +3. **Retry Logic**: Failed sends are retried on different underlays +4. **Type Affinity**: Different underlay types can handle different message flows + +This multi-underlay architecture provides robust, scalable communication with automatic failover and load distribution capabilities.