|
| 1 | +# Logs and Metrics in RAM |
| 2 | + |
| 3 | +The SAS Retrieval Agent Manager (RAM) system collects and stores logs and metrics using [Vector](https://vector.dev/), a high-performance observability data pipeline. Vector aggregates telemetry data from Kubernetes clusters and routes it to PostgreSQL via PostgREST for persistent storage and querying. |
| 4 | + |
| 5 | +## Architecture Overview |
| 6 | + |
| 7 | +```text |
| 8 | +Kubernetes Logs / RAM APIMetrics → Vector → PostgREST → PostgreSQL |
| 9 | +``` |
| 10 | + |
| 11 | +Vector runs as a DaemonSet in the cluster, collecting: |
| 12 | + |
| 13 | +- **Logs**: Container logs from all pods via Kubernetes log files |
| 14 | + |
| 15 | +- **Metrics**: Performance metrics, resource usage, and custom application metrics |
| 16 | + |
| 17 | +## Configuration |
| 18 | + |
| 19 | +### Vector Pipeline Components |
| 20 | + |
| 21 | +The Vector configuration consists of three main components: |
| 22 | + |
| 23 | +1. **Sources**: Data collection from Kubernetes |
| 24 | +2. **Transforms**: Data processing and enrichment using VRL (Vector Remap Language) |
| 25 | +3. **Sinks**: Delivery to PostgREST endpoints |
| 26 | + |
| 27 | +### Logs Pipeline |
| 28 | + |
| 29 | +Vector collects Kubernetes pod logs and enriches them with metadata: |
| 30 | + |
| 31 | +```yaml |
| 32 | +sources: |
| 33 | + kube_logs: |
| 34 | + type: kubernetes_logs |
| 35 | + auto_partial_merge: true |
| 36 | + |
| 37 | +transforms: |
| 38 | + logs_transform: |
| 39 | + type: remap |
| 40 | + inputs: |
| 41 | + - kube_logs |
| 42 | + source: | |
| 43 | + # Remove fields not in database schema |
| 44 | + del(.source_type) |
| 45 | + del(.stream) |
| 46 | + |
| 47 | +sinks: |
| 48 | + logs_postgrest: |
| 49 | + type: http |
| 50 | + inputs: |
| 51 | + - logs_transform |
| 52 | + uri: "http://sas-retrieval-agent-manager-postgrest.retagentmgr.svc.cluster.local:3002/logs" |
| 53 | + encoding: |
| 54 | + codec: json |
| 55 | + method: post |
| 56 | +``` |
| 57 | +
|
| 58 | +> Note: See a full [Vector example values file here](../../examples/vector.yaml) |
| 59 | +
|
| 60 | +#### Log Schema |
| 61 | +
|
| 62 | +Logs are stored in PostgreSQL with the following schema: |
| 63 | +
|
| 64 | +| Column | Type | Description | |
| 65 | +|--------|------|-------------| |
| 66 | +| `file` | TEXT | Path to the log file in Kubernetes | |
| 67 | +| `kubernetes` | JSONB | Kubernetes metadata (pod, namespace, labels, etc.) | |
| 68 | +| `message` | TEXT | The actual log message | |
| 69 | +| `timestamp` | TIMESTAMPTZ | When the log entry was created | |
| 70 | + |
| 71 | +#### Kubernetes Metadata |
| 72 | + |
| 73 | +The `kubernetes` JSONB column includes the following context: |
| 74 | + |
| 75 | +- `pod_name`, `pod_namespace`, `pod_uid` |
| 76 | + |
| 77 | +- `container_name`, `container_image` |
| 78 | + |
| 79 | +- `node_labels` |
| 80 | + |
| 81 | +- `pod_labels` |
| 82 | + |
| 83 | +- `pod_ip`, `pod_owner` |
| 84 | + |
| 85 | + |
| 86 | +### Metrics Pipeline |
| 87 | + |
| 88 | +Metrics collection follows a similar pattern but does not need transformations: |
| 89 | + |
| 90 | +```yaml |
| 91 | +sources: |
| 92 | + otel: |
| 93 | + type: opentelemetry |
| 94 | + grpc: |
| 95 | + address: 0.0.0.0:4317 |
| 96 | + http: |
| 97 | + address: 0.0.0.0:4318 |
| 98 | +
|
| 99 | +sinks: |
| 100 | + metrics_postgrest: |
| 101 | + type: http |
| 102 | + inputs: |
| 103 | + - otel.metrics |
| 104 | + uri: "http://sas-retrieval-agent-manager-postgrest.retagentmgr.svc.cluster.local:3002/metrics" |
| 105 | + headers: |
| 106 | + Content-Type: "Application/json" |
| 107 | + encoding: |
| 108 | + codec: json |
| 109 | +``` |
| 110 | + |
| 111 | +## PostgREST Integration |
| 112 | + |
| 113 | +Vector sends data directly to PostgREST HTTP endpoints, which provides: |
| 114 | + |
| 115 | +- Automatic API generation from PostgreSQL schema |
| 116 | + |
| 117 | +- Role-based access control via PostgreSQL roles |
| 118 | + |
| 119 | +- JSON validation and type safety |
| 120 | + |
| 121 | +## Testing |
| 122 | + |
| 123 | +### Manual Log Injection |
| 124 | + |
| 125 | +Test the postgREST endpoint with a curl from within the cluster: |
| 126 | + |
| 127 | +```bash |
| 128 | +curl -X POST \ |
| 129 | + "http://sas-retrieval-agent-manager-postgrest.retagentmgr.svc.cluster.local:3002/logs" \ |
| 130 | + -H "Content-Type: application/json" \ |
| 131 | + -H "Prefer: return=representation" \ |
| 132 | + -d '{ |
| 133 | + "file": "/var/log/pods/test_pod/container/0.log", |
| 134 | + "kubernetes": { |
| 135 | + "container_name": "test-container", |
| 136 | + "pod_name": "test-pod", |
| 137 | + "pod_namespace": "default", |
| 138 | + "pod_uid": "test-uid-12345" |
| 139 | + }, |
| 140 | + "message": "Test log message", |
| 141 | + "timestamp": "2025-11-10T18:00:00.000000Z" |
| 142 | + }' |
| 143 | +``` |
| 144 | + |
| 145 | +### Verify Vector is Running |
| 146 | + |
| 147 | +```bash |
| 148 | +# Check Vector pods |
| 149 | +kubectl get pods -n vector |
| 150 | +
|
| 151 | +# View Vector logs |
| 152 | +kubectl logs -n vector -l app.kubernetes.io/name=vector --tail=100 |
| 153 | +
|
| 154 | +# Check for errors |
| 155 | +kubectl logs -n vector -l app.kubernetes.io/name=vector | grep ERROR |
| 156 | +``` |
| 157 | + |
| 158 | +## Troubleshooting |
| 159 | + |
| 160 | +### Common Issues |
| 161 | + |
| 162 | +#### 1. Schema Mismatch Errors |
| 163 | + |
| 164 | +**Error**: `Could not find the 'source_type' column` |
| 165 | + |
| 166 | +**Solution**: Add a VRL transform to remove fields not in your database schema: |
| 167 | + |
| 168 | +```yaml |
| 169 | +transforms: |
| 170 | + remove_extra_fields: |
| 171 | + type: remap |
| 172 | + inputs: |
| 173 | + - kube_logs |
| 174 | + source: | |
| 175 | + del(.source_type) |
| 176 | + del(.stream) |
| 177 | +``` |
| 178 | + |
| 179 | +#### 2. PostgREST Connection Failures |
| 180 | + |
| 181 | +**Error**: `Service call failed. No retries or retries exhausted` |
| 182 | + |
| 183 | +Check PostgREST is accessible: |
| 184 | + |
| 185 | +```bash |
| 186 | +
|
| 187 | +kubectl get svc -n retagentmgr sas-retrieval-agent-manager-postgrest |
| 188 | +kubectl get pods -n retagentmgr -l app.kubernetes.io/name=postgrest |
| 189 | +
|
| 190 | +``` |
| 191 | + |
| 192 | +## Related Documentation |
| 193 | + |
| 194 | +- [Vector Documentation](https://vector.dev/docs/) |
| 195 | +- [PostgREST API Reference](https://postgrest.org/en/stable/api.html) |
| 196 | +- [OpenTelemetry Specification](https://opentelemetry.io/docs/) |
| 197 | +- [VRL Language Reference](https://vector.dev/docs/reference/vrl/) |
0 commit comments