Skip to content

Commit bcc0f43

Browse files
IshwarKanseIshwarKanse
committed
Add documentation for OpenShift tests
Assisted by Cursor IDE using Claude AI model.
1 parent 5ddc440 commit bcc0f43

File tree

9 files changed

+1447
-0
lines changed

9 files changed

+1447
-0
lines changed

‎tests/e2e-openshift/README.md‎

Lines changed: 164 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,164 @@
1+
# OpenTelemetry Operator OpenShift End-to-End Test Suite
2+
3+
This directory contains a comprehensive set of OpenShift-specific end-to-end tests for the OpenTelemetry Operator. These tests serve as **configuration blueprints** for users to understand and deploy various OpenTelemetry observability patterns on OpenShift.
4+
5+
## 🎯 Purpose
6+
7+
These test scenarios provide OpenTelemetry configuration blueprints that demonstrate:
8+
- Integration with OpenShift-specific features (Routes, Monitoring, Security)
9+
- Real-world observability patterns and configurations
10+
- Step-by-step deployment instructions for various use cases
11+
12+
## 📋 Test Scenarios Overview
13+
14+
| Scenario | Purpose | Key Features |
15+
|----------|---------|-------------|
16+
| [route](./route/) | External Access via OpenShift Routes | Route ingress, OTLP HTTP/gRPC endpoints |
17+
| [scrape-in-cluster-monitoring](./scrape-in-cluster-monitoring/) | Prometheus Metrics Federation | In-cluster monitoring integration, metrics scraping |
18+
| [otlp-metrics-traces](./otlp-metrics-traces/) | OTLP Endpoint with Tempo | Metrics & traces collection, Tempo integration |
19+
| [multi-cluster](./multi-cluster/) | Secure Multi-Cluster Communication | TLS certificates, cross-cluster telemetry |
20+
| [must-gather](./must-gather/) | Diagnostic Information Collection | Must-gather functionality, target allocator |
21+
| [monitoring](./monitoring/) | Platform Monitoring Integration | OpenShift monitoring stack integration |
22+
| [kafka](./kafka/) | Messaging Layer for Telemetry | Kafka-based telemetry distribution |
23+
| [export-to-cluster-logging-lokistack](./export-to-cluster-logging-lokistack/) | Log Export to LokiStack | Log shipping to OpenShift logging |
24+
25+
## 🔗 OpenTelemetry Collector Components Tests
26+
27+
For detailed component-specific configurations and testing patterns, see the **OpenTelemetry Component E2E Test Suite** in the [distributed-tracing-qe](https://github.com/openshift/distributed-tracing-qe.git) repository:
28+
29+
**📡 Receivers:**
30+
- [filelog](https://github.com/openshift/distributed-tracing-qe/tree/main/tests/e2e-otel/filelog) - File-based log collection from Kubernetes pods
31+
- [hostmetricsreceiver](https://github.com/openshift/distributed-tracing-qe/tree/main/tests/e2e-otel/hostmetricsreceiver) - Host system metrics (CPU, memory, disk, network)
32+
- [journaldreceiver](https://github.com/openshift/distributed-tracing-qe/tree/main/tests/e2e-otel/journaldreceiver) - Systemd journal log collection
33+
- [k8sclusterreceiver](https://github.com/openshift/distributed-tracing-qe/tree/main/tests/e2e-otel/k8sclusterreceiver) - Kubernetes cluster-wide metrics
34+
- [k8seventsreceiver](https://github.com/openshift/distributed-tracing-qe/tree/main/tests/e2e-otel/k8seventsreceiver) - Kubernetes events collection
35+
- [k8sobjectsreceiver](https://github.com/openshift/distributed-tracing-qe/tree/main/tests/e2e-otel/k8sobjectsreceiver) - Kubernetes objects monitoring
36+
- [kubeletstatsreceiver](https://github.com/openshift/distributed-tracing-qe/tree/main/tests/e2e-otel/kubeletstatsreceiver) - Kubelet and container metrics
37+
- [otlpjsonfilereceiver](https://github.com/openshift/distributed-tracing-qe/tree/main/tests/e2e-otel/otlpjsonfilereceiver) - OTLP JSON file log ingestion
38+
39+
**📤 Exporters:**
40+
- [awscloudwatchlogsexporter](https://github.com/openshift/distributed-tracing-qe/tree/main/tests/e2e-otel/awscloudwatchlogsexporter) - AWS CloudWatch Logs integration
41+
- [awsxrayexporter](https://github.com/openshift/distributed-tracing-qe/tree/main/tests/e2e-otel/awsxrayexporter) - AWS X-Ray tracing export
42+
- [googlemanagedprometheusexporter](https://github.com/openshift/distributed-tracing-qe/tree/main/tests/e2e-otel/googlemanagedprometheusexporter) - Google Cloud Managed Prometheus
43+
- [loadbalancingexporter](https://github.com/openshift/distributed-tracing-qe/tree/main/tests/e2e-otel/loadbalancingexporter) - High availability load balancing
44+
- [prometheusremotewriteexporter](https://github.com/openshift/distributed-tracing-qe/tree/main/tests/e2e-otel/prometheusremotewriteexporter) - Prometheus remote write integration
45+
46+
**⚙️ Processors:**
47+
- [batchprocessor](https://github.com/openshift/distributed-tracing-qe/tree/main/tests/e2e-otel/batchprocessor) - Batching for performance optimization
48+
- [filterprocessor](https://github.com/openshift/distributed-tracing-qe/tree/main/tests/e2e-otel/filterprocessor) - Selective data filtering
49+
- [groupbyattrsprocessor](https://github.com/openshift/distributed-tracing-qe/tree/main/tests/e2e-otel/groupbyattrsprocessor) - Attribute-based data grouping
50+
- [memorylimiterprocessor](https://github.com/openshift/distributed-tracing-qe/tree/main/tests/e2e-otel/memorylimiterprocessor) - Memory usage protection
51+
- [resourceprocessor](https://github.com/openshift/distributed-tracing-qe/tree/main/tests/e2e-otel/resourceprocessor) - Resource attribute manipulation
52+
- [tailsamplingprocessor](https://github.com/openshift/distributed-tracing-qe/tree/main/tests/e2e-otel/tailsamplingprocessor) - Intelligent trace sampling
53+
- [transformprocessor](https://github.com/openshift/distributed-tracing-qe/tree/main/tests/e2e-otel/transformprocessor) - Advanced data transformation
54+
55+
**🔗 Connectors:**
56+
- [forwardconnector](https://github.com/openshift/distributed-tracing-qe/tree/main/tests/e2e-otel/forwardconnector) - Data forwarding between pipelines
57+
- [routingconnector](https://github.com/openshift/distributed-tracing-qe/tree/main/tests/e2e-otel/routingconnector) - Conditional data routing
58+
- [countconnector](https://github.com/openshift/distributed-tracing-qe/tree/main/tests/e2e-otel/countconnector) - Metrics generation from telemetry data
59+
60+
**🔧 Extensions:**
61+
- [oidcauthextension](https://github.com/openshift/distributed-tracing-qe/tree/main/tests/e2e-otel/oidcauthextension) - OIDC authentication
62+
- [filestorageextension](https://github.com/openshift/distributed-tracing-qe/tree/main/tests/e2e-otel/filestorageextension) - Persistent file storage
63+
64+
These component test blueprints provide configurations for individual OpenTelemetry components that can be combined with the OpenShift integration patterns documented here.
65+
66+
## 🚀 Quick Start
67+
68+
### Prerequisites
69+
70+
- OpenShift cluster (4.12+)
71+
- OpenTelemetry Operator installed
72+
- `oc` CLI tool configured
73+
- Appropriate cluster permissions
74+
75+
### Running Tests
76+
77+
These tests use [Chainsaw](https://kyverno.github.io/chainsaw/) for end-to-end testing:
78+
79+
```bash
80+
# Run all OpenShift tests
81+
chainsaw test --test-dir tests/e2e-openshift/
82+
83+
# Run specific test scenario
84+
chainsaw test --test-dir tests/e2e-openshift/route/
85+
```
86+
87+
### Using as Configuration Templates
88+
89+
Each test directory contains:
90+
- **Configuration Files**: YAML configuration blueprints
91+
- **README.md**: Step-by-step deployment instructions
92+
- **Scripts**: Verification and setup automation
93+
94+
## 📁 Directory Structure
95+
96+
```
97+
tests/e2e-openshift/
98+
├── README.md # This overview
99+
├── route/ # External access patterns
100+
├── scrape-in-cluster-monitoring/ # Prometheus integration
101+
├── otlp-metrics-traces/ # OTLP with Tempo
102+
├── multi-cluster/ # Cross-cluster telemetry
103+
├── must-gather/ # Diagnostic collection
104+
├── monitoring/ # Platform monitoring
105+
├── kafka/ # Messaging patterns
106+
└── export-to-cluster-logging-lokistack/ # Log export patterns
107+
```
108+
109+
## 🔧 Configuration Patterns
110+
111+
### Common OpenShift Integrations
112+
113+
1. **Security Context Constraints (SCCs)**
114+
- Automated SCC annotations for namespaces
115+
- Service account configurations
116+
117+
2. **OpenShift Routes**
118+
- TLS termination options
119+
- External endpoint exposure
120+
121+
3. **Monitoring Stack Integration**
122+
- Prometheus federation
123+
- Platform monitoring labels
124+
125+
4. **RBAC Configurations**
126+
- Cluster roles and bindings
127+
- Service account permissions
128+
129+
## 📖 Documentation
130+
131+
Each test scenario includes:
132+
- **Configuration blueprints** for reference and adaptation
133+
- **Step-by-step instructions** for manual deployment
134+
- **Verification steps** to ensure proper operation
135+
- **Troubleshooting guidance** for common issues
136+
137+
## 🏷️ Labels and Annotations
138+
139+
OpenShift-specific labels and annotations used across scenarios:
140+
- `openshift.io/cluster-monitoring=true` - Enable platform monitoring
141+
- `openshift.io/sa.scc.uid-range` - UID range for security contexts
142+
- `openshift.io/sa.scc.supplemental-groups` - Supplemental groups for SCCs
143+
144+
## 🤝 Contributing
145+
146+
When adding new test scenarios:
147+
1. Include comprehensive README with step-by-step instructions
148+
2. Provide configuration blueprint examples
149+
3. Add verification scripts for testing
150+
4. Document OpenShift-specific considerations
151+
152+
## 📝 Documentation Note
153+
154+
The comprehensive READMEs in this test suite were generated using Claude AI to provide detailed, step-by-step configuration blueprints for OpenTelemetry deployments on OpenShift. These AI-generated guides aim to accelerate user adoption by providing clear, actionable documentation for complex observability scenarios.
155+
156+
To regenerate / update the docs, the following prompt can be used.
157+
158+
> To enhance our OpenShift End-to-End (E2E) tests, we need to create comprehensive README files within the tests/e2e-openshift directory. These READMEs should provide a maintained set of OpenTelemetry configuration blueprints to assist users in easily deploying and configuring their observability stack, enabling them to quickly access and learn from out-of-the-box observability collection patterns. Each README must include step-by-step instructions referenced directly from the test cases, citing test resources and scripts. Since the test cases are written using the Chainsaw E2E testing tool, the READMEs should be designed from a user perspective for clear and easy follow-through.
159+
160+
## 📚 Additional Resources
161+
162+
- [OpenTelemetry Operator Documentation](https://github.com/open-telemetry/opentelemetry-operator)
163+
- [OpenShift Documentation](https://docs.openshift.com/)
164+
- [Chainsaw Testing Framework](https://kyverno.github.io/chainsaw/)

‎tests/e2e-openshift/export-to-cluster-logging-lokistack/README.md‎

Lines changed: 203 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,203 @@
1+
# Export to Cluster Logging LokiStack Test
2+
3+
This test demonstrates how to export OpenTelemetry logs to OpenShift's cluster logging infrastructure using LokiStack for centralized log management and analysis.
4+
5+
## Test Overview
6+
7+
This test creates:
8+
1. A MinIO instance for LokiStack object storage
9+
2. A LokiStack instance for log storage and querying
10+
3. An OpenTelemetry Collector that processes and exports logs to LokiStack
11+
4. Log generation to test the end-to-end flow
12+
5. Integration with OpenShift logging UI plugin
13+
14+
## Prerequisites
15+
16+
- OpenShift cluster (4.12+)
17+
- OpenTelemetry Operator installed
18+
- Red Hat OpenShift Logging Operator installed
19+
- `oc` CLI tool configured
20+
- Appropriate cluster permissions
21+
22+
## Configuration Resources
23+
24+
### MinIO Object Storage
25+
26+
Deploy MinIO for LokiStack storage backend:
27+
28+
**Configuration:** [`install-minio.yaml`](./install-minio.yaml)
29+
30+
Creates MinIO infrastructure:
31+
- 2Gi PersistentVolumeClaim for storage
32+
- MinIO deployment with demo credentials (tempo/supersecret)
33+
- Service for internal cluster access
34+
- Secret with S3-compatible access configuration
35+
36+
### LokiStack Instance
37+
38+
Deploy LokiStack for log storage:
39+
40+
**Configuration:** [`install-loki.yaml`](./install-loki.yaml)
41+
42+
Creates a LokiStack with:
43+
- 1x.demo size for testing environments
44+
- S3-compatible storage via MinIO
45+
- v13 schema with openshift-logging tenant mode
46+
- Integration with cluster logging infrastructure
47+
48+
### OpenTelemetry Collector Configuration
49+
50+
Deploy collector with LokiStack integration:
51+
52+
**Configuration:** [`otel-collector.yaml`](./otel-collector.yaml)
53+
54+
Configures a complete collector setup with:
55+
- Service account with LokiStack write permissions
56+
- ClusterRole for accessing pods, namespaces, and nodes
57+
- Bearer token authentication for LokiStack gateway
58+
- k8sattributes processor for Kubernetes metadata
59+
- Transform processor for log level normalization
60+
- Dual pipeline: one for LokiStack export, one for debug output
61+
62+
### Log Generation
63+
64+
Generate test logs to validate the pipeline:
65+
66+
**Configuration:** [`generate-logs.yaml`](./generate-logs.yaml)
67+
68+
Creates a job that:
69+
- Generates 20 structured log entries in OTLP format
70+
- Sends logs via HTTP POST to the collector
71+
- Includes service metadata and custom attributes
72+
- Uses proper OTLP JSON structure for compatibility
73+
74+
### Logging UI Plugin
75+
76+
Enable the logging UI plugin for log visualization:
77+
78+
**Configuration:** [`logging-uiplugin.yaml`](./logging-uiplugin.yaml)
79+
80+
Configures the OpenShift console plugin for:
81+
- Log visualization in the OpenShift web console
82+
- Integration with LokiStack for log querying
83+
- Enhanced logging user interface experience
84+
85+
## Deployment Steps
86+
87+
1. **Install MinIO for object storage:**
88+
```bash
89+
oc apply -f install-minio.yaml
90+
```
91+
92+
2. **Deploy LokiStack instance:**
93+
```bash
94+
oc apply -f install-loki.yaml
95+
```
96+
97+
3. **Deploy OpenTelemetry Collector with RBAC:**
98+
```bash
99+
oc apply -f otel-collector.yaml
100+
```
101+
102+
4. **Enable logging UI plugin:**
103+
```bash
104+
oc apply -f logging-uiplugin.yaml
105+
```
106+
107+
5. **Generate test logs:**
108+
```bash
109+
oc apply -f generate-logs.yaml
110+
```
111+
112+
## Expected Resources
113+
114+
The test creates and verifies these resources:
115+
116+
### Storage Infrastructure
117+
- **MinIO**: Object storage backend with `tempo` bucket
118+
- **PVC**: 2Gi persistent volume for MinIO storage
119+
- **Secret**: `logging-loki-s3` with MinIO access credentials
120+
121+
### Logging Stack
122+
- **LokiStack**: `logging-loki` instance in demo mode
123+
- **Gateway**: HTTP gateway for log ingestion
124+
- **Storage Schema**: v13 schema with 2023-10-15 effective date
125+
126+
### OpenTelemetry Integration
127+
- **Service Account**: `otel-collector-deployment` with logging permissions
128+
- **Collector**: `otel-collector` with LokiStack exporter
129+
- **RBAC**: Cluster role for writing to LokiStack
130+
131+
### Log Generation
132+
- **Job**: `generate-logs` creating test log entries
133+
- **UI Plugin**: Logging view plugin for OpenShift console
134+
135+
## Testing the Configuration
136+
137+
The test includes verification logic in the Chainsaw test configuration.
138+
139+
**Verification Script:** [`check_logs.sh`](./check_logs.sh)
140+
141+
The script verifies:
142+
- Log generation job completes successfully
143+
- Collector exports logs to LokiStack gateway
144+
- LokiStack gateway receives and processes log entries
145+
- End-to-end log flow from generation to storage
146+
147+
## Additional Verification Commands
148+
149+
Verify the logging infrastructure:
150+
151+
```bash
152+
# Check LokiStack status
153+
oc get lokistack logging-loki -o yaml
154+
155+
# Check MinIO deployment
156+
oc get deployment minio -o yaml
157+
158+
# View LokiStack gateway service
159+
oc get svc logging-loki-gateway-http
160+
161+
# Check collector service account permissions
162+
oc auth can-i create application --as=system:serviceaccount:openshift-logging:otel-collector-deployment
163+
164+
# Port forward to MinIO for bucket verification
165+
oc port-forward svc/minio 9000:9000 &
166+
curl -u tempo:supersecret http://localhost:9000/minio/health/live # Using demo test credentials
167+
168+
# Check collector metrics
169+
oc port-forward svc/otel-collector 8888:8888 &
170+
curl http://localhost:8888/metrics | grep otelcol_exporter
171+
```
172+
173+
## Verification
174+
175+
The test verifies:
176+
- ✅ MinIO is deployed and accessible as object storage
177+
- ✅ LokiStack instance is ready and configured
178+
- ✅ OpenTelemetry Collector has proper RBAC permissions
179+
- ✅ Collector is configured with LokiStack OTLP exporter
180+
- ✅ Bearer token authentication is working
181+
- ✅ Log generation job completes successfully
182+
- ✅ Logs are processed through k8sattributes and transform processors
183+
- ✅ Logs are successfully exported to LokiStack
184+
- ✅ Logging UI plugin is enabled for log visualization
185+
186+
## Key Features
187+
188+
- **LokiStack Integration**: Native integration with OpenShift cluster logging
189+
- **Object Storage**: MinIO backend for log persistence
190+
- **Authentication**: Bearer token authentication with service accounts
191+
- **Log Processing**: Kubernetes attributes and log transformation
192+
- **OTLP Protocol**: Uses OTLP HTTP for log export to LokiStack
193+
- **Multi-Pipeline**: Separate pipelines for LokiStack export and debug output
194+
- **UI Integration**: Logging console plugin for log visualization
195+
196+
## Configuration Notes
197+
198+
- LokiStack runs in `1x.demo` size for testing environments
199+
- MinIO uses ephemeral storage with demo credentials (tempo/supersecret) - **FOR TESTING ONLY**
200+
- Collector uses bearer token authentication for LokiStack access
201+
- Service CA certificate is used for TLS communication with LokiStack
202+
- Log processors add Kubernetes metadata and normalize severity levels
203+
- The application log type is set for proper tenant routing in LokiStack

0 commit comments

Comments
 (0)