Skip to content

[Chore] Add e2e test case for multi-cluster by exposing OTEL collector.#1956

Merged
pavolloffay merged 2 commits intoopen-telemetry:mainfrom
IshwarKanse:multi-cluster
Oct 9, 2023
Merged

[Chore] Add e2e test case for multi-cluster by exposing OTEL collector.#1956
pavolloffay merged 2 commits intoopen-telemetry:mainfrom
IshwarKanse:multi-cluster

Conversation

@IshwarKanse
Copy link
Contributor

@IshwarKanse IshwarKanse commented Jul 24, 2023

Adds e2e test case for multi-cluster by exposing OTEL collector.

@IshwarKanse IshwarKanse requested a review from a team July 24, 2023 11:20
@IshwarKanse IshwarKanse force-pushed the multi-cluster branch 2 times, most recently from 5e6583f to 9c2e0fd Compare August 17, 2023 15:05
@jaronoff97
Copy link
Contributor

jaronoff97 commented Aug 18, 2023

@IshwarKanse can you add this to the e2e.yml in the .github directory so this runs?

@IshwarKanse
Copy link
Contributor Author

IshwarKanse commented Aug 24, 2023

@jaronoff97 This test requires a OpenShift cluster to run, it depends on several operators like Jaeger and cert-manager. Its still in WIP cause there are several issues we detected exposing the route for OTEL. I'll update the test case once the issues are fixed and will check if we can get it running on Kind clusters.

@jaronoff97 jaronoff97 requested a review from pavolloffay August 24, 2023 14:47
@jaronoff97
Copy link
Contributor

jaronoff97 commented Aug 24, 2023

Thanks for letting me know, i tagged Pavol on this PR as he has much more context on OpenShift things :)

@IshwarKanse IshwarKanse changed the title [WIP] [Chore] Add e2e test case for multi-cluster by exposing OTEL collector. [Chore] Add e2e test case for multi-cluster by exposing OTEL collector. Sep 7, 2023
@IshwarKanse
Copy link
Contributor Author

IshwarKanse commented Sep 7, 2023 •
edited
Loading 

2023/09/07 11:47:05 kutt-test config testdirs is overridden with args: [ tests/e2e-openshift ]
=== RUN   kuttl
    harness.go:462: starting setup
    harness.go:252: running tests using configured kubeconfig.
    harness.go:275: Successful connection to cluster at:  https://REDACTED:6443
    harness.go:360: running tests
    harness.go:73: going to run test suite with timeout of 300 seconds for each step
    harness.go:372: testsuite: tests/e2e-openshift has 1 tests
=== RUN   kuttl/harness
=== RUN   kuttl/harness/multi-cluster
=== PAUSE kuttl/harness/multi-cluster
=== CONT  kuttl/harness/multi-cluster
    logger.go:42: 11:47:09 | multi-cluster | Ignoring check_traces.sh as it does not match file name regexp: ^(\d+)-(?:[^\.]+)(?:\.yaml)?$
    logger.go:42: 11:47:09 | multi-cluster | Ignoring create_otlp_receiver.sh as it does not match file name regexp: ^(\d+)-(?:[^\.]+)(?:\.yaml)?$
    logger.go:42: 11:47:09 | multi-cluster | Ignoring create_otlp_sender.sh as it does not match file name regexp: ^(\d+)-(?:[^\.]+)(?:\.yaml)?$
    logger.go:42: 11:47:09 | multi-cluster | Ignoring generate_certs.sh as it does not match file name regexp: ^(\d+)-(?:[^\.]+)(?:\.yaml)?$
    logger.go:42: 11:47:09 | multi-cluster | Creating namespace: kuttl-test-subtle-lobster
    logger.go:42: 11:47:09 | multi-cluster/0-create-namespaces | starting test step 0-create-namespaces
    logger.go:42: 11:47:10 | multi-cluster/0-create-namespaces | Namespace:/kuttl-multi-cluster-send created
    logger.go:42: 11:47:11 | multi-cluster/0-create-namespaces | Namespace:/kuttl-multi-cluster-receive created
    logger.go:42: 11:47:12 | multi-cluster/0-create-namespaces | test step completed 0-create-namespaces
    logger.go:42: 11:47:12 | multi-cluster/1-create-jaeger | starting test step 1-create-jaeger
    logger.go:42: 11:47:13 | multi-cluster/1-create-jaeger | Jaeger:kuttl-multi-cluster-receive/jaeger-allinone created
    logger.go:42: 11:47:23 | multi-cluster/1-create-jaeger | test step completed 1-create-jaeger
    logger.go:42: 11:47:23 | multi-cluster/2-otlp-receiver | starting test step 2-otlp-receiver
    logger.go:42: 11:47:23 | multi-cluster/2-otlp-receiver | running command: [sh -c ./generate_certs.sh]
    logger.go:42: 11:47:25 | multi-cluster/2-otlp-receiver | ...+...........+...+...+...+.+...+..+.........+.+.........+.....+.......+..+...+....+...+.....+..........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.................+..+...+.+......+..+.+..............+.+.....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.......+.....+.......+..............+.+...+......+.....+.+.........+......+.....+...+...+.........+.........+......+................+..+.+......+.........+.....+.+........+......+......+.........+.+.....................+........+......+.+.....+.........+..........+...+..................+..............+...+.+.....+.+.....+.+...+..+.......+...+..+......+......+......+.........+.............+...............+...+.........+........+......+............+.......+...+..+...+.........+.+..............+...+.+......+.....+..........+...+.....+...............+.+............+..+.+..+...+......+.+..+...+.............+.........+...+.........+...+..+......+....+...+......+..............+.+...........+...+............+......+..........+..................+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
    logger.go:42: 11:47:25 | multi-cluster/2-otlp-receiver | ..+......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*..+...+..+.........+...+.......+...+.....+...+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*........+...+......+..........+...+...+.....+.......+..+....+...........+...+...+.......+.....+.............+..+............+.+.....+...+....+........+...+.+.........+.....................+.......................+...+.......+...+...+......+........+...+...+......+.+.........+...........+...+....+...+..+...+....+........+......+....+...........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
    logger.go:42: 11:47:25 | multi-cluster/2-otlp-receiver | Certificate request self-signature ok
    logger.go:42: 11:47:25 | multi-cluster/2-otlp-receiver | subject=C = US, ST = California, L = San Francisco, O = My Organization, CN = *.apps.ikanse-99.qe.devcluster.openshift.com
    logger.go:42: 11:47:25 | multi-cluster/2-otlp-receiver | Certificates generated successfully in /tmp/kuttl-certs directory.
    logger.go:42: 11:47:26 | multi-cluster/2-otlp-receiver | Error from server (NotFound): configmaps "kuttl-certs" not found
    logger.go:42: 11:47:27 | multi-cluster/2-otlp-receiver | Error from server (NotFound): configmaps "kuttl-certs" not found
    logger.go:42: 11:47:28 | multi-cluster/2-otlp-receiver | configmap/kuttl-certs created
    logger.go:42: 11:47:29 | multi-cluster/2-otlp-receiver | configmap/kuttl-certs created
    logger.go:42: 11:47:29 | multi-cluster/2-otlp-receiver | ConfigMaps created successfully.
    logger.go:42: 11:47:29 | multi-cluster/2-otlp-receiver | running command: [sh -c ./create_otlp_receiver.sh]
    logger.go:42: 11:47:31 | multi-cluster/2-otlp-receiver | opentelemetrycollector.opentelemetry.io/otlp-receiver created
    logger.go:42: 11:47:38 | multi-cluster/2-otlp-receiver | test step completed 2-otlp-receiver
    logger.go:42: 11:47:38 | multi-cluster/3-otlp-sender | starting test step 3-otlp-sender
    logger.go:42: 11:47:38 | multi-cluster/3-otlp-sender | running command: [sh -c ./create_otlp_sender.sh]
    logger.go:42: 11:47:41 | multi-cluster/3-otlp-sender | opentelemetrycollector.opentelemetry.io/otel-sender created
    logger.go:42: 11:47:43 | multi-cluster/3-otlp-sender | ServiceAccount:kuttl-multi-cluster-send/kuttl-multi-cluster created
    logger.go:42: 11:47:43 | multi-cluster/3-otlp-sender | ClusterRole:/kuttl-multi-cluster updated
    logger.go:42: 11:47:44 | multi-cluster/3-otlp-sender | ClusterRoleBinding:/kuttl-multi-cluster updated
    logger.go:42: 11:47:49 | multi-cluster/3-otlp-sender | test step completed 3-otlp-sender
    logger.go:42: 11:47:49 | multi-cluster/4-generate-traces | starting test step 4-generate-traces
    logger.go:42: 11:47:51 | multi-cluster/4-generate-traces | Job:kuttl-multi-cluster-send/generate-traces-http created
    logger.go:42: 11:47:51 | multi-cluster/4-generate-traces | Job:kuttl-multi-cluster-send/generate-traces-grpc created
    logger.go:42: 11:47:57 | multi-cluster/4-generate-traces | test step completed 4-generate-traces
    logger.go:42: 11:47:57 | multi-cluster/5- | starting test step 5-
    logger.go:42: 11:47:59 | multi-cluster/5- | running command: [sh -c ./tests/e2e-openshift/multi-cluster/check_traces.sh]
    logger.go:42: 11:48:01 | multi-cluster/5- | Traces for telemetrygen-http exist in Jaeger.
    logger.go:42: 11:48:02 | multi-cluster/5- | Traces for telemetrygen-grpc exist in Jaeger.
    logger.go:42: 11:48:02 | multi-cluster/5- | test step completed 5-
    logger.go:42: 11:48:02 | multi-cluster | multi-cluster events from ns kuttl-test-subtle-lobster:
    logger.go:42: 11:48:04 | multi-cluster | Deleting namespace: kuttl-test-subtle-lobster
=== CONT  kuttl
    harness.go:405: run tests finished
    harness.go:513: cleaning up
    harness.go:570: removing temp folder: ""
--- PASS: kuttl (65.29s)
    --- PASS: kuttl/harness (0.00s)
        --- PASS: kuttl/harness/multi-cluster (61.95s)
PASS

@IshwarKanse
Copy link
Contributor Author

IshwarKanse commented Sep 7, 2023

@frzifus @pavolloffay Can you review this test case.

Since we cannot use two clusters, the test case simulates by creating the sender and receiver collector in two separate namespaces.

*Sends traces to both OTLP HTTP and GRPC receiver exposed via route.
*Uses mTLS in collector config and the route is using passthrough termination.
*Checks the presence of traces in for both HTTP and GRPC.

frzifus
frzifus approved these changes Sep 12, 2023
View reviewed changes
Copy link
Member

@frzifus frzifus left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks rly good to me @IshwarKanse!

Since we generate our own certificates and jaeger is deployed without the operator, what would be missing to run this test in this github ci too?

Primary the routes? The last time I checked, they have been installed for some tests too.

@IshwarKanse
Copy link
Contributor Author

IshwarKanse commented Sep 21, 2023

cc @frzifus @pavolloffay Can you help to reapprove and get this one merged.

iblancasa
iblancasa approved these changes Sep 25, 2023
View reviewed changes
pavolloffay
pavolloffay reviewed Sep 29, 2023
View reviewed changes
tests/e2e-openshift/multi-cluster/create_otlp_receiver.sh Outdated
otlp:
protocols:
http:
tls:
Copy link
Member

@pavolloffay pavolloffay Sep 29, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we use edge termination or insecure routes?

The recent version of the operator improved support for routes:

  • hostname can be omitted
  • the grpc route works for insecure and edge termination

Copy link
Contributor Author

@IshwarKanse IshwarKanse Oct 3, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pavolloffay I have updated the test case to remove setting the hostname. But for setting custom certs, ran into this issue. https://issues.redhat.com/browse/TRACING-3609 For the time being, I've kept the termination as passthrough and using the custom certs.

@pavolloffay pavolloffay merged commit c6531de into open-telemetry:main Oct 9, 2023
@IshwarKanse IshwarKanse deleted the multi-cluster branch April 11, 2024 09:08
ItielOlenick pushed a commit to ItielOlenick/opentelemetry-operator that referenced this pull request May 1, 2024
@IshwarKanse
Add e2e test case for multi-cluster by exposing OTEL collector. (open…
6f56a8b 
…-telemetry#1956)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@iblancasa iblancasa iblancasa approved these changes

@pavolloffay pavolloffay pavolloffay approved these changes

@frzifus frzifus frzifus approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@IshwarKanse @jaronoff97 @iblancasa @pavolloffay @frzifus