Skip to content

Commit 9ccc8e7

Browse files
TheKongersTheKongers
committed
added corbench code without manifest files
1 parent 67dff51 commit 9ccc8e7

File tree

17 files changed

+3252
-1
lines changed

17 files changed

+3252
-1
lines changed

‎.gitignore

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -30,3 +30,9 @@ go.work.sum
3030
# Editor/IDE
3131
# .idea/
3232
# .vscode/
33+
34+
# Authentication file
35+
auth_file.yaml
36+
37+
# Infra binary
38+
corbench/infra/infra

‎README.md

Lines changed: 18 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,21 @@
11
# Corbench
22
Benchmarking tool for cortex
33

4-
Refer to [corbench/README.md](corbench/README.md)
4+
Refer to [corbench/README.md](corbench/README.md) for documetation on how to deploy the tool, and how the tool works.
5+
6+
## Tool Usage
7+
8+
Create a PR in the Cortex repository, and you are able to run the following commands in the comments of a PR
9+
10+
**Available Commands:**
11+
- To start benchmark: `/corbench <cortex-tag>`
12+
- To stop benchmark: `/corbench cancel`
13+
- To print help: `/corbench help`
14+
15+
**Cortex tag format:** `master-<commit-hash>` (e.g., `master-6b3bd7b`) This will be the version of cortex that your PR will be benchmarked against
16+
- MAKE SURE TO PICK AN EXISTING CORTEX TAG FROM BELOW LINK, OR DEPLOYMENT WILL BE STUCK AND YOU WILL HAVE TO CANCEL AND START AGAIN
17+
- Check what cortex tags are avaliable to choose from at https://hub.docker.com/r/cortexproject/cortex/tags
18+
19+
**Examples:**
20+
- `/corbench master-6b3bd7b`
21+
- `/corbench master-9861229`

‎corbench/Dockerfile

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
FROM --platform=linux/amd64 golang:1.15-alpine
2+
3+
WORKDIR /corbench
4+
5+
RUN apk add git make
6+
7+
# Copy Makefiles and manifests
8+
# Need 'cd' since ghActions ignores WORKDIR
9+
# Need 'eval' to prevent bash keywords be run as commands
10+
COPY ./ ./
11+
12+
RUN echo -e '#!/bin/sh\ncd /corbench\neval "$@"' >/bin/docker_entrypoint
13+
14+
RUN chmod u+x /bin/docker_entrypoint
15+
16+
ENTRYPOINT ["docker_entrypoint"]

‎corbench/Makefile

Lines changed: 105 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,105 @@
1+
INFRA_CMD ?= ./infra/infra
2+
PROVIDER ?= eks
3+
MANIFEST_FILE = c-manifests
4+
5+
cluster_create:
6+
${INFRA_CMD} ${PROVIDER} cluster create -a ${AUTH_FILE} \
7+
-v ZONE:${ZONE} -v GKE_PROJECT_ID:${GKE_PROJECT_ID} \
8+
-v EKS_WORKER_ROLE_ARN:${EKS_WORKER_ROLE_ARN} -v EKS_CLUSTER_ROLE_ARN:${EKS_CLUSTER_ROLE_ARN} \
9+
-v EKS_SUBNET_IDS:${EKS_SUBNET_IDS} -v SEPARATOR:${SEPARATOR} \
10+
-v CLUSTER_NAME:${CLUSTER_NAME} -v PR_NUMBER:${PR_NUMBER} \
11+
-v AWS_ACCOUNT_ID:$$(echo ${EKS_WORKER_ROLE_ARN} | cut -d':' -f5) \
12+
-f ${MANIFEST_FILE}/cluster_${PROVIDER}.yaml
13+
ifeq (${PROVIDER},eks)
14+
./setup-ebs-csi.sh
15+
endif
16+
17+
cluster_resource_apply:
18+
${INFRA_CMD} ${PROVIDER} resource apply -a ${AUTH_FILE} \
19+
-v ZONE:${ZONE} -v GKE_PROJECT_ID:${GKE_PROJECT_ID} \
20+
-v EKS_WORKER_ROLE_ARN:${EKS_WORKER_ROLE_ARN} -v EKS_CLUSTER_ROLE_ARN:${EKS_CLUSTER_ROLE_ARN} \
21+
-v EKS_SUBNET_IDS:${EKS_SUBNET_IDS} -v SEPARATOR:${SEPARATOR} \
22+
-v CLUSTER_NAME:${CLUSTER_NAME} -v PR_NUMBER:${PR_NUMBER} -v DOMAIN_NAME:${DOMAIN_NAME} -v RELEASE:${RELEASE} \
23+
-v GRAFANA_ADMIN_PASSWORD:${GRAFANA_ADMIN_PASSWORD} \
24+
-v SERVICEACCOUNT_CLIENT_EMAIL:${SERVICEACCOUNT_CLIENT_EMAIL} \
25+
-v OAUTH_TOKEN="$(printf ${OAUTH_TOKEN} | base64 -w 0)" \
26+
-v WH_SECRET="$(printf ${WH_SECRET} | base64 -w 0)" \
27+
-v GITHUB_ORG:${GITHUB_ORG} -v GITHUB_REPO:${GITHUB_REPO} \
28+
-f ${MANIFEST_FILE}/cluster-infra
29+
# Patch secrets immediately after deployment with correct values since for some reason 1_secrets isn't working properly
30+
export AWS_ACCESS_KEY_ID=$$(grep "accesskeyid:" ${AUTH_FILE} | awk '{print $$2}') && \
31+
export AWS_SECRET_ACCESS_KEY=$$(grep "secretaccesskey:" ${AUTH_FILE} | awk '{print $$2}') && \
32+
kubectl patch secret oauth-token -p '{"data":{"oauth":"'$$(printf "${OAUTH_TOKEN}" | base64)'"}}' && \
33+
kubectl patch secret whsecret -p '{"data":{"whsecret":"'$$(printf "${WH_SECRET}" | base64)'"}}' && \
34+
kubectl rollout restart deployment/comment-monitor
35+
36+
cluster_delete:
37+
${INFRA_CMD} ${PROVIDER} cluster delete -a ${AUTH_FILE} \
38+
-v ZONE:${ZONE} -v GKE_PROJECT_ID:${GKE_PROJECT_ID} \
39+
-v EKS_WORKER_ROLE_ARN:${EKS_WORKER_ROLE_ARN} -v EKS_CLUSTER_ROLE_ARN:${EKS_CLUSTER_ROLE_ARN} \
40+
-v EKS_SUBNET_IDS:${EKS_SUBNET_IDS} -v SEPARATOR:${SEPARATOR} \
41+
-v CLUSTER_NAME:${CLUSTER_NAME} -v PR_NUMBER:${PR_NUMBER} \
42+
-v AWS_ACCOUNT_ID:$$(echo ${EKS_WORKER_ROLE_ARN} | cut -d':' -f5) \
43+
-f ${MANIFEST_FILE}/cluster_${PROVIDER}.yaml
44+
45+
46+
BENCHMARK_DIRECTORY := $(if $(BENCHMARK_DIRECTORY),$(BENCHMARK_DIRECTORY),c-manifests/benchmarks)
47+
48+
CORBENCH_DIR ?= .
49+
50+
.PHONY: deploy
51+
deploy: node_create resource_apply
52+
53+
.PHONY: clean
54+
clean: resource_delete node_delete
55+
56+
# Default PR_NUMBER to 'default' if not set to avoid invalid nodegroup names
57+
PR_NUMBER ?= default
58+
59+
node_create:
60+
${INFRA_CMD} ${PROVIDER} nodes create -a ${AUTH_FILE} \
61+
-v ZONE:${ZONE} -v GKE_PROJECT_ID:${GKE_PROJECT_ID} \
62+
-v EKS_WORKER_ROLE_ARN:${EKS_WORKER_ROLE_ARN} -v EKS_CLUSTER_ROLE_ARN:${EKS_CLUSTER_ROLE_ARN} \
63+
-v EKS_SUBNET_IDS:${EKS_SUBNET_IDS} \
64+
-v CLUSTER_NAME:${CLUSTER_NAME} -v PR_NUMBER:${PR_NUMBER} \
65+
-f ${CORBENCH_DIR}/${BENCHMARK_DIRECTORY}/nodes_${PROVIDER}.yaml
66+
67+
resource_apply:
68+
$(INFRA_CMD) ${PROVIDER} resource apply -a ${AUTH_FILE} \
69+
-v ZONE:${ZONE} -v GKE_PROJECT_ID:${GKE_PROJECT_ID} \
70+
-v CLUSTER_NAME:${CLUSTER_NAME} \
71+
-v PR_NUMBER:${PR_NUMBER} -v RELEASE:${RELEASE} -v DOMAIN_NAME:${DOMAIN_NAME} \
72+
-v GITHUB_ORG:${GITHUB_ORG} -v GITHUB_REPO:${GITHUB_REPO} \
73+
-f ${CORBENCH_DIR}/${BENCHMARK_DIRECTORY}/remote-write
74+
75+
# Required because namespace and cluster-role are not part of the created nodes
76+
resource_delete:
77+
$(INFRA_CMD) ${PROVIDER} resource delete -a ${AUTH_FILE} \
78+
-v ZONE:${ZONE} -v GKE_PROJECT_ID:${GKE_PROJECT_ID} \
79+
-v CLUSTER_NAME:${CLUSTER_NAME} -v PR_NUMBER:${PR_NUMBER} \
80+
-f ${CORBENCH_DIR}/${BENCHMARK_DIRECTORY}/remote-write/1c_cluster-role-binding.yaml \
81+
-f ${CORBENCH_DIR}/${BENCHMARK_DIRECTORY}/remote-write/1a_namespace.yaml
82+
83+
node_delete:
84+
$(INFRA_CMD) ${PROVIDER} nodes delete -a ${AUTH_FILE} \
85+
-v ZONE:${ZONE} -v GKE_PROJECT_ID:${GKE_PROJECT_ID} \
86+
-v EKS_WORKER_ROLE_ARN:${EKS_WORKER_ROLE_ARN} -v EKS_CLUSTER_ROLE_ARN:${EKS_CLUSTER_ROLE_ARN} \
87+
-v EKS_SUBNET_IDS:${EKS_SUBNET_IDS} \
88+
-v CLUSTER_NAME:${CLUSTER_NAME} -v PR_NUMBER:${PR_NUMBER} \
89+
-f ${CORBENCH_DIR}/${BENCHMARK_DIRECTORY}/nodes_${PROVIDER}.yaml
90+
91+
all_nodes_running:
92+
$(INFRA_CMD) ${PROVIDER} nodes check-running -a ${AUTH_FILE} \
93+
-v ZONE:${ZONE} -v GKE_PROJECT_ID:${GKE_PROJECT_ID} \
94+
-v EKS_WORKER_ROLE_ARN:${EKS_WORKER_ROLE_ARN} -v EKS_CLUSTER_ROLE_ARN:${EKS_CLUSTER_ROLE_ARN} \
95+
-v EKS_SUBNET_IDS:${EKS_SUBNET_IDS} -v SEPARATOR:${SEPARATOR} \
96+
-v CLUSTER_NAME:${CLUSTER_NAME} -v PR_NUMBER:${PR_NUMBER} \
97+
-f ${CORBENCH_DIR}/${BENCHMARK_DIRECTORY}/nodes_${PROVIDER}.yaml
98+
99+
all_nodes_deleted:
100+
$(INFRA_CMD) ${PROVIDER} nodes check-deleted -a ${AUTH_FILE} \
101+
-v ZONE:${ZONE} -v GKE_PROJECT_ID:${GKE_PROJECT_ID} \
102+
-v EKS_WORKER_ROLE_ARN:${EKS_WORKER_ROLE_ARN} -v EKS_CLUSTER_ROLE_ARN:${EKS_CLUSTER_ROLE_ARN} \
103+
-v EKS_SUBNET_IDS:${EKS_SUBNET_IDS} -v SEPARATOR:${SEPARATOR} \
104+
-v CLUSTER_NAME:${CLUSTER_NAME} -v PR_NUMBER:${PR_NUMBER} \
105+
-f ${CORBENCH_DIR}/${BENCHMARK_DIRECTORY}/nodes_${PROVIDER}.yaml

‎corbench/README.md

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
# Automated Cortex E2E Testing and Benchmarking
2+
3+
## Corbench Setup
4+
5+
To deploy this tool, follow the steps here: [Getting Started](docs/deployment.md)
6+
7+
# Archetecture
8+
9+
![Cortex Benchmarking Archetecture](docs/Blank%20diagram%20(2).png)
10+
11+
1. START: User opens a Github PR in the cortex repository and sends a /corbench PR comment to start a benchmark test. Github then makes a POST request with the command as the payload to the Comment Monitor Webhook Server (step 2).
12+
2. The Comment Monitor Webhook Server monitors for /corbench calls from Cortex github PRs. Upon recieving a POST request from a Github PR, the server verifies the validity of the command (i.e. does the command exist?). The server then initiates a dispatch event to the PR (step 3) as well as posts an update comment indicating the status of the tests, links to grafana dashboards, etc, or a warning message if the command was used incorrectly/with wrong syntax.
13+
3. The PR runs the dispatch event using the docker image of the Corbench repo code to initiate benchmarks (step 4).
14+
4. Within the Corbench docker image, two versions of Cortex that will be benchmarked are built & deployed in separate Kubernetes nodes to EKS. After this, we can start to deploy the metrics pushers (Avalanche). Avalanche instances will be deployed to EKS and will simulate prometheus instances pushing load to Cortex’s remote write (or other micro-service specific) endpoints with mock data.
15+
6. While the Metrics Pushers push load into the Cortex instances, prometheus will continuously scrape metrics from the Cortex instances and the kubernetes nodes they run on.
16+
7. Grafana displays collected benchmark metrics & github notifier notifies the original pr of results & other info

‎corbench/docs/Blank diagram (2).png

82 KB
Loading

0 commit comments

Comments
 (0)