Concurrent jobs with configmaps #87
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,91 @@ | ||
| # Concurrent Jobs With Configmaps Workload | ||
|
|
||
| The Concurrent Jobs with Configmaps test playbook is `workloads/concurrent-jobs-with-configmaps.yaml` | ||
| This workload test is designed to check how many of ConfigMaps and pods can slowdown the cluster. | ||
|
|
||
| ```sh | ||
| $ cp workloads/inventory.example inventory | ||
| $ # Add orchestration host to inventory | ||
| $ # Edit vars in workloads/vars/concurrent-jobs-with-configmaps.yml or define Environment vars (See below) | ||
| $ time ansible-playbook -vv -i inventory workloads/concurrent-jobs-with-configmaps.yml | ||
| ``` | ||
|
|
||
| ## Environment variables | ||
|
|
||
| ### PUBLIC_KEY | ||
| Default: `~/.ssh/id_rsa.pub` | ||
| Public ssh key file for Ansible. | ||
|
|
||
| ### PRIVATE_KEY | ||
| Default: `~/.ssh/id_rsa` | ||
| Private ssh key file for Ansible. | ||
|
|
||
| ### ORCHESTRATION_USER | ||
| Default: `root` | ||
| User for Ansible to log in as. Must authenticate with PUBLIC_KEY/PRIVATE_KEY. | ||
|
|
||
| ### WORKLOAD_IMAGE | ||
| Default: `quay.io/openshift-scale/scale-ci-workload` | ||
| Container image that runs the workload script. | ||
|
|
||
| ### WORKLOAD_JOB_NODE_SELECTOR | ||
| Default: `false` | ||
| Enables/disables the node selector that places the workload job on the `workload` node. | ||
|
|
||
| ### WORKLOAD_JOB_TAINT | ||
| Default: `false` | ||
| Enables/disables the toleration on the workload job to permit the `workload` taint. | ||
|
|
||
| ### WORKLOAD_JOB_PRIVILEGED | ||
| Default: `true` | ||
| Enables/disables running the workload pod as privileged. | ||
|
|
||
| ### KUBECONFIG_FILE | ||
| Default: `~/.kube/config` | ||
| Location of kubeconfig on orchestration host. | ||
|
|
||
| ### PBENCH_INSTRUMENTATION | ||
| Default: `false` | ||
| Enables/disables running the workload wrapped by pbench-user-benchmark. When enabled, pbench agents can then be enabled (`ENABLE_PBENCH_AGENTS`) for further instrumentation data and pbench-copy-results can be enabled (`ENABLE_PBENCH_COPY`) to export captured data for further analysis. | ||
|
|
||
| ### ENABLE_PBENCH_AGENTS | ||
| Default: `false` | ||
| Enables/disables the collection of pbench data on the pbench agent Pods. These Pods are deployed by the tooling playbook. | ||
|
|
||
| ### ENABLE_PBENCH_COPY | ||
| Default: `false` | ||
| Enables/disables the copying of pbench data to a remote results server for further analysis. | ||
|
|
||
| ### PBENCH_SSH_PRIVATE_KEY_FILE | ||
| Default: `~/.ssh/id_rsa` | ||
| Location of ssh private key to authenticate to the pbench results server. | ||
|
|
||
| ### PBENCH_SSH_PUBLIC_KEY_FILE | ||
| Default: `~/.ssh/id_rsa.pub` | ||
| Location of the ssh public key to authenticate to the pbench results server. | ||
|
|
||
| ### PBENCH_SERVER | ||
| Default: There is no public default. | ||
| DNS address of the pbench results server. | ||
|
|
||
| ### NUMBER_OF_CONCURRENT_JOBS | ||
| Default: 300 | ||
| Number of concurrent jobs with configmaps to create during workload. | ||
|
|
||
| ### JOB_COMPLETION_POLL_ATTEMPTS | ||
| Default: `360` | ||
| Number of retries for Ansible to poll if the workload job has completed. Poll attempts delay 10s between polls with some additional time taken for each polling action depending on the orchestration host setup. | ||
|
|
||
| ## Smoke test variables | ||
|
|
||
| ``` | ||
| CONCURRENT_JOBS_NODE_COUNT=4 | ||
|
There was a problem hiding this comment. The CONCURRENT_JOBS_NODE_COUNT, CONCURRENT_JOBS_MAXPODS, CONCURRENT_JOBS_POD_IMAGE, CONCURRENT_JOBS_STEPSIZE and CONCURRENT_JOBS_PAUSE vars are not being used by the workload. |
||
| CONCURRENT_JOBS_TEST_PREFIX=concurrentjobs_smoke | ||
| CONCURRENT_JOBS_CLEANUP=true | ||
| CONCURRENT_JOBS_BASENAME=concurrentjobs | ||
| CONCURRENT_JOBS_MAXPODS=1000 | ||
| CONCURRENT_JOBS_POD_IMAGE="gcr.io/google_containers/pause-amd64:3.0" | ||
| CONCURRENT_JOBS_STEPSIZE=50 | ||
| CONCURRENT_JOBS_PAUSE=60 | ||
| CONCURRENT_JOBS_TS_TIMEOUT=180 | ||
| ``` | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,140 @@ | ||
| --- | ||
| # | ||
| # Runs concurrent jobs with configmaps benchmarks on existing cluster. | ||
| # | ||
|
|
||
| - name: Runs concurrent jobs with configmaps | ||
| hosts: orchestration | ||
| gather_facts: true | ||
| remote_user: "{{orchestration_user}}" | ||
| vars_files: | ||
| - vars/concurrent-jobs-with-configmaps.yml | ||
| vars: | ||
| workload_job: "concurrent-jobs" | ||
| tasks: | ||
| - name: Create scale-ci-tooling directory | ||
| file: | ||
| path: "{{ansible_user_dir}}/scale-ci-tooling" | ||
| state: directory | ||
|
|
||
| - name: Copy workload files | ||
| copy: | ||
| src: "{{item.src}}" | ||
| dest: "{{item.dest}}" | ||
| with_items: | ||
| - src: scale-ci-tooling-ns.yml | ||
| dest: "{{ansible_user_dir}}/scale-ci-tooling/scale-ci-tooling-ns.yml" | ||
| - src: workload-concurrent-jobs-with-configmaps-script-cm.yml | ||
| dest: "{{ansible_user_dir}}/scale-ci-tooling/workload-concurrent-jobs-with-configmaps-script-cm.yml" | ||
|
|
||
| - name: Slurp kubeconfig file | ||
| slurp: | ||
| src: "{{kubeconfig_file}}" | ||
| register: kubeconfig_file_slurp | ||
|
|
||
| - name: Slurp ssh private key file | ||
| slurp: | ||
| src: "{{pbench_ssh_private_key_file}}" | ||
| register: pbench_ssh_private_key_file_slurp | ||
|
|
||
| - name: Slurp ssh public key file | ||
| slurp: | ||
| src: "{{pbench_ssh_public_key_file}}" | ||
| register: pbench_ssh_public_key_file_slurp | ||
|
|
||
| - name: Template workload templates | ||
| template: | ||
| src: "{{item.src}}" | ||
| dest: "{{item.dest}}" | ||
| with_items: | ||
| - src: pbench-cm.yml.j2 | ||
| dest: "{{ansible_user_dir}}/scale-ci-tooling/pbench-cm.yml" | ||
| - src: pbench-ssh-secret.yml.j2 | ||
| dest: "{{ansible_user_dir}}/scale-ci-tooling/pbench-ssh-secret.yml" | ||
| - src: kubeconfig-secret.yml.j2 | ||
| dest: "{{ansible_user_dir}}/scale-ci-tooling/kubeconfig-secret.yml" | ||
| - src: workload-job.yml.j2 | ||
| dest: "{{ansible_user_dir}}/scale-ci-tooling/workload-job.yml" | ||
| - src: workload-env.yml.j2 | ||
| dest: "{{ansible_user_dir}}/scale-ci-tooling/workload-{{workload_job}}-env.yml" | ||
|
|
||
| - name: Check if scale-ci-tooling namespace exists | ||
| shell: | | ||
| oc project scale-ci-tooling | ||
| ignore_errors: true | ||
| changed_when: false | ||
| register: scale_ci_tooling_ns_exists | ||
|
|
||
| - name: Ensure any stale scale-ci-concurrent-jobs job is deleted | ||
| shell: | | ||
| oc delete job scale-ci-{{workload_job}} -n scale-ci-tooling | ||
| register: scale_ci_tooling_project | ||
| failed_when: scale_ci_tooling_project.rc == 0 | ||
| until: scale_ci_tooling_project.rc == 1 | ||
| retries: 60 | ||
| delay: 1 | ||
| when: scale_ci_tooling_ns_exists.rc == 0 | ||
|
|
||
| - name: Ensure project concurrent-jobs-workload from previous workload is deleted | ||
| shell: | | ||
| oc delete project concurrent-jobs-workload | ||
| register: concurrent_jobs_workload_project | ||
| failed_when: concurrent_jobs_workload_project.rc == 0 | ||
| until: concurrent_jobs_workload_project.rc == 1 | ||
| retries: 60 | ||
| delay: 1 | ||
|
|
||
| - name: Block for non-existing tooling namespace | ||
| block: | ||
| - name: Create tooling namespace | ||
| shell: | | ||
| oc create -f {{ansible_user_dir}}/scale-ci-tooling/scale-ci-tooling-ns.yml | ||
|
|
||
| - name: Create tooling service account | ||
| shell: | | ||
| oc create serviceaccount useroot -n scale-ci-tooling | ||
| oc adm policy add-scc-to-user privileged -z useroot -n scale-ci-tooling | ||
| when: enable_pbench_agents|bool or workload_job_privileged|bool | ||
| when: scale_ci_tooling_ns_exists.rc != 0 | ||
|
|
||
| - name: Create/replace kubeconfig secret | ||
| shell: | | ||
| oc replace --force -n scale-ci-tooling -f "{{ansible_user_dir}}/scale-ci-tooling/kubeconfig-secret.yml" | ||
|
|
||
| - name: Create/replace the pbench configmap | ||
| shell: | | ||
| oc replace --force -n scale-ci-tooling -f "{{ansible_user_dir}}/scale-ci-tooling/pbench-cm.yml" | ||
|
|
||
| - name: Create/replace pbench ssh secret | ||
| shell: | | ||
| oc replace --force -n scale-ci-tooling -f "{{ansible_user_dir}}/scale-ci-tooling/pbench-ssh-secret.yml" | ||
|
|
||
| - name: Create/replace workload script configmap | ||
| shell: | | ||
| oc replace --force -n scale-ci-tooling -f "{{ansible_user_dir}}/scale-ci-tooling/workload-concurrent-jobs-with-configmaps-script-cm.yml" | ||
|
|
||
| - name: Create/replace workload script environment configmap | ||
| shell: | | ||
| oc replace --force -n scale-ci-tooling -f "{{ansible_user_dir}}/scale-ci-tooling/workload-{{workload_job}}-env.yml" | ||
|
|
||
| - name: Create/replace workload job to that runs workload script | ||
| shell: | | ||
| oc replace --force -n scale-ci-tooling -f "{{ansible_user_dir}}/scale-ci-tooling/workload-job.yml" | ||
|
|
||
| - name: Poll until job pod is running | ||
| shell: | | ||
| oc get pods --selector=job-name=scale-ci-{{workload_job}} -n scale-ci-tooling -o json | ||
| register: pod_json | ||
| retries: 60 | ||
| delay: 2 | ||
| until: pod_json.stdout | from_json | json_query('items[0].status.phase==`Running`') | ||
|
|
||
| - name: Poll until job is complete | ||
| shell: | | ||
| oc get job scale-ci-{{workload_job}} -n scale-ci-tooling -o json | ||
| register: job_json | ||
| retries: "{{job_completion_poll_attempts}}" | ||
| delay: 10 | ||
| until: job_json.stdout | from_json | json_query('status.succeeded==`1` || status.failed==`1`') | ||
| failed_when: job_json.stdout | from_json | json_query('status.succeeded==`1`') == false | ||
| when: job_completion_poll_attempts|int > 0 |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.