Skip to content

Commit cf86c6f

Browse files
rjeberhardrjeberhard
authored
Merge pull request #352 from oracle/feature/java-integration-tests
Feature/java integration tests
2 parents 36b0e35 + 158dc92 commit cf86c6f

File tree

7 files changed

+448
-39
lines changed

7 files changed

+448
-39
lines changed

‎integration-tests/README.md

Lines changed: 293 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,293 @@
1+
# Integration Tests for Oracle WebLogic Server Kubernetes Operator
2+
3+
This documentation describes the functional use cases that are covered in integration testing for he Oracle WebLogic Server Kubernetes Operator. The tests are written in Java (JUnit tests) and driven by Maven profile. All the use cases covered in run.sh will be covered in Java integration tests. Currently, QUICKTEST use cases and some of Full test(till domain4) use cases are covered in Java tests.
4+
5+
# Environments
6+
7+
The tests currently run in three modes: "Wercker", "Jenkins", and "standalone" Oracle Linux, where the mode is controlled by the WERCKER and JENKINS environment variables described below. The default is "standalone".
8+
9+
* "Standalone" Oracle Linux, i.e, run the tests manually with mvn command.
10+
* Wercker - https://app.wercker.com/Oracle/weblogic-kubernetes-operator/runs - integration-test-java is the pipeline name
11+
* Jenkins - http://wls-jenkins.us.oracle.com/view/weblogic-operator/job/weblogic-kubernetes-operator-javatest/
12+
13+
# Use Cases
14+
15+
Currently Java integration tests cover all QUICKTEST and some Full test use cases from run.sh. The below are the use cases:
16+
17+
* create operator operator1 which manages default and test1 namespaces, verify its deployed successfully, pod created, operator Ready and verify external REST service if configured
18+
* create domain domain1 in default namespace and verify the pods, services are created and servers are in Ready
19+
* verify admin external service by accessing admin REST endpoint with nodeport in URL
20+
* verify admin t3 channel port by exec into the admin pod and deploying webapp using the channel port for WLST
21+
* verify web app load balancing
22+
* verify domain life cycle(destroy and create) should not any impact on Operator managing the domain and web app load balancing and admin external service
23+
* cluster scale up/down using Operator REST endpoint, webapp load balancing should adjust accordingly. (run.sh does scaling by editing the replicas in domain-custom-resource.yaml.)
24+
* Operator life cycle(destroy and create) should not impact the running domain
25+
26+
Full test use cases
27+
28+
* keep the first domain and operator running
29+
* create another domain domain2 in default namespace and verify the domain by doing the checks 2 - 5 listed in quick test
30+
* destroy domain domain2
31+
* create another domain domain3 in test1 namespace and verify the domain by doing the checks 2 - 5 listed in quick test
32+
* destroy domain domain3
33+
* create another operator operator2 which manages test2 namespace and verify domain1 is not affected
34+
* create another domain domain4 in test2 namespace and verify the domain by doing the checks 2 - 5 listed in quick test
35+
* verify scaling for domain4 cluster from 2 to 3 servers and back to 2, plus verify no impact on domain1
36+
* cycle domain1 down and back up, plus verify no impact on domain4
37+
* create domain5 in the default namespace with startupControl="ADMIN", and verify that only admin server is created. on Jenkins, this domain will also test NFS instead of HOSTPATH PV storage
38+
* create domain6 in the default namespace with pvReclaimPolicy="Recycle", and verify that the PV is deleted once the domain and PVC are deleted
39+
* test managed server 1 pod auto-restart in domain1
40+
* destroy domain1
41+
* test that create domain fails when its pv is already populated by a shutdown domain
42+
43+
44+
# Directory Configuration and Structure
45+
46+
Directory structure of source code:
47+
48+
A new module "integration-tests" is added to the Maven project weblogic-kubernetes-operator.
49+
50+
weblogic-kubernetes-operator/integration-tests - location of module pom.xml
51+
weblogic-kubernetes-operator/integration-tests/src/test/java - integration test(JUnit) classes and utility classes
52+
weblogic-kubernetes-operator/integration-tests/src/test/resources - properties files(see Configuration Files section) and other scripts
53+
54+
Directory structure used for the test run:
55+
56+
Main external env vars:
57+
58+
| Variable | Description |
59+
| --- | --- |
60+
| RESULT_ROOT | Root path for local test files. |
61+
| PV_ROOT | Root NFS path behind PV/C directories. This must have permissions suitable for WL pods to add files |
62+
63+
Defaults for RESULT_ROOT & PV_ROOT:
64+
65+
| Test Mode | RESULT_ROOT | PV_ROOT | Where initialized |
66+
| --- | --- | --- | --- |
67+
| stand-alone | /scratch/$USER/wl_k8s_test_results | /scratch/$USER/wl_k8s_test_results | test defaults |
68+
| Jenkins | /scratch/k8s_dir | /scratch/k8s_dir | jenkins configuration |
69+
| Wercker | /pipeline/output/k8s_dir | /scratch | wercker.yml |
70+
71+
72+
'Physical' subdirectories created by test:
73+
Local tmp files: RESULT_ROOT/acceptance_test_tmp/...
74+
75+
PV dirs K8S NFS: PV_ROOT/acceptance_test_pv/persistentVolume-${domain_uid}/...
76+
77+
Archives of above: PV_ROOT/acceptance_test_pv_archive/...
78+
RESULT_ROOT/acceptance_test_tmp_archive/...
79+
80+
'Logical' to 'Physical' K8S PV/PVC mappings:
81+
82+
| | Logical | Actual |
83+
| --- | --- | --- |
84+
| job.sh job | /scratch | PV_ROOT on K8S machines |
85+
| domain pod: | /shared | PV_ROOT/acceptance_test_pv/persistentVolume-${domain_uid} on K8S machines |
86+
87+
# Configuration Files
88+
89+
A module "integration-tests" is added in Maven weblogic-kubernetes-operator project.
90+
91+
Below configuration files are used:
92+
93+
src/integration-tests/resources/OperatorIT.properties - This file is used for configuring common attributes for all integration tests
94+
```
95+
baseDir=/scratch
96+
username=weblogic
97+
password=welcome1
98+
maxIterationsPod=50
99+
waitTimePod=5
100+
```
101+
102+
src/integration-tests/resources/ITFirstOperator.properties - customized properties for Operator. Any property can be provided here from create-weblogic-operator-inputs.yaml, for all the properties that are not defined here default values in create-weblogic-operator-inputs.yaml are used while generating the operator inputs yaml file.
103+
104+
```
105+
serviceAccount= weblogic-operator
106+
namespace= weblogic-operator1
107+
targetNamespaces= default
108+
#if SELF_SIGNED_CERT, externalSans value is set to the local host name if not provided here
109+
externalRestOption= SELF_SIGNED_CERT
110+
elkIntegrationEnabled= false
111+
```
112+
113+
src/integration-tests/resources/ITFirstDomain.properties - customized properties for WebLogic Domain. Any property can be provided here from create-weblogic-domain-inputs.yaml. For all the properties that are not defined here, the default values in create-weblogic-domain-inputs.yaml are used while generating the domain inputs yaml file.
114+
115+
```
116+
adminServerName= admin-server
117+
domainName= base_domain
118+
domainUID= domain1
119+
startupControl= AUTO
120+
clusterName= cluster-1
121+
configuredManagedServerCount= 4
122+
initialManagedServerReplicas= 2
123+
managedServerNameBase= managed-server
124+
#weblogicDomainStoragePath will be ignored, PV dir will be created at /<baseDir>/<USER>/acceptance_test_pv
125+
#weblogicDomainStoragePath= /scratch/external-domain-home/pv001/
126+
secretName= domain1-weblogic-credentials
127+
exposeAdminT3Channel= true
128+
exposeAdminNodePort= true
129+
namespace= default
130+
javaOptions= -Dweblogic.StdoutDebugEnabled=false
131+
132+
```
133+
Certain properties like weblogicDomainStoragePath, weblogicOperatorImage, externalSans are populated at run time.
134+
135+
136+
# How does it work
137+
138+
When the tests are run manually with mvn command on hosted Linux, WebLogic image and server jre images are pulled from a local repository. Operator image is built with the git branch from where the mvn command is executed.
139+
All the tests that start with IT*.java are run. The test builds the operator, runs a series of tests and archives the results into tar.gz files upon completion.
140+
141+
Integration test classes:
142+
143+
When the integration test class ITFirstDomain is executed, staticPrepare() method is called once before any of the test methods in the class and staticUnPrepare() method once at the end.
144+
145+
staticPrepare() - initializes the application properties from OperatorIT.properties and creates resultRoot, pvRoot, userprojectsDir directories by calling initialize() method from the base class BaseTest. It also creates the operator and domain objects which are used later in the test methods.
146+
147+
staticUnPrepare() - releases the cluster lease on wercker env.
148+
149+
test methods - testAdminServerExternalService, testAdminT3Channel, testDomainLifecyle, testClusterScaling, testOperatorLifecycle
150+
151+
Utility classes:
152+
153+
Operator - contains methods to create/destroy operator, verify operator created, scale using rest api, etc
154+
Domain - contains methods to create/destroy domain, verify domain created,deploy webapp, load balancing, etc
155+
PersistentVolume - to create PV
156+
Secret - to create secret
157+
158+
# How to run the Java integration tests
159+
160+
Maven and latest Git should be in PATH
161+
162+
export JAVA_HOME
163+
164+
The tests accepts optional env var overrides:
165+
166+
| Variable | Description |
167+
| --- | --- |
168+
| RESULT_ROOT | The root directory to use for the tests temporary files. See "Directory Configuration and Structure" for defaults and a detailed description of test directories. |
169+
| PV_ROOT | The root directory on the kubernetes cluster used for persistent volumes. See "Directory Configuration and Structure" for defaults and a detailed description of test directories. |
170+
| QUICKTEST | When set to "true", limits testing to a subset of the tests. |
171+
| WERCKER | Set to true if invoking from Wercker, set to false or "" if running stand-alone or from Jenkins. Default is "". |
172+
| JENKINS | Set to true if invoking from Jenkins, set to false or "" if running stand-alone or from Wercker. Default is "". |
173+
| NODEPORT_HOST | DNS name of a Kubernetes worker node. Default is the local host's hostname. |
174+
| BRANCH_NAME | Git branch name. Default is determined by calling 'git branch'. |
175+
| LEASE_ID | Set to a unique value to (A) periodically renew a lease on the k8s cluster that indicates that no other test run should attempt to use the cluster, and (B) delete this lease when the test completes. |
176+
177+
The following additional overrides are currently only used when
178+
WERCKER=true:
179+
180+
| Variable | Description |
181+
| --- | --- |
182+
| IMAGE_TAG_OPERATOR | Docker image tag for operator. Default generated based off the BRANCH_NAME. |
183+
| IMAGE_NAME_OPERATOR | Docker image name for operator. Default is wlsldi-v2.docker.oraclecorp.com/weblogic-operator |
184+
| IMAGE_PULL_POLICY_OPERATOR | Default 'Never'. |
185+
| IMAGE_PULL_SECRET_OPERATOR | Default ''. |
186+
| IMAGE_PULL_SECRET_WEBLOGIC | Default ''.
187+
188+
Command to run the tests:
189+
```
190+
mvn clean verify -P java-integration-tests 2>&1 | tee log.txt
191+
```
192+
Successful run will have the output like below:
193+
```
194+
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 970.765 s - in oracle.kubernetes.operator.ITSingleDomain
195+
[INFO]
196+
[INFO] Results:
197+
[INFO]
198+
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0
199+
[INFO]
200+
[INFO]
201+
[INFO] --- maven-failsafe-plugin:2.20.1:verify (integration-tests) @ operator-integration-tests ---
202+
[INFO]
203+
[INFO] --- maven-failsafe-plugin:2.20.1:integration-test (acceptance-test) @ operator-integration-tests ---
204+
[INFO]
205+
[INFO] --- maven-failsafe-plugin:2.20.1:verify (acceptance-test) @ operator-integration-tests ---
206+
[INFO] ------------------------------------------------------------------------
207+
[INFO] Reactor Summary:
208+
[INFO]
209+
[INFO] weblogic-kubernetes-operator ....................... SUCCESS [ 0.472 s]
210+
[INFO] operator-model ..................................... SUCCESS [ 9.696 s]
211+
[INFO] operator-swagger ................................... SUCCESS [ 2.300 s]
212+
[INFO] operator-runtime ................................... SUCCESS [ 52.737 s]
213+
[INFO] operator-integration-tests ......................... SUCCESS [16:33 min]
214+
[INFO] ------------------------------------------------------------------------
215+
[INFO] BUILD SUCCESS
216+
[INFO] ------------------------------------------------------------------------
217+
[INFO] Total time: 17:39 min
218+
[INFO] Finished at: 2018-05-16T14:25:30-07:00
219+
[INFO] Final Memory: 54M/871M
220+
[INFO] ------------------------------------------------------------------------
221+
bash-4.2$
222+
223+
```
224+
Failed run will have the output
225+
```
226+
[ERROR] Tests run: 5, Failures: 0, Errors: 3, Skipped: 0, Time elapsed: 1,271.204 s <<< FAILURE! - in oracle.kubernetes.operator.ITSingleDomain
227+
[ERROR] testAdminT3Channel(oracle.kubernetes.operator.ITSingleDomain) Time elapsed: 145.59 s <<< ERROR!
228+
java.lang.RuntimeException: FAILURE: testwebapp did not return 200 status code, got 404
229+
at oracle.kubernetes.operator.ITSingleDomain.testAdminT3Channel(ITSingleDomain.java:84)
230+
231+
[ERROR] testDomainLifecyle(oracle.kubernetes.operator.ITSingleDomain) Time elapsed: 378.096 s <<< ERROR!
232+
java.lang.RuntimeException: FAILURE: testwebapp did not return 200 status code, got 404
233+
at oracle.kubernetes.operator.ITSingleDomain.testDomainLifecyle(ITSingleDomain.java:103)
234+
235+
[ERROR] testClusterScaling(oracle.kubernetes.operator.ITSingleDomain) Time elapsed: 223.707 s <<< ERROR!
236+
java.lang.RuntimeException: FAILURE: testwebapp did not return 200 status code, got 404
237+
at oracle.kubernetes.operator.ITSingleDomain.testClusterScaling(ITSingleDomain.java:144)
238+
239+
[INFO]
240+
[INFO] Results:
241+
[INFO]
242+
[ERROR] Errors:
243+
[ERROR] ITSingleDomain.testAdminT3Channel:84 » Runtime FAILURE: testwebapp did not ret...
244+
[ERROR] ITSingleDomain.testClusterScaling:144 » Runtime FAILURE: testwebapp did not re...
245+
[ERROR] ITSingleDomain.testDomainLifecyle:103 » Runtime FAILURE: testwebapp did not re...
246+
[INFO]
247+
[ERROR] Tests run: 5, Failures: 0, Errors: 3, Skipped: 0
248+
[INFO]
249+
[INFO]
250+
[INFO] --- maven-failsafe-plugin:2.20.1:verify (integration-tests) @ operator-integration-tests ---
251+
[INFO] ------------------------------------------------------------------------
252+
[INFO] Reactor Summary:
253+
[INFO]
254+
[INFO] weblogic-kubernetes-operator ....................... SUCCESS [ 0.756 s]
255+
[INFO] operator-model ..................................... SUCCESS [ 12.394 s]
256+
[INFO] operator-swagger ................................... SUCCESS [ 2.014 s]
257+
[INFO] operator-runtime ................................... SUCCESS [ 53.771 s]
258+
[INFO] operator-integration-tests ......................... FAILURE [21:35 min]
259+
[INFO] ------------------------------------------------------------------------
260+
[INFO] BUILD FAILURE
261+
[INFO] ------------------------------------------------------------------------
262+
[INFO] Total time: 22:44 min
263+
[INFO] Finished at: 2018-05-17T06:42:35-07:00
264+
[INFO] Final Memory: 55M/933M
265+
[INFO] ------------------------------------------------------------------------
266+
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-failsafe-plugin:2.20.1:verify (integration-tests) on project operator-integration-tests: There are test failures.
267+
[ERROR]
268+
[ERROR] Please refer to /scratch/vmukkara/mychanges/weblogic-kubernetes-operator/integration-tests/target/failsafe-reports for the individual test results.
269+
[ERROR] Please refer to dump files (if any exist) [date]-jvmRun[N].dump, [date].dumpstream and [date]-jvmRun[N].dumpstream
270+
271+
```
272+
JUnit test results can be seen at "integration-tests/target/failsafe-reports/TEST-oracle.kubernetes.operator.ITSingleDomain.xml". This file shows how much time each test case took to run and the failed test results if any.
273+
274+
# Logging/Archiving
275+
276+
Java utils logging is used, writes all the messages to console and java_test_suite.out in $RESULT_ROOT/acceptance_test_tmp directory.
277+
At the end of the test run, all pods logs, describes are logged in individual files and are written to state-dump-logs directory in $RESULT_ROOT/acceptance_test_tmp.
278+
279+
$RESULT_ROOT/acceptance_test_tmp is archived under $RESULT_ROOT/acceptance_test_tmp_archive
280+
281+
$PV_ROOT/acceptance_test_pv is archived under $PV_ROOT/acceptance_test_pv_archive
282+
283+
# How to add a new test
284+
285+
Add a new JUnit test under integration-tests/src/test/java/oracle/kubernetes/operator.
286+
287+
class name must start with IT(Integration Test), IT*.java
288+
289+
ITFirstDomain.java - take a look at this test for reference
290+
291+
# Future enhancement
292+
293+
Add Full test use cases - part1 done.

‎integration-tests/pom.xml

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -91,6 +91,16 @@
9191
<executable>${resource-integration-test}/setupenv.sh</executable>
9292
</configuration>
9393
</execution>
94+
<execution><!-- dump pod, domain, etc logs -->
95+
<id>statedump</id>
96+
<phase>post-integration-test</phase>
97+
<goals>
98+
<goal>exec</goal>
99+
</goals>
100+
<configuration>
101+
<executable>${resource-integration-test}/statedump.sh</executable>
102+
</configuration>
103+
</execution>
94104
</executions>
95105
</plugin>
96106
<plugin>

0 commit comments

Comments
 (0)