Merge pull request #907 from oracle/domainhomeinimage-tests

READY TO MERGE: Adding domain home in image tests

Author: rjeberhard

integration-tests/README.md

@@ -16,7 +16,7 @@ Wercker runs only Quick test use cases, Jenkins runs both Quick and Full test us
 
 Java integration tests cover the below use cases:
 
-Quick test use cases
+Quick test Configuration & Use Cases - 
 
 |  |  |
 | --- | --- |
@@ -42,25 +42,28 @@ Also the below use cases are covered for Quick test:
 9. Verify the liveness probe by killing managed server 1 process 3 times to kick pod auto-restart.
 10. Shutdown the domain by changing domain `serverStartPolicy` to `NEVER`.
 
-Full test use cases
+
+Full test Configuration & Use Cases - Runs Quick test Configuration & Use cases and the below
 
 |  |  |
 | --- | --- |
-| Operator Configuration | operator2 deployed in `weblogic-operator2` namespace and manages domains `test2` namespace. |
-| Domain Configuration | Domain on PV using WDT; domain with `serverStartPolicy` `ADMIN_ONLY`; domain with auto and custom situational configuration; two domains managed by two operators; domain with recycle `weblogicDomainStorageReclaimPolicy`; domain with default sample values. |
+| Operator Configuration | operator2 deployed in weblogic-operator2 namespace and manages domains test2 namespace |
+| Domain Configuration | Domain on PV using WDT <p> Domain home in image using WLST <p> Domain home in image using WDT <p> Domain with serverStartPolicy ADMIN_ONLY <p> Domain with auto and custom situational configuration <p> Two domains managed by two operators <p> Domain with Recycle weblogicDomainStorageReclaimPolicy <p> Domain with default sample values |
 
 
 Basic Use Cases described above are verified in all the domain configurations. Also the below use cases are covered:
 
 | Domain | Use Case |
 | --- | --- |
 | Domain on PV using WDT | WLDF scaling |
-| Domain with ADMIN_ONLY | Making sure only admin server is started and managed servers are not started. Shutdown domain by deleting domain CRD. Create domain on existing PV dir, pv is already populated by a shutdown domain. |
-| Domain with situational config | Create domain with listen address not set for admin server and t3 channel/NAP and incorrect file for admin server log location. Introspector should override these with sit-config automatically. Also, with some junk value for t3 channel public address and using custom situational config override replace with valid public address using secret. Also, on Jenkins this domain uses NFS instead of HOSTPATH PV storage. |
-| Two domains managed by two operators | Verify scaling and restart of one domain doesn't impact another domain. Delete domain resources using delete script from samples. |			
-| Domain with Recycle policy | Create domain with pvReclaimPolicy="Recycle" Verify that the PV is deleted once the domain and PVC are deleted. |
-| Domain with default sample values | Create domain using mostly default values for inputs. |					
-
+| Domain with ADMIN_ONLY | making sure only admin server is started and managed servers are not started. Shutdown domain by deleting domain CRD. Create domain on existing PV dir, pv is already populated by a shutdown domain. |
+| Domain with situational config | create domain with listen address not set for admin server and t3 channel/NAP and incorrect file for admin server log location. Introspector should override these with sit-config automatically. Also, with some junk value for t3 channel public address and using custom situational config override replace with valid public address using secret. Also, on Jenkins this domain uses NFS instead of HOSTPATH PV storage |	
+| Two domains managed by two operators | verify scaling and restart of one domain doesn't impact another domain. Delete domain resources using delete script from samples. |			
+| Domain with Recycle policy | create domain with pvReclaimPolicy="Recycle" Verify that the PV is deleted once the domain and PVC are deleted |
+| Domain with default sample values | create domain using mostly default values for inputs |	
+| Domain home in image using WLST | cluster scaling |
+| Domain home in image using WDT  | cluster scaling |
+						
 
 # Directory Configuration and Structure
 
@@ -166,10 +169,21 @@ Certain properties like weblogicDomainStoragePath, image, externalOperatorCert a
 
 # How does it work
 
-When the tests are run manually with mvn command on hosted Linux, WebLogic image store/oracle/weblogic:12.2.1.3 is pulled from docker hub or uses local image if one exists. Server jre images are pulled from a local repository wlsldi-v2.docker.oraclecorp.com. Operator image is built with the git branch from where the mvn command is executed.
-All the tests that start with IT*.java are run. The test builds the operator, runs a series of tests and archives the results into tar.gz files upon completion.
-
-Integration test classes:
+When the tests are run with mvn command, 
+- cleanup the test tmp files, PV dir and k8s artifacts created for the test if any
+- creates the required secrets to pull the WL image from docker hub
+- Operator image is built with the git branch from where the mvn command is executed. 
+- creates Operator and verifies operator is running
+- creates Domain crd using samples
+- verifies the domain is started and servers are ready, services are created
+- executes the basic and advanced use cases 
+- shutdown the domain 
+- archive logs and results
+- cleanup the tmp files, PV dir and k8s artifacts created for the test
+	
+All the tests that start with IT*.java in integration-tests/src/test/java are run. 
+
+**Integration test classes:**
 
 When the integration test class ITOperator is executed, staticPrepare() method is called once before any of the test methods in the class and staticUnPrepare() method once at the end.
 
@@ -179,27 +193,43 @@ staticUnPrepare() - releases the cluster lease on wercker env.
 
 test methods -  testDomainOnPVUsingWLST, testDomainOnPVUsingWDT, testTwoDomainsManagedByTwoOperators, testCreateDomainWithStartPolicyAdminOnly, testCreateDomainPVReclaimPolicyRecycle, testCreateDomainWithDefaultValuesInSampleInputs, testAutoAndCustomSitConfigOverrides, testOperatorRESTIdentityBackwardCompatibility, testOperatorRESTUsingCertificateChain
 
-Utility classes:
+**Utility classes:**
 
-Operator - contains methods to create/destroy operator, verify operator created, scale using rest api, etc
-Domain - contains methods to create/destroy domain, verify domain created,deploy webapp, load balancing, etc
-PersistentVolume - to create PV
-LoadBalancer - to create load balancer
-Secret - to create secret
+Operator - constructor takes yaml file with operator properties and generates operator valus yaml with required properties and certs,  creates service account, namespace and calls helm install using the generated values yaml file. Also contains methods to delete operator release, verify operator created and ready, scale using rest api, verify a given domain via rest, verify external rest service, etc  <p>
+Domain - constructor takes Map with domain, LB, PV properties and creates domain crd, LB operator/ingress and PV artifacts using the sample scripts provided in the project. Also contains helper methods to destroy domain by deleting domain crd, verify domain created and servers are ready, deploy webapp, verify load balancing of http requests, etc <p>
+PersistentVolume - runs k8s job to create PV directory and creates PV and PVC using sample scripts  <p>
+LoadBalancer - creates load balancer, currently TREFIK and VOYAGER are supported <p>
+Secret - creates a k8s secret <p>
+TestUtils - mostly runs kubectl commands. Contains utility methods to check if a pod is created, ready, deleted, service created, get pod restart cnt, get cluster replica, delete PVC, check PV released, create rbac policy, create wldf module, etc. <p>
+ExecCommand - Class for executing shell commands from java <p>
+ExecResult - Class that holds the results of using java to exec a command (i.e. exit value, stdout and stderr) <p>
+K8sTestUtils - uses k8s java client api, this is used only for delete domain use cases for now. <p>
 
 # How to run the Java integration tests
 
 * Maven and latest Git should be in PATH
 * export JAVA_HOME
-
-Command to run the tests:
+* export WEBLOGIC_IMAGE_NAME and WEBLOGIC_IMAGE_TAG if different from store/oracle/weblogic and 12.2.1.3
+* Setup docker access to WebLogic 12c Images 
+
+ Method 1 
+  - Setup a personal account on hub.docker.com
+  - Then sign in to hub.docker.com and signup for access to WebLogic 12c Images via https://hub.docker.com/_/oracle-weblogic-server-12c
+  - Then export the following before running the tests: 
+	```
+	export DOCKER_USERNAME=<docker_username>
+	export DOCKER_PASSWORD=<docker_password>
+	export DOCKER_EMAIL=<docker_email>
+	```
+ 
+ Method 2 
+ - Make sure the weblogic image i.e. store/oracle/weblogic:12.2.1.3 already exists locally in a docker repository the k8s cluster can access
+ - Make sure the weblogic image has patch p29135930 (required for the WebLogic Kubernetes Operator). 
+  	- If not, see [https://github.com/oracle/docker-images/tree/master/OracleWebLogic/samples/12213-patch-wls-for-k8s].
+		
+		
+* Command to run the tests: 
 ```
-export DOCKER_USERNAME=<docker_username>
-export DOCKER_PASSWORD=<docker_password>
-export DOCKER_EMAIL=<docker_email>
-or
-make sure the weblogic image i.e. store/oracle/weblogic:12.2.1.3 already exists locally
-
 mvn clean verify -P java-integration-tests 2>&1 | tee log.txt
 ```
 

integration-tests/src/test/java/oracle/kubernetes/operator/ITOperator.java

@@ -521,7 +521,7 @@ public void testOperatorRESTUsingCertificateChain() throws Exception {
    *
    * @throws Exception
    */
-  // @Test
+  @Test
   public void testDomainInImageUsingWLST() throws Exception {
     Assume.assumeFalse(QUICKTEST);
     String testMethodName = new Object() {}.getClass().getEnclosingMethod().getName();
@@ -553,7 +553,7 @@ public void testDomainInImageUsingWLST() throws Exception {
    *
    * @throws Exception
    */
-  // @Test
+  @Test
   public void testDomainInImageUsingWDT() throws Exception {
     Assume.assumeFalse(QUICKTEST);
     String testMethodName = new Object() {}.getClass().getEnclosingMethod().getName();

integration-tests/src/test/java/oracle/kubernetes/operator/utils/Domain.java

@@ -57,7 +57,7 @@ public class Domain {
   private int loadBalancerWebPort = 30305;
   private String userProjectsDir = "";
   private String projectRoot = "";
-  private boolean ingressPerDomain = false;
+  private boolean ingressPerDomain = true;
 
   private String createDomainScript = "";
   private String inputTemplateFile = "";

integration-tests/src/test/java/oracle/kubernetes/operator/utils/Operator.java

@@ -314,6 +314,9 @@ private void generateInputYaml() throws Exception {
     sb.append(generatedInputYamlFile);
     logger.info("Invoking " + sb.toString());
     ExecCommand.exec(sb.toString());
+
+    /* String content = new String(Files.readAllBytes(Paths.get(generatedInputYamlFile)));
+    logger.info("Content of weblogic-operator-values.yaml \n" + content); */
   }
 
   private void runCommandInLoop(String command) throws Exception {

integration-tests/src/test/resources/deploywebapp.py

@@ -1,5 +1,22 @@
 # Copyright 2018, Oracle Corporation and/or its affiliates.  All rights reserved.
 # Licensed under the Universal Permissive License v 1.0 as shown at http://oss.oracle.com/licenses/upl.
 
+import time as systime
+
 connect(sys.argv[1],sys.argv[2],sys.argv[3])
-deploy(sys.argv[4],sys.argv[5],sys.argv[6],upload='false',remote='false')
+deploy(sys.argv[4],sys.argv[5],sys.argv[6],upload='false',remote='false')
+#systime.sleep(15)
+
+#cd ('AppDeployments')
+#myapps=cmo.getAppDeployments()
+ 
+#for appName in myapps:
+#       domainConfig()
+#       cd ('/AppDeployments/'+appName.getName()+'/Targets')
+#       mytargets = ls(returnMap='true')
+#       domainRuntime()
+#       cd('AppRuntimeStateRuntime')
+#       cd('AppRuntimeStateRuntime')
+#       for targetinst in mytargets:
+#             curstate4=cmo.getCurrentState(appName.getName(),targetinst)
+#             print '-----------', curstate4, '-----------', appName.getName()

integration-tests/src/test/resources/domaininimagewdt.yaml

@@ -8,4 +8,5 @@ initialManagedServerReplicas: 2
 exposeAdminT3Channel: true
 exposeAdminNodePort: true
 domainHomeImageBuildPath: ./docker-images/OracleWebLogic/samples/12213-domain-home-in-image-wdt
-domainHomeImageBase: "store/oracle/weblogic:12.2.1.3"
+domainHomeImageBase: "store/oracle/weblogic:12.2.1.3"
+logHomeOnPV: true

integration-tests/src/test/resources/setupenv.sh

@@ -87,8 +87,8 @@ function pull_tag_images {
 	    echo "secret $IMAGE_PULL_SECRET_WEBLOGIC was not created successfully"
 	    exit 1
 	  fi
-	  docker login -u $DOCKER_USERNAME -p $DOCKER_PASSWORD
-   	  docker pull $IMAGE_NAME_WEBLOGIC:$IMAGE_TAG_WEBLOGIC
+	  # docker login -u $DOCKER_USERNAME -p $DOCKER_PASSWORD
+   	  # docker pull $IMAGE_NAME_WEBLOGIC:$IMAGE_TAG_WEBLOGIC
   fi
   set -x
   echo "Pull and tag the images we need"

integration-tests/src/test/resources/statedump.sh

@@ -16,6 +16,9 @@ function state_dump {
      local PROJECT_ROOT="$PROJECT_ROOT"
      local SCRIPTPATH="$PROJECT_ROOT/src/integration-tests/bash"
      local LEASE_ID="$LEASE_ID"
+     local ARCHIVE_DIR="$RESULT_ROOT/acceptance_test_pv_archive"
+     local ARCHIVE_FILE="IntSuite.`date '+%Y%m%d%H%M%S'`.jar"
+     local ARCHIVE="$ARCHIVE_DIR/$ARCHIVE_FILE"
 
      if [ ! -d "$RESULT_DIR" ]; then
         echo State dump exiting early.  RESULT_DIR \"$RESULT_DIR\" does not exist or is not a directory.
@@ -60,6 +63,8 @@ function state_dump {
     done
   done
 
+  mkdir -p $ARCHIVE_DIR || fail Could not archive, could not create target directory \'$ARCHIVE_DIR\'.
+  
   # Get various k8s resource describes and redirect/copy to files 
 
   set +x
@@ -85,17 +90,44 @@ function state_dump {
     done
   done
   set -x
-
+  
   # use a job to archive PV, /scratch mounts to PV_ROOT in the K8S cluster
   echo "Archiving pv directory using a kubernetes job.  Look for it on k8s cluster in $PV_ROOT/acceptance_test_pv_archive"
   local outfile=${DUMP_DIR}/archive_pv_job.out
-  $SCRIPTPATH/job.sh "/scripts/archive.sh /scratch/acceptance_test_pv /scratch/acceptance_test_pv_archive" 2>&1 | tee ${outfile}
-  if [ "$?" = "0" ]; then
-     echo Job complete.
+
+  if [ "$WERCKER" = "true" ]; then
+	$SCRIPTPATH/job.sh "/scripts/archive.sh /scratch/acceptance_test_pv /scratch/acceptance_test_pv_archive" 2>&1 | tee ${outfile}
+	if [ "$?" = "0" ]; then
+     	echo Job complete.
+  	else
+    	echo Job failed.  See ${outfile}.
+  	fi
   else
-     echo Job failed.  See ${outfile}.
+  
+  	$SCRIPTPATH/krun.sh -t 300 -d ${RESULT_DIR} -m "${PV_ROOT}:/sharedparent" -c 'jar cf /sharedparent/pvarchive.jar /sharedparent/acceptance_test_pv' 2>&1 | tee ${outfile}
+  	if [ "$?" = "0" ]; then
+    	$SCRIPTPATH/krun.sh -t 300 -d ${RESULT_DIR} -m  "${PV_ROOT}:/sharedparent" -c 'base64 /sharedparent/pvarchive.jar' > $RESULT_DIR/pvarchive.b64 2>&1
+	 	if [ "$?" = "0" ]; then
+   			base64 -di $RESULT_DIR/pvarchive.b64 > $ARCHIVE
+   			if [ "$?" = "0" ]; then
+   				echo Run complete. Archived to $ARCHIVE
+   			else 
+   				echo Run failed. 
+   			fi
+	 	else
+     		# command failed
+  			cat $RESULT_DIR/pvarchive.b64 | head -100
+	 	fi
+	 	# rm $RESULT_DIR/pvarchive.b64
+  	else
+    	 echo Job failed.  See ${outfile}.
+  	fi	
   fi
+  
+  
+  
 
+  
   if [ ! "$LEASE_ID" = "" ]; then
     # release the lease if we own it
     ${SCRIPTPATH}/lease.sh -d "$LEASE_ID" 2>&1 | tee ${RESULT_DIR}/release_lease.out
@@ -112,7 +144,7 @@ function state_dump {
 
   # now archive all the local test files
   $SCRIPTPATH/archive.sh "${RESULT_DIR}" "${RESULT_DIR}_archive"
-
+  
   echo Done with state dump
 }
 

src/integration-tests/bash/cleanup.sh

@@ -61,6 +61,7 @@ USER_PROJECTS_DIR="$RESULT_DIR/user-projects"
 TMP_DIR="$RESULT_DIR/cleanup_tmp"
 JOB_NAME="weblogic-command-job"
 
+
 function fail {
   echo @@ cleanup.sh: Error "$@"
   exit 1
@@ -346,17 +347,24 @@ SUCCESS="$?"
 
 if [ "${DELETE_FILES:-true}" = "true" ]; then
 
-  # Delete pv directories using a job (/scratch maps to PV_ROOT on the k8s cluster machines).
+  # Delete pv directories using a run (/sharedparent maps to PV_ROOT on the k8s cluster machines).
 
-  echo @@ Launching job to delete all pv contents.  This runs in the k8s cluster, /scratch mounts PV_ROOT.
-  $SCRIPTPATH/job.sh "rm -fr /scratch/acceptance_test_pv"
+  echo @@ Launching run to delete all pv contents.  This runs in the k8s cluster, /sharedparent mounts PV_ROOT.
+  # $SCRIPTPATH/job.sh "rm -fr /scratch/acceptance_test_pv"
+  if [ "$WERCKER" = "true" ]; then
+	$SCRIPTPATH/job.sh "rm -fr /scratch/acceptance_test_pv"
+  else 
+  	$SCRIPTPATH/krun.sh -m "${PV_ROOT}:/sharedparent" -c 'rm -fr /sharedparent/acceptance_test_pv'
+  fi
   [ "$?" = "0" ] || SUCCESS="1"
+  echo @@ SUCCESS=$SUCCESS
 
   # Delete old test files owned by the current user.  
 
   echo @@ Deleting local $RESULT_DIR contents.
   rm -fr $RESULT_ROOT/acceptance_test_tmp
   [ "$?" = "0" ] || SUCCESS="1"
+  echo @@ SUCCESS=$SUCCESS
 
   echo @@ Deleting /tmp/test_suite.\* files.
   rm -f /tmp/test_suite.*