merge with develop

Author: vanajamukkara

integration-tests/README.md

@@ -1,16 +1,16 @@
 # Integration Tests for Oracle WebLogic Server Kubernetes Operator
 
-This documentation describes the functional use cases that are covered in integration testing for the Oracle WebLogic Server Kubernetes Operator. The tests are written in Java (JUnit tests) and driven by Maven profile. 
+This documentation describes the functional use cases that are covered in integration testing for the Oracle WebLogic Server Kubernetes Operator. The tests are written in Java (JUnit tests) and driven by Maven profile.
 
 # Environments
 
-The tests currently run in three modes: "Wercker", "Jenkins", and "standalone" Oracle Linux, where the mode is controlled by the WERCKER and JENKINS environment variables described below. The default is "standalone".
+The tests currently run in three modes: "Wercker", "Jenkins", and "standalone" Oracle Linux, where the mode is controlled by the `WERCKER` and `JENKINS` environment variables described below. The default is "standalone".
 
-* "Standalone" Oracle Linux, i.e, run the tests manually with mvn command. 
-* Wercker - https://app.wercker.com/Oracle/weblogic-kubernetes-operator/runs - integration-test-java is the pipeline name
-* Jenkins - http://wls-jenkins.us.oracle.com/view/weblogic-operator/job/weblogic-kubernetes-operator-javatest/ - Jenkins Run is restricted to Oracle Internal development Process
+* "Standalone" Oracle Linux, i.e, run the tests manually with the `mvn` command.
+* Wercker - https://app.wercker.com/Oracle/weblogic-kubernetes-operator/runs - `integration-test-java` is the pipeline name.
+* Jenkins - http://wls-jenkins.us.oracle.com/view/weblogic-operator/job/weblogic-kubernetes-operator-javatest/ - Jenkins Run is restricted to Oracle Internal development process.
 
-Wercker runs only Quick test use cases, Jenkins run both Quick and Full test use cases.
+Wercker runs only Quick test use cases, Jenkins runs both Quick and Full test use cases.
 
 # Use Cases
 
@@ -20,27 +20,28 @@ Quick test Configuration & Use Cases -
 
 |  |  |
 | --- | --- |
-| Operator Configuration | operator1 deployed in weblogic-operator1 namespace and manages domains in default and test1 namespaces |
-| Domain Configuration | Domain on PV using WLST, traefik load balancer |
+| Operator Configuration | operator1 deployed in `weblogic-operator1` namespace and manages domains in `default` and `test1` namespaces |
+| Domain Configuration | Domain on PV using WLST, Traefik load balancer |
 
 Basic Use Cases
 
-1. create operator operator1 which manages default and test1 namespaces, verify its deployed successfully, pod created, operator Ready and verify external REST service if configured
-2. create domain domain1 in default namespace and verify the pods, services are created and servers are in Ready
-3. verify admin external service by accessing admin REST endpoint with nodeport in URL
-4. verify admin t3 channel port by exec into the admin pod and deploying webapp using the channel port for WLST
-5. verify web app load balancing by accessing the webapp using loadBalancerWebPort
+1. Create operator `operator1` which manages `default` and `test1` namespaces, verify it's deployed successfully, pods created, operator ready and verify external REST service, if configured.
+2. Create domain `domain1` in `default` namespace and verify the pods, services are created and servers are in ready state.
+3. Verify the admin external service by accessing the admin REST endpoint with `nodeport` in URL.
+4. Verify admin t3 channel port by exec into the admin pod and deploying webapp using the channel port for WLST.
+5. Verify web app load balancing by accessing the webapp using `loadBalancerWebPort`.
 
 Advanced Use Cases
 
-6. verify domain life cycle(destroy and create) should not any impact on Operator managing the domain and web app load balancing and admin external service
-7. cluster scale up/down using Operator REST endpoint, webapp load balancing should adjust accordingly.
-8. Operator life cycle(destroy and create) should not impact the running domain
+6. Verify domain life cycle (destroy and create) should not have any impact on operator managing the domain and web app load balancing and admin external service.
+7. Cluster scale up/down using operator REST endpoint, webapp load balancing should adjust accordingly.
+8. Operator life cycle (destroy and create) should not impact the running domain.
 
-Also the below use cases are covered for Quick test
+Also the below use cases are covered for Quick test:
+
+9. Verify the liveness probe by killing managed server 1 process 3 times to kick pod auto-restart.
+10. Shutdown the domain by changing domain `serverStartPolicy` to `NEVER`.
 
-9. verify liveness probe by killing managed server 1 process 3 times to kick pod auto-restart
-10. shutdown the domain by changing domain serverStartPolicy to NEVER
 
 Full test Configuration & Use Cases - Runs Quick test Configuration & Use cases and the below
 
@@ -68,22 +69,22 @@ Basic Use Cases described above are verified in all the domain configurations. A
 
 Directory structure of source code:
 
-A new module "integration-tests" is added to the Maven project weblogic-kubernetes-operator.
+A new module "integration-tests" is added to the Maven project `weblogic-kubernetes-operator`.
 
-weblogic-kubernetes-operator/integration-tests - location of module pom.xml  
-weblogic-kubernetes-operator/integration-tests/src/test/java - integration test(JUnit) classes and utility classes  
-weblogic-kubernetes-operator/integration-tests/src/test/resources - properties, yaml files(see Configuration Files section) and other scripts
+`weblogic-kubernetes-operator/integration-tests` - location of module pom.xml  
+`weblogic-kubernetes-operator/integration-tests/src/test/java` - integration test(JUnit) classes and utility classes  
+`weblogic-kubernetes-operator/integration-tests/src/test/resources` - properties, YAML files (see Configuration Files section) and other scripts.
 
 Directory structure used for the test run:
 
-Main external env vars:
+Main external `env vars`:
 
 | Variable | Description |
 | --- | --- |
 | RESULT_ROOT  | Root path for local test files. |
 | PV_ROOT      | Root NFS path behind PV/C directories.  This must have permissions suitable for WL pods to add files |
 
-Defaults for RESULT_ROOT & PV_ROOT:
+Defaults for `RESULT_ROOT` & `PV_ROOT`:
 
 | Test Mode  |	RESULT_ROOT |	PV_ROOT |	Where initialized |
 | --- | --- | --- | --- |
@@ -175,7 +176,7 @@ Integration test classes:
 
 When the integration test class ITOperator is executed, staticPrepare() method is called once before any of the test methods in the class and staticUnPrepare() method once at the end.
 
-staticPrepare() - initializes the application properties from OperatorIT.properties and creates resultRoot, pvRoot, userprojectsDir directories by calling initialize() method from the base class BaseTest. 
+staticPrepare() - initializes the application properties from OperatorIT.properties and creates resultRoot, pvRoot, userprojectsDir directories by calling initialize() method from the base class BaseTest.
 
 staticUnPrepare() - releases the cluster lease on wercker env.
 
@@ -238,7 +239,7 @@ WERCKER=true:
 Successful run will have the output like below:
 ```
 [INFO] Reactor Summary:
-[INFO] 
+[INFO]
 [INFO] weblogic-kubernetes-operator ....................... SUCCESS [  0.305 s]
 [INFO] operator-model ..................................... SUCCESS [ 10.274 s]
 [INFO] operator-swagger ................................... SUCCESS [  0.436 s]
@@ -256,19 +257,19 @@ Successful run will have the output like below:
 Failed run will have the output like
 ```
 
-[INFO] 
+[INFO]
 [INFO] Results:
-[INFO] 
-[ERROR] Errors: 
+[INFO]
+[ERROR] Errors:
 [ERROR]   ITOperator.testDomainOnPVUsingWLST:145 ? Runtime FAILURE: Couldn't create serv...
-[INFO] 
+[INFO]
 [ERROR] Tests run: 9, Failures: 0, Errors: 1, Skipped: 0
-[INFO] 
-[INFO] 
+[INFO]
+[INFO]
 [INFO] --- maven-failsafe-plugin:2.20.1:verify (integration-tests) @ operator-integration-tests ---
 [INFO] ------------------------------------------------------------------------
 [INFO] Reactor Summary:
-[INFO] 
+[INFO]
 [INFO] Build Tools ........................................ SUCCESS [  1.193 s]
 [INFO] weblogic-kubernetes-operator ....................... SUCCESS [  2.671 s]
 [INFO] json-schema ........................................ SUCCESS [ 14.917 s]
@@ -311,7 +312,7 @@ $RESULT_ROOT/acceptance_test_tmp is archived under $RESULT_ROOT/acceptance_test_
 
 $PV_ROOT/acceptance_test_pv is archived under $PV_ROOT/acceptance_test_pv_archive
 
-On Wercker, these logs can be downloaded by clicking "Download artifact" on cleanup and store step. 
+On Wercker, these logs can be downloaded by clicking "Download artifact" on cleanup and store step.
 
 # How to add a new test
 
@@ -324,3 +325,18 @@ ITOperator.java - take a look at this test for reference
 # Future enhancement
 
 Add functional tests
+
+## Troubleshooting
+
+The integration tests are not completely independent of the environment.
+
+You may run into one or more of the following errors when you attempt to execute the command:
+```
+mvn clean verify -P java-integration-tests 2>&1 | tee log.txt
+```
+1. `[ERROR] No permision to create directory /scratch/...`  
+
+  There are a couple ways to resolve this issue:
+
+  * Create a world writable directory named `/scratch`.
+  * Create some other world writable directory and then define the environment variables `RESULT_ROOT` and `PV_ROOT` to point to that directory. If you want, you can create two directories to keep things separated. See [Directory Configuration and Structure](#directory-configuration-and-structure) for more details.

integration-tests/src/test/java/oracle/kubernetes/operator/BaseTest.java

@@ -251,6 +251,7 @@ public void testAdminT3ChannelWithJMS(Domain domain) throws Exception {
     c.close();
     logger.info("Done - testAdminT3ChannelWithJMS");
   }
+
   /**
    * Restarting the domain should not have any impact on Operator managing the domain, web app load
    * balancing and node port service
@@ -271,6 +272,7 @@ public void testDomainLifecyle(Operator operator, Domain domain) throws Exceptio
       domain.verifyWebAppLoadBalancing(TESTWEBAPP);
     }
     domain.verifyAdminServerExternalService(getUsername(), getPassword());
+    domain.verifyHasClusterServiceChannelPort("TCP", 8011, TESTWEBAPP + "/");
     logger.info("Done - testDomainLifecyle");
   }
 
@@ -315,22 +317,23 @@ public void testClusterScaling(Operator operator, Domain domain) throws Exceptio
 
     // make sure cluster service endpoint is updated with the scaled up servers before verifying
     // load balancing
-    int i = 0;
-    while (i < BaseTest.getMaxIterationsPod()) {
-      int numberOfServersInEndpoint =
-          domain.getNumberOfServersInClusterServiceEndpoint((String) domainMap.get("clusterName"));
-      if (replicaCnt != numberOfServersInEndpoint) {
-        // check for last iteration
-        if (i == (BaseTest.getMaxIterationsPod() - 1)) {
-          throw new RuntimeException("FAILURE: Cluster Serice Endpoint is not updated");
-        }
-        Thread.sleep(BaseTest.getWaitTimePod() * 1000);
-        i++;
-      } else {
-        logger.info("Number of servers addresses in endpoint is same as replica count ");
-        break;
-      }
-    }
+    //    int i = 0;
+    //    while (i < BaseTest.getMaxIterationsPod()) {
+    //      int numberOfServersInEndpoint =
+    //          domain.getNumberOfServersInClusterServiceEndpoint((String)
+    // domainMap.get("clusterName"));
+    //      if (replicaCnt != numberOfServersInEndpoint) {
+    //        // check for last iteration
+    //        if (i == (BaseTest.getMaxIterationsPod() - 1)) {
+    //          throw new RuntimeException("FAILURE: Cluster Serice Endpoint is not updated");
+    //        }
+    //        Thread.sleep(BaseTest.getWaitTimePod() * 1000);
+    //        i++;
+    //      } else {
+    //        logger.info("Number of servers addresses in endpoint is same as replica count ");
+    //        break;
+    //      }
+    //    }
     // commenting the load balance check, bug 29325139
     // domain.verifyWebAppLoadBalancing(TESTWEBAPP);
 

integration-tests/src/test/java/oracle/kubernetes/operator/ITOperator.java

@@ -433,6 +433,9 @@ public void testAutoAndCustomSitConfigOverrides() throws Exception {
       // load input yaml to map and add configOverrides
       Map<String, Object> domainMap = TestUtils.loadYaml(domainonpvwlstFile);
       domainMap.put("configOverrides", "sitconfigcm");
+      domainMap.put(
+          "configOverridesFile",
+          "/integration-tests/src/test/resources/domain-home-on-pv/customsitconfig");
       domainMap.put("domainUID", "customsitdomain");
       domainMap.put("adminNodePort", new Integer("30704"));
       domainMap.put("t3ChannelPort", new Integer("30051"));

integration-tests/src/test/java/oracle/kubernetes/operator/utils/Domain.java

@@ -234,6 +234,7 @@ public void verifyServersReady() throws Exception {
       }
     }
   }
+
   /**
    * verify nodeport by accessing admin REST endpoint
    *
@@ -280,6 +281,73 @@ public void verifyAdminServerExternalService(String username, String password) t
     }
   }
 
+  /**
+   * Verify that we have the channel set for the cluster
+   *
+   * @param protocol
+   * @param port
+   * @param path
+   * @throws Exception
+   */
+  public void verifyHasClusterServiceChannelPort(String protocol, int port, String path)
+      throws Exception {
+
+    /* Make sure the service exists in k8s */
+    if (!TestUtils.checkHasServiceChannelPort(
+        this.getDomainUid() + "-cluster-" + this.clusterName, domainNS, protocol, port)) {
+      throw new RuntimeException(
+          "FAILURE: Cannot find channel port in cluster, but expecting one: "
+              + port
+              + "/"
+              + protocol);
+    }
+
+    /*
+     * Construct a curl command that will be run via kubectl exec on the admin server
+     */
+    StringBuffer curlCmd =
+        new StringBuffer(
+            "kubectl exec " + this.getDomainUid() + "-" + this.adminServerName + " /usr/bin/curl ");
+
+    /*
+     * Make sure we can reach the port,
+     * first via each managed server URL
+     */
+    for (int i = 1; i <= TestUtils.getClusterReplicas(domainUid, clusterName, domainNS); i++) {
+      StringBuffer serverAppUrl = new StringBuffer("http://");
+      serverAppUrl
+          .append(this.getDomainUid() + "-" + managedServerNameBase + i)
+          .append(":")
+          .append(port)
+          .append("/");
+      serverAppUrl.append(path);
+
+      callWebAppAndWaitTillReady(
+          new StringBuffer(curlCmd.toString())
+              .append(serverAppUrl.toString())
+              .append(" -- --write-out %{http_code} -o /dev/null")
+              .toString());
+    }
+
+    /*
+     * Make sure we can reach the port,
+     * second via cluster URL which should round robin through each managed server.
+     * Use the callWebAppAndCheckForServerNameInResponse method with verifyLoadBalancing
+     * enabled to verify each managed server is responding.
+     */
+    StringBuffer clusterAppUrl = new StringBuffer("http://");
+    clusterAppUrl
+        .append(this.getDomainUid() + "-cluster-" + this.clusterName)
+        .append(":")
+        .append(port)
+        .append("/");
+    clusterAppUrl.append(path);
+
+    // execute curl and look for each managed server name in response
+    callWebAppAndCheckForServerNameInResponse(
+        new StringBuffer(curlCmd.toString()).append(clusterAppUrl.toString()).toString(), true);
+  }
+
   /**
    * deploy webapp using nodehost and nodeport
    *
@@ -416,8 +484,8 @@ public void callWebAppAndVerifyLoadBalancing(String webappName, boolean verifyLo
       StringBuffer curlCmdResCode = new StringBuffer(curlCmd.toString());
       curlCmdResCode.append(" --write-out %{http_code} -o /dev/null");
 
-      logger.info("Curd cmd with response code " + curlCmdResCode);
-      logger.info("Curd cmd " + curlCmd);
+      logger.info("Curl cmd with response code " + curlCmdResCode);
+      logger.info("Curl cmd " + curlCmd);
 
       // call webapp iteratively till its deployed/ready
       callWebAppAndWaitTillReady(curlCmdResCode.toString());
@@ -1103,9 +1171,12 @@ private void initialize(Map<String, Object> inputDomainMap) throws Exception {
     domainMap.values().removeIf(Objects::isNull);
 
     // create config map and secret for custom sit config
-    if (domainMap.get("configOverrides") != null) {
+    if ((domainMap.get("configOverrides") != null)
+        && (domainMap.get("configOverridesFile") != null)) {
       // write hostname in config file for public address
 
+      String configOverridesFile = domainMap.get("configOverridesFile").toString();
+
       String cmd =
           "kubectl -n "
               + domainNS
@@ -1115,7 +1186,7 @@ private void initialize(Map<String, Object> inputDomainMap) throws Exception {
               + domainMap.get("configOverrides")
               + " --from-file "
               + BaseTest.getProjectRoot()
-              + "/integration-tests/src/test/resources/domain-home-on-pv/customsitconfig";
+              + configOverridesFile;
       ExecResult result = ExecCommand.exec(cmd);
       if (result.exitValue() != 0) {
         throw new RuntimeException(

integration-tests/src/test/java/oracle/kubernetes/operator/utils/TestUtils.java

@@ -220,6 +220,39 @@ public static boolean checkPVReleased(String pvBaseName, String namespace) throw
     }
     return true;
   }
+
+  /**
+   * NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) domain1-cluster-cluster-1 ClusterIP 10.105.146.61
+   * <none> 30032/TCP,8001/TCP domain1-managed-server1 ClusterIP None <none> 30032/TCP,8001/TCP
+   *
+   * @param service
+   * @param namespace
+   * @param protocol
+   * @param port
+   * @return
+   * @throws Exception
+   */
+  public static boolean checkHasServiceChannelPort(
+      String service, String namespace, String protocol, int port) throws Exception {
+    StringBuffer cmd = new StringBuffer("kubectl get services ");
+    cmd.append(" -n ").append(namespace);
+    logger.info(" Find services in namespage " + namespace + " with command: '" + cmd + "'");
+
+    ExecResult result = ExecCommand.exec(cmd.toString());
+    String stdout = result.stdout();
+    logger.info(" Services found: ");
+    logger.info(stdout);
+    String stdoutlines[] = stdout.split("\\r?\\n");
+    if (result.exitValue() == 0 && stdoutlines.length > 0) {
+      for (String stdoutline : stdoutlines) {
+        if (stdoutline.contains(service) && stdoutline.contains(port + "/" + protocol)) {
+          return true;
+        }
+      }
+    }
+    return false;
+  }
+
   /**
    * First, kill the mgd server process in the container three times to cause the node manager to
    * mark the server 'failed not restartable'. This in turn is detected by the liveness probe, which