Merge pull request #1074 from oracle/jrftest3

add JRFUSECASES.MD

Author: rjeberhard

integration-tests/JRFUSECASES.MD

@@ -0,0 +1,61 @@
+# Use Cases for FMW Infra Domain in the Operator
+
+Java integration tests cover the following use cases:
+
+## Basic test Configuration & Use Cases
+
+|  |  |
+| --- | --- |
+| Operator Configuration | operator1 deployed in `weblogic-operator` namespace and manages domains in `default` and `jrfdomains` namespaces |
+| Domain Configuration | Domain on PV using WLST, Traefik load balancer |
+
+**Basic Use Cases**
+
+1. Create operator `weblogic-operator` which manages `default` and `jrfdomains` namespaces, verify it's deployed successfully, pods created, operator ready and verify external REST service, if configured.
+2. Create domain `jrfdomain` in `jrfdomains` namespace and verify the pods, services are created and servers are in ready state.
+3. Verify the admin external service by accessing the admin REST endpoint with `nodeport` in URL.
+4. Verify exec into the admin pod and deploying webapp using the admin port with WLST.
+5. Verify web app load balancing by accessing the webapp using `loadBalancerWebPort`.
+
+**Advanced Use Cases**
+
+6. Verify domain life cycle (destroy and create) should not have any impact on operator managing the domain and web app load balancing and admin external service.
+7. Cluster scale up/down using operator REST endpoint, webapp load balancing should adjust accordingly.
+8. Operator life cycle (destroy and create) should not impact the running domain.
+
+Also the below use cases are covered for each test:
+
+9. Verify the liveness probe by killing managed server 1 process 3 times to kick pod auto-restart.
+10. Shutdown the domain by changing domain `serverStartPolicy` to `NEVER`.
+
+
+## Full test Configuration & Use Cases 
+
+Additional Operator Configuration:
+
+|  |  |
+| --- | --- |
+| Operator Configuration | `operator2` deployed in `weblogic-operator2` namespace and manages domains in `test2` namespace |
+
+Basic Use Cases described above are verified in all the domain configurations. Also the below use cases are covered:
+
+| Domain | Use Case |
+| --- | --- |
+| Domain on PV using WLST |  as above in basic use cases |
+| Domain with ADMIN_ONLY | verify only admin server is started and managed servers are not started. Shutdown domain by deleting domain CRD. Create domain on existing PV dir, pv is already populated by a shutdown domain. |
+| Domain with situational config | create domain with listen address not set for admin server and t3 channel/NAP and incorrect file for admin server log location. Introspector should override these with sit-config automatically. Also, with some junk value for t3 channel public address and using custom situational config override replace with valid public address using secret. Also, on Jenkins this domain uses NFS instead of HOSTPATH PV storage |	
+| Two domains managed by two operators | verify scaling and restart of one domain doesn't impact another domain. Delete domain resources using delete script from samples. |			
+| Two domains in the same namespace managed by one operator | create two FMW Infra Domains in the same namespace managed by one operator. Verify scaling and restart of one domain doesn't impact another domain. |
+| Two domains in the different namespaces managed by one operator | create two FMW Infra Domains in the different namespaces managed by one operator. Domain1 uses VOYAGER load balancer. Domain2 uses TRAEFIK load balancer. Verify scaling and restart of one domain doesn't impact another domain. |
+| Domain with Recycle policy | create domain with pvReclaimPolicy="Recycle" and using Configured cluster. Verify that the PV is deleted once the domain and PVC are deleted |
+| Domain with default sample values | create domain using mostly default values for inputs |	
+
+Test cases cover the bugs found:
+
+| Domain | Use Case |
+| --- | --- | 
+| Domain on PV using WLST | cluster rolling restart with restartVersion and maxUnavailable set to 2, verify the not-ready server counts can not exceed maxUnavailable value |
+| Domain on PV using WLST | in create domain input file, set exposeAdminNodePort to false and exposeAdminT3Channel to true, verify the managed server pods are created |
+| Domain on PV using WLST | in create domain input file, set createDomainScriptsMountPath to non-default value, verify the create domain sample script works |
+| Domain on PV using WLST | in createFMWDomain.py file, set administration_port_enabled to true, verify the admin sever pod is running and ready and all managed server pods are created |
+| Domain on PV using WLST | in create domain input file, set exposeAdminT3Channel to true, verify the admin t3 channel is exposed |

integration-tests/README.md

@@ -16,6 +16,8 @@ Shared cluster runs only Quick test use cases, Jenkins runs both Quick and Full
 
 Use Cases covered in integration tests for the operator is available [here](USECASES.MD)
 
+JRF Use Cases are covered [here](JRFUSECASES.MD)
+
 # Directory Configuration and Structure
 
 Directory structure of source code:
@@ -282,6 +284,35 @@ Failed run will have the output like
 ```
 JUnit test results can be seen at "integration-tests/target/failsafe-reports/TEST-oracle.kubernetes.operator.ITOperator.xml". This file shows how much time each test case took to run and the failed test results if any.
 
+# How to run JRF domain In Operator related tests
+* Setup docker access to FMW Infrastructure 12c Image and Oracle Database 12c Image
+
+ Method 1 
+  - Setup a personal account on hub.docker.com
+  - Then sign in to hub.docker.com and signup for access to Oracle Database 12c Images via https://hub.docker.com/_/oracle-database-enterprise-edition
+  - Then export the following before running the tests: 
+	```
+	export DOCKER_USERNAME=<docker_username>
+	export DOCKER_PASSWORD=<docker_password>
+	export DOCKER_EMAIL=<docker_email>
+	```
+  - Setup an account in phx.ocir.io
+  - Then sign in to phx.ocir.io to get access to FMW Infrastructure 12c Image: **_phx.ocir.io/weblogick8s/oracle/fmw-infrastructure:12.2.1.3_**
+  - export the following before running the tests:
+    ```
+    export REPO_USERNAME=<ocir_username>
+    export REPO_PASSWORD=<ocir_password>
+    export REPO_EMAIL=<ocir_email>
+    ```
+ 
+ Method 2 
+ - Make sure the FMW Infrastructure image i.e. **_phx.ocir.io/weblogick8s/oracle/fmw-infrastructure:12.2.1.3_** and the Oracle database image i.e. **_store/oracle/database-enterprise:12.2.0.1_** already exist locally in a docker repository the k8s cluster can access
+		
+* Command to run the tests: 
+```
+mvn clean verify -P jrf-integration-tests 2>&1 | tee log.txt
+```
+
 # How to run a single test
 
 mvn -Dit.test="ITOperator#testDomainOnPVUsingWLST" -DfailIfNoTests=false integration-test -P java-integration-tests

integration-tests/src/test/java/oracle/kubernetes/operator/JrfInOperatorTest.java

@@ -230,6 +230,190 @@ public void testTwoJRFDomainsManagedByTwoOperators() throws Exception {
     logger.info("SUCCESS - " + testMethodName);
   }
 
+  /**
+   * create two JRF domains in the same namespace and managed by one operator. Verify scaling up and
+   * down for domain1 cluster will have no impact on domain2. Cycle domain2 down and back up, verify
+   * there is no impact on domain1. shutdown by the domains using the delete resource script from
+   * samples.
+   *
+   * @throws Exception - if any error occurs
+   */
+  @Test
+  public void testTwoJRFDomainsManagedByOneOperatorInSameNS() throws Exception {
+    Assume.assumeFalse(QUICKTEST);
+    String testMethodName = new Object() {}.getClass().getEnclosingMethod().getName();
+    logTestBegin(testMethodName);
+
+    logger.info("Checking if operator1 and domain1 are running, if not creating");
+    if (operator1 == null) {
+      operator1 = TestUtils.createOperator(JRF_OPERATOR_FILE_1);
+    }
+
+    JRFDomain domain1 = null, domain2 = null;
+    boolean testCompletedSuccessfully = false;
+    try {
+      // load input yaml to map and add configOverrides
+      Map<String, Object> domain1Map = TestUtils.loadYaml(JRF_DOMAIN_ON_PV_WLST_FILE);
+      domain1Map.put("domainUID", "jrfd1");
+      domain1Map.put("adminNodePort", 30705);
+      domain1Map.put("t3ChannelPort", 30025);
+      domain1Map.put("voyagerWebPort", 30309);
+      domain1Map.put("rcuSchemaPrefix", "jrfd1");
+
+      // run RCU script to load db schema
+      DBUtils.runRCU(rcuPodName, domain1Map);
+
+      Map<String, Object> domain2Map = TestUtils.loadYaml(JRF_DOMAIN_ON_PV_WLST_FILE);
+      domain2Map.put("domainUID", "jrfd2");
+      domain2Map.put("adminNodePort", 30706);
+      domain2Map.put("t3ChannelPort", 30026);
+      domain2Map.put("voyagerWebPort", 30310);
+      domain2Map.put("rcuSchemaPrefix", "jrfd2");
+
+      // run RCU script to load db schema
+      DBUtils.runRCU(rcuPodName, domain2Map);
+
+      // create domain1
+      logger.info("Creating Domain domain1 & verifying the domain creation");
+      domain1 = new JRFDomain(domain1Map);
+      domain1.verifyDomainCreated();
+
+      testBasicUseCases(domain1);
+
+      // create domain2
+      domain2 = new JRFDomain(domain2Map);
+      domain2.verifyDomainCreated();
+
+      testBasicUseCases(domain2);
+
+      logger.info("Verify the running domain domain1 is unaffected");
+      domain1.verifyDomainCreated();
+
+      testClusterScaling(operator1, domain1);
+
+      logger.info("Verify the running domain domain2 is unaffected");
+      domain2.verifyDomainCreated();
+
+      logger.info("Destroy and create domain2 and verify no impact on domain1");
+      domain2.destroy();
+      domain2.create();
+
+      logger.info("Verify no impact on domain1");
+      domain1.verifyDomainCreated();
+      testCompletedSuccessfully = true;
+
+    } finally {
+      String domainUidsToBeDeleted = "";
+
+      if (domain1 != null && (JENKINS || testCompletedSuccessfully)) {
+        domainUidsToBeDeleted = domain1.getDomainUid();
+      }
+      if (domain2 != null && (JENKINS || testCompletedSuccessfully)) {
+        domainUidsToBeDeleted = domainUidsToBeDeleted + "," + domain2.getDomainUid();
+      }
+      if (!domainUidsToBeDeleted.equals("")) {
+        logger.info("About to delete domains: " + domainUidsToBeDeleted);
+        TestUtils.deleteWeblogicDomainResources(domainUidsToBeDeleted);
+        TestUtils.verifyAfterDeletion(domain1);
+        TestUtils.verifyAfterDeletion(domain2);
+      }
+    }
+    logger.info("SUCCESS - " + testMethodName);
+  }
+
+  /**
+   * create two JRF domains in the different namespaces and managed by one operator. Domain1 uses
+   * Voyager load balancer, Domain2 uses Traefik load balancer. Verify scaling up and down for
+   * domain1 cluster will have no impact on domain2. Cycle domain2 down and back up, verify there is
+   * no impact on domain1. shutdown by the domains using the delete resource script from samples.
+   *
+   * @throws Exception - if any error occurs
+   */
+  @Test
+  public void testTwoJRFDomainsManagedByOneOperatorInDifferentNS() throws Exception {
+    Assume.assumeFalse(QUICKTEST);
+    String testMethodName = new Object() {}.getClass().getEnclosingMethod().getName();
+    logTestBegin(testMethodName);
+
+    logger.info("Checking if operator1 and domain1 are running, if not creating");
+    if (operator1 == null) {
+      operator1 = TestUtils.createOperator(JRF_OPERATOR_FILE_1);
+    }
+
+    JRFDomain domain1 = null, domain2 = null;
+    boolean testCompletedSuccessfully = false;
+    try {
+      // load input yaml to map and add configOverrides
+      Map<String, Object> domain1Map = TestUtils.loadYaml(JRF_DOMAIN_ON_PV_WLST_FILE);
+      domain1Map.put("domainUID", "jrfd11");
+      domain1Map.put("adminNodePort", 30707);
+      domain1Map.put("t3ChannelPort", 30027);
+      domain1Map.put("voyagerWebPort", 30311);
+      domain1Map.put("rcuSchemaPrefix", "jrfd11");
+      domain1Map.put("namespace", "default");
+      domain1Map.put("loadBalancer", "VOYAGER");
+
+      // run RCU script to load db schema
+      DBUtils.runRCU(rcuPodName, domain1Map);
+
+      Map<String, Object> domain2Map = TestUtils.loadYaml(JRF_DOMAIN_ON_PV_WLST_FILE);
+      domain2Map.put("domainUID", "jrfd21");
+      domain2Map.put("adminNodePort", 30708);
+      domain2Map.put("t3ChannelPort", 30028);
+      domain2Map.put("voyagerWebPort", 30312);
+      domain2Map.put("rcuSchemaPrefix", "jrfd21");
+
+      // run RCU script to load db schema
+      DBUtils.runRCU(rcuPodName, domain2Map);
+
+      // create domain1
+      logger.info("Creating Domain domain1 & verifying the domain creation");
+      domain1 = new JRFDomain(domain1Map);
+      domain1.verifyDomainCreated();
+
+      testBasicUseCases(domain1);
+
+      // create domain2
+      domain2 = new JRFDomain(domain2Map);
+      domain2.verifyDomainCreated();
+
+      testBasicUseCases(domain2);
+
+      logger.info("Verify the running domain domain1 is unaffected");
+      domain1.verifyDomainCreated();
+
+      testClusterScaling(operator1, domain1);
+
+      logger.info("Verify the running domain domain2 is unaffected");
+      domain2.verifyDomainCreated();
+
+      logger.info("Destroy and create domain2 and verify no impact on domain1");
+      domain2.destroy();
+      domain2.create();
+
+      logger.info("Verify no impact on domain1");
+      domain1.verifyDomainCreated();
+      testCompletedSuccessfully = true;
+
+    } finally {
+      String domainUidsToBeDeleted = "";
+
+      if (domain1 != null && (JENKINS || testCompletedSuccessfully)) {
+        domainUidsToBeDeleted = domain1.getDomainUid();
+      }
+      if (domain2 != null && (JENKINS || testCompletedSuccessfully)) {
+        domainUidsToBeDeleted = domainUidsToBeDeleted + "," + domain2.getDomainUid();
+      }
+      if (!domainUidsToBeDeleted.equals("")) {
+        logger.info("About to delete domains: " + domainUidsToBeDeleted);
+        TestUtils.deleteWeblogicDomainResources(domainUidsToBeDeleted);
+        TestUtils.verifyAfterDeletion(domain1);
+        TestUtils.verifyAfterDeletion(domain2);
+      }
+    }
+    logger.info("SUCCESS - " + testMethodName);
+  }
+
   /**
    * Create operator if its not running and create domain with serverStartPolicy="ADMIN_ONLY".
    * Verify only admin server is created. shutdown by deleting domain CRD. Create domain on existing