Merge remote-tracking branch 'origin/develop' into voyager-integ-test

Author: [email protected]

docs-source/content/userguide/managing-domains/persistent-storage/_index.md

@@ -22,14 +22,41 @@ Persistent volumes can point to different storage locations, for example NFS ser
 
 The persistent volume for the domain must be created using the appropriate tools before running the script to create the domain.  In the simplest case, namely the `HOST_PATH` provider, this means creating a directory on the Kubernetes master and ensuring that it has the correct permissions:
 
-```
+```bash
 $ mkdir -m 777 -p /path/to/domain1PersistentVolume
 ```
 
 **Note regarding NFS**: In the current GA version, the OCI Container Engine for Kubernetes supports network block storage that can be shared across nodes with access permission RWOnce (meaning that only one can write, others can read only). At this time, the WebLogic on Kubernetes domain created by the WebLogic Server Kubernetes Operator, requires a shared file system to store the WebLogic domain configuration, which MUST be accessible from all the pods across the nodes. As a workaround, you need to install an NFS server on one node and share the file system across all the nodes.
 
 Currently, we recommend that you use NFS version 3.0 for running WebLogic Server on OCI Container Engine for Kubernetes. During certification, we found that when using NFS 4.0, the servers in the WebLogic domain went into a failed state intermittently. Because multiple threads use NFS (default store, diagnostics store, Node Manager, logging, and `domain_home`), there are issues when accessing the file store. These issues are removed by changing the NFS to version 3.0.
 
+#### Persistent volume GID annotation
+
+The `HOST_PATH` directory permissions can be made more secure by using a Kubernetes annotation on the
+persistent volume that provides the group identifier (GID) which will be added to pods using the persistent volume.
+
+For example, if the GID of the directory is `6789`, then the directory can be updated to remove permissions
+other than for the user and group along with the persistent volume being annotated with the specified GID:
+
+```bash
+$ chmod 770 /path/to/domain1PersistentVolume
+$ kubectl annotate pv domain1-weblogic-sample-pv pv.beta.kubernetes.io/gid=6789
+```
+
+Typically, after the domain is created and servers are running, the group ownership of the persistent volume files
+can be updated to the specified GID which will provide read access to the group members. Normally
+files created from a pod onto the persistent volume will have UID `1000` and GID `1000` which is the
+`oracle` user from the WebLogic Docker image.
+
+An example of updating the group ownership on the persistent volume would be as follows:
+
+```bash
+$ cd /path/to/domain1PersistentVolume
+$ sudo chgrp 6789 applications domains logs stores
+$ sudo chgrp -R 6789 domains/
+$ sudo chgrp -R 6789 logs/
+```
+
 ### YAML files
 
 Persistent volumes and claims are described in YAML files. For each persistent volume, you should create one persistent volume YAML file and one persistent volume claim YAML file. In the example below, you will find two YAML templates, one for the volume and one for the claim. As stated above, they either can be dedicated to a specific domain, or shared across multiple domains. For the use cases where a volume will be dedicated to a particular domain, it is a best practice to label it with `weblogic.domainUID=[domain name]`. This makes it easy to search for, and clean up resources associated with that particular domain.

integration-tests/USECASES.MD

@@ -34,8 +34,7 @@ Also the below use cases are covered for Quick test:
 |  |  |
 | --- | --- |
 | Operator Configuration | operator2 deployed in weblogic-operator2 namespace and manages domains test2 namespace |
-| Domain Configuration | Domain on PV using WDT <p> Domain home in image using WLST <p> Domain home in image using WDT <p> Domain with serverStartPolicy ADMIN_ONLY <p> Domain with auto and custom situational configuration <p> Two domains managed by two operators <p> Domain with Recycle weblogicDomainStorageReclaimPolicy <p> Domain with default sample values |
-
+| Domain Configuration | Domain on PV using WDT <p> Domain home in image using WLST <p> Domain home in image using WDT <p> Domain with serverStartPolicy ADMIN_ONLY <p> Domain with auto and custom situational configuration <p> Two domains managed by two operators <p> Domain with Recycle weblogicDomainStorageReclaimPolicy <p> Domain with default sample values <p> Domain on PV with two configured clusters <p> Domain on PV with one configured cluster and one dynamic cluster | 
 
 Basic Use Cases described above are verified in all the domain configurations. Also the below use cases are covered:
 
@@ -96,5 +95,10 @@ Configuration Overrides Usecases
 
 | Server Discovery | Use Case |
 | --- | --- |
-| Discover a newly started server | stop operator and apply the modified domain.yaml with replicas count increated, restart Operator and verify that the cluster is scaled up accordingly |
-| Discover dead weblogic servers | stop Operator, kill admin server and all managed servers in the cluster, restart Operator and verify that it restarts all dead servers | 
+| Discover a newly started server | Stop operator and apply the modified domain.yaml with replicas count increased, restart Operator and verify that the cluster is scaled up accordingly |
+| Discover dead weblogic servers | Stop Operator, kill admin server and all managed servers in the cluster, restart Operator and verify that it restarts all dead servers | 
+
+| Sticky Session | Use Case |
+| --- | --- |
+| Server affinity | Use a web application deployed on Weblogic cluster to track HTTP session. Test server-affinity by sending two HTTP requests to Weblogic and verify that all requests are directed to same Weblogic server |
+| Session state isolation | Verify that values saved in a client session state are not visible to another client |

integration-tests/src/test/java/oracle/kubernetes/operator/BaseTest.java

@@ -64,7 +64,7 @@ public class BaseTest {
   private static int waitTimePod = 5;
   private static String leaseId = "";
   private static String branchName = "";
-  private static String appLocationInPod = "/u01/oracle/apps";
+  protected static String appLocationInPod = "/u01/oracle/apps";
   private static String appLocationOnHost;
   private static Properties appProps;
 
@@ -213,6 +213,17 @@ public static void initialize(String appPropsFile) throws Exception {
     logger.info("Env var BRANCH_NAME " + System.getenv("BRANCH_NAME"));
   }
 
+  /**
+   * Call the basic usecases tests
+   *
+   * @param domain
+   * @throws Exception
+   */
+  protected void testBasicUseCases(Domain domain) throws Exception {
+    testAdminT3Channel(domain);
+    testAdminServerExternalService(domain);
+  }
+
   /**
    * Access Admin REST endpoint using admin node host and node port
    *
@@ -339,13 +350,24 @@ public void testWSLoadBalancing(Domain domain) throws Exception {
     logger.info("Done - testWSLoadBalancing");
   }
 
+  /**
+   * use default cluster service port 8011
+   *
+   * @param operator
+   * @param domain
+   * @throws Exception
+   */
+  public void testDomainLifecyle(Operator operator, Domain domain) throws Exception {
+    testDomainLifecyle(operator, domain, 8011);
+  }
+
   /**
    * Restarting the domain should not have any impact on Operator managing the domain, web app load
    * balancing and node port service
    *
    * @throws Exception
    */
-  public void testDomainLifecyle(Operator operator, Domain domain) throws Exception {
+  public void testDomainLifecyle(Operator operator, Domain domain, int port) throws Exception {
     logger.info("Inside testDomainLifecyle");
     domain.destroy();
     domain.create();
@@ -362,7 +384,7 @@ public void testDomainLifecyle(Operator operator, Domain domain) throws Exceptio
     // intermittent failure, see OWLS-73416
     // testWSLoadBalancing(domain);
     domain.verifyAdminServerExternalService(getUsername(), getPassword());
-    domain.verifyHasClusterServiceChannelPort("TCP", 8011, TESTWEBAPP + "/");
+    domain.verifyHasClusterServiceChannelPort("TCP", port, TESTWEBAPP + "/");
     logger.info("Done - testDomainLifecyle");
   }
 

integration-tests/src/test/java/oracle/kubernetes/operator/ITMultipleClusters.java

@@ -0,0 +1,209 @@
+// Copyright 2018, 2019 Oracle Corporation and/or its affiliates.  All rights reserved.
+// Licensed under the Universal Permissive License v 1.0 as shown at
+// http://oss.oracle.com/licenses/upl.
+package oracle.kubernetes.operator;
+
+import static org.junit.Assert.assertTrue;
+
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Paths;
+import java.nio.file.StandardCopyOption;
+import java.nio.file.StandardOpenOption;
+import java.util.Map;
+import oracle.kubernetes.operator.utils.Domain;
+import oracle.kubernetes.operator.utils.K8sTestUtils;
+import oracle.kubernetes.operator.utils.Operator;
+import oracle.kubernetes.operator.utils.TestUtils;
+import org.junit.AfterClass;
+import org.junit.BeforeClass;
+import org.junit.FixMethodOrder;
+import org.junit.Test;
+import org.junit.runners.MethodSorters;
+
+/**
+ * Multiple clusters in a domain tests
+ *
+ * <p>More than 1 cluster is created in a domain , like configured and dynamic
+ */
+@FixMethodOrder(MethodSorters.NAME_ASCENDING)
+public class ITMultipleClusters extends BaseTest {
+
+  private static Operator operator1;
+  private static final String TWO_CONFIGURED_CLUSTER_SCRIPT =
+      "create-domain-two-configured-cluster.py";
+  private static final String TWO_MIXED_CLUSTER_SCRIPT = "create-domain-two-mixed-cluster.py";
+  private static String template;
+  private static final String DOMAINUID = "twoconfigclustdomain";
+
+  /**
+   * This method gets called only once before any of the test methods are executed. It does the
+   * initialization of the integration test properties defined in OperatorIT.properties and setting
+   * the resultRoot, pvRoot and projectRoot attributes.
+   *
+   * @throws Exception
+   */
+  @BeforeClass
+  public static void staticPrepare() throws Exception {
+    // initialize test properties and create the directories
+    initialize(APP_PROPS_FILE);
+    template =
+        BaseTest.getProjectRoot() + "/kubernetes/samples/scripts/common/domain-template.yaml";
+  }
+
+  /**
+   * Releases k8s cluster lease, archives result, pv directories
+   *
+   * @throws Exception
+   */
+  @AfterClass
+  public static void staticUnPrepare() throws Exception {
+    logger.info("+++++++++++++++++++++++++++++++++---------------------------------+");
+    logger.info("BEGIN");
+    logger.info("Run once, release cluster lease");
+    tearDown(new Object() {}.getClass().getEnclosingClass().getSimpleName());
+    logger.info("SUCCESS");
+  }
+
+  /**
+   * Create 2 configured clusters in a domain each having 2 managed servers. Verify the managed
+   * servers are running and verify the basic use cases.
+   *
+   * @throws Exception
+   */
+  @Test
+  public void testCreateDomainTwoConfiguredCluster() throws Exception {
+
+    String testMethodName = new Object() {}.getClass().getEnclosingMethod().getName();
+    logTestBegin(testMethodName);
+
+    logger.info("Creating Operator & waiting for the script to complete execution");
+    // create operator1
+    if (operator1 == null) {
+      operator1 = TestUtils.createOperator(OPERATOR1_YAML);
+    }
+    Domain domain = null;
+    boolean testCompletedSuccessfully = false;
+    try {
+      Map<String, Object> domainMap = TestUtils.loadYaml(DOMAINONPV_WLST_YAML);
+      domainMap.put("domainUID", DOMAINUID);
+      domainMap.put("clusterType", "CONFIGURED");
+      domainMap.put(
+          "createDomainPyScript",
+          "integration-tests/src/test/resources/domain-home-on-pv/"
+              + TWO_CONFIGURED_CLUSTER_SCRIPT);
+      addCluster2ToDomainTemplate();
+      domain = TestUtils.createDomain(domainMap);
+      domain.verifyDomainCreated();
+      verifyServersStatus(domain);
+      testBasicUseCases(domain);
+      if (!SMOKETEST) {
+        domain.testWlsLivenessProbe();
+      }
+      testCompletedSuccessfully = true;
+    } finally {
+      if (domain != null && !SMOKETEST && (JENKINS || testCompletedSuccessfully)) {
+        domain.destroy();
+      }
+      restoreDomainTemplate();
+    }
+    logger.info("SUCCESS - " + testMethodName);
+  }
+
+  /**
+   * Create 2 clusters(configured and dynamic) in a domain each having 2 managed servers. Verify the
+   * managed servers are running and verify the basic use cases.
+   *
+   * @throws Exception
+   */
+  @Test
+  public void testCreateDomainTwoMixedCluster() throws Exception {
+    String DOMAINUID = "twomixedclusterdomain";
+    String testMethodName = new Object() {}.getClass().getEnclosingMethod().getName();
+    logTestBegin(testMethodName);
+    String template =
+        BaseTest.getProjectRoot() + "/kubernetes/samples/scripts/common/domain-template.yaml";
+    logger.info("Creating Operator & waiting for the script to complete execution");
+    if (operator1 == null) {
+      operator1 = TestUtils.createOperator(OPERATOR1_YAML);
+    }
+    Domain domain = null;
+    boolean testCompletedSuccessfully = false;
+    try {
+      Map<String, Object> domainMap = TestUtils.loadYaml(DOMAINONPV_WLST_YAML);
+      domainMap.put("domainUID", DOMAINUID);
+      domainMap.put(
+          "createDomainPyScript",
+          "integration-tests/src/test/resources/domain-home-on-pv/" + TWO_MIXED_CLUSTER_SCRIPT);
+      addCluster2ToDomainTemplate();
+      domain = TestUtils.createDomain(domainMap);
+      domain.verifyDomainCreated();
+      verifyServersStatus(domain);
+      testBasicUseCases(domain);
+      if (!SMOKETEST) {
+        domain.testWlsLivenessProbe();
+      }
+      testCompletedSuccessfully = true;
+    } finally {
+      if (domain != null && !SMOKETEST && (JENKINS || testCompletedSuccessfully)) {
+        domain.destroy();
+      }
+      restoreDomainTemplate();
+    }
+    logger.info("SUCCESS - " + testMethodName);
+  }
+
+  /**
+   * Append a second cluster to the domain template
+   *
+   * @throws IOException when append fails
+   */
+  private void addCluster2ToDomainTemplate() throws IOException {
+    String add =
+        "  - clusterName: %CLUSTER_NAME%-2\n"
+            + "    serverStartState: \"RUNNING\"\n"
+            + "    replicas: %INITIAL_MANAGED_SERVER_REPLICAS%\n";
+    logger.info("Making a backup of the domain template file:" + template);
+    if (!Files.exists(Paths.get(template + ".org"))) {
+      Files.copy(Paths.get(template), Paths.get(template + ".org"));
+    }
+    Files.write(Paths.get(template), add.getBytes(), StandardOpenOption.APPEND);
+    byte[] readAllBytes = Files.readAllBytes(Paths.get(template));
+    logger.info(new String(readAllBytes, StandardCharsets.UTF_8));
+  }
+
+  /**
+   * Restore the domain template to original state when test is finished
+   *
+   * @throws IOException
+   */
+  private void restoreDomainTemplate() throws IOException {
+    Files.copy(
+        Paths.get(template + ".org"), Paths.get(template), StandardCopyOption.REPLACE_EXISTING);
+    Files.delete(Paths.get(template + ".org"));
+  }
+
+  /**
+   * Verifies all of the servers in the cluster are in Running status
+   *
+   * @param Domain
+   */
+  private void verifyServersStatus(Domain domain) {
+    K8sTestUtils testUtil = new K8sTestUtils();
+    String domain1LabelSelector = String.format("weblogic.domainUID in (%s)", DOMAINUID);
+    String namespace = domain.getDomainNS();
+    String pods[] = {
+      DOMAINUID + "-" + domain.getAdminServerName(),
+      DOMAINUID + "-managed-server",
+      DOMAINUID + "-managed-server1",
+      DOMAINUID + "-managed-server2",
+      DOMAINUID + "-new-managed-server1",
+      DOMAINUID + "-new-managed-server2",
+    };
+    for (String pod : pods) {
+      assertTrue(
+          pod + " Pod not running", testUtil.isPodRunning(namespace, domain1LabelSelector, pod));
+    }
+  }
+}

integration-tests/src/test/java/oracle/kubernetes/operator/ITOperator.java

@@ -502,9 +502,4 @@ private Domain testAdvancedUseCasesForADomain(Operator operator, Domain domain)
     }
     return domain;
   }
-
-  private void testBasicUseCases(Domain domain) throws Exception {
-    testAdminT3Channel(domain);
-    testAdminServerExternalService(domain);
-  }
 }

integration-tests/src/test/java/oracle/kubernetes/operator/ITServerDiscovery.java

@@ -89,13 +89,15 @@ public static void staticPrepare() throws Exception {
    */
   @AfterClass
   public static void staticUnPrepare() throws Exception {
-    logger.info("+++++++++++++++++++++++++++++++++---------------------------------+");
-    logger.info("BEGIN");
-    logger.info("Run once, release cluster lease");
+    if (!QUICKTEST) {
+      logger.info("++++++++++++++++++++++++++++++++++");
+      logger.info("BEGIN");
+      logger.info("Run once, release cluster lease");
 
-    tearDown(new Object() {}.getClass().getEnclosingClass().getSimpleName());
+      tearDown(new Object() {}.getClass().getEnclosingClass().getSimpleName());
 
-    logger.info("SUCCESS");
+      logger.info("SUCCESS");
+    }
   }
 
   /**