Add situational configuration override tests with running domain (#1168)

* update usecases doc for init container tests

* adding situational config override after domain startup

* fix the create jdbc resource

* fix DOMAINUID

* fix

* make the domain as development

* add the test in SitConfigTests.java

* fix

* fix the target

* remove encryption

* remove connections on reserve check and change db name

* fix the descriptor

* added config override usecase

* fix the expected condition

* add the tests in USECASES.md

* restart using patch instead of recreating the domain

* fix the patching

* Verify the server pods are deleted

* add assertion for verifying the startup/shutdown class length

* Add documentation

* add test for new secrets

* Remove the create jdbc resource

* Modify the config.xml as well

* Modify secret in domainyaml

* Add overwrite option

* Add configoverridefiles

* Fix the destination location

* Apply the changes to domain.yaml

* FIx

* Make the db name a random character

* fix

Author: sankarpn

integration-tests/USECASES.MD

@@ -100,6 +100,8 @@ Configuration Overrides Usecases
 | JDBC Resource Override | Override JDBC connection pool properties; `initialCapacity`, `maxCapacity`, `test-connections-on-reserve`, `connection-harvest-max-count`, `inactive-connection-timeout-seconds`. Override the JDBC driver parameters like data source `URL`, `DB` `user` and `password` using kubernetes secret. The test verifies the overridden functionality datasource `URL`, `user`, `password` by getting the data source connection and running DDL statement it is connected to. |
 | JMS Resource Override | Override UniformDistributedTopic Delivery Failure Parameters, `redelivery-limit` and `expiration-policy`. The JMX test client verifies the serverConfig MBean tree for the expected delivery failure parameters, `redelivery-limit` and `expiration-policy`. |
 | WLDF Resource Override | Override `wldf-instrumentation-monitor` and `harvester` in a diagnostics module. The test client verifies the new instrumentation monitors/harvesters set by getting the WLDF resource from serverConfig tree with expected values.  |
+| Configuration override with running domain | Override the administration server with Startup and Shutdown class by editing the configmap and recreating the domain CRD. The override is verified by JMX client connecting to the serverConfig MBean tree and the values are checked against the expected values. |
+| JDBC Resource Override with running domain | Override non dynamic JDBC connection pool properties; `ignore-in-use-connections`, `login-delay-Seconds`, `connection-cache-type`, `global-transactions-protocol`  by editing the configmap and recreating the domain CRD. The test only verifies the expected values against the config tree |
 
 | Session Migration | Use Case |
 | --- | --- |
@@ -151,11 +153,12 @@ Configuration Overrides Usecases
 
 | Init Container | Use Case |
 | --- | --- |
-| Add initContainers to domain | Add a initContainers object to spec level and verify the init containers are created for weblogic server pods prior to starting it and runs to completion and then weblogic pod are started |
-| Add initContainers to adminServer | Add a initContainers object to adminServer level and verify the init container is created for administration server weblogic server pod prior to starting it and runs to completion and then weblogic pod is started |
-| Add initContainers to Clusters | Add a initContainers object to Clusters level and verify the init containers are created for weblogic server pods prior to starting the clusters and runs to completion and then weblogic pod are started |
-| Add initContainers to managedServers | Add a initContainers object to managed server level and verify the init container is created for managed server weblogic server pod prior to starting it and runs to completion and then weblogic pod is started |
-| Add bad initContainers to domain | Add a bad initContainers object to domain and verify the init container run fails and no weblogic pod is started |
-| Add multiple initContainers to domain | Add multiple initContainers object to domain level and verify all of the init container are run before weblogic server pod are started |
-| Add initContainers with different names at different level | Add a multiple initContainers object at domain level and server level and verify all of the init containers are run before weblogic server pods are started |
-| Add initContainers with same names at different level | Add a multiple initContainers object at domain level and server level and verify only the server level init containers are run before weblogic server pods are started |
+| Add initContainers to domain | Add a initContainers object to spec level and verify the init containers are created for Weblogic server pods prior to starting it and runs to completion and then Weblogic pod are started |
+| Add initContainers to adminServer | Add a initContainers object to adminServer level and verify the init container is created for administration server Weblogic server pod prior to starting it and runs to completion and then Weblogic pod is started |
+| Add initContainers to Clusters | Add a initContainers object to Clusters level and verify the init containers are created for Weblogic server pods prior to starting the clusters and runs to completion and then Weblogic pod are started |
+| Add initContainers to managedServers | Add a initContainers object to managed server level and verify the init container is created for managed server Weblogic server pod prior to starting it and runs to completion and then Weblogic pod is started |
+| Add bad initContainers to domain | Add a bad initContainers object to domain and verify the init container run fails and no Weblogic pod is started |
+| Add multiple initContainers to domain | Add multiple initContainers object to domain level and verify all of the init container are run before Weblogic server pod are started |
+| Add initContainers with different names at different level | Add a multiple initContainers object at domain level and server level and verify all of the init containers are run before Weblogic server pods are started |
+| Add initContainers with same names at different level | Add a multiple initContainers object at domain level and server level and verify only the server level init containers are run before Weblogic server pods are started |
+

integration-tests/src/test/java/oracle/kubernetes/operator/ItSitConfig.java

@@ -1,7 +1,6 @@
 // Copyright 2019, Oracle Corporation and/or its affiliates.  All rights reserved.
 // Licensed under the Universal Permissive License v 1.0 as shown at
 // http://oss.oracle.com/licenses/upl.
-
 package oracle.kubernetes.operator;
 
 import java.io.IOException;
@@ -10,9 +9,10 @@
 import java.nio.file.Files;
 import java.nio.file.Path;
 import java.nio.file.Paths;
+import java.nio.file.StandardCopyOption;
+import java.nio.file.StandardOpenOption;
 import java.util.Map;
 import java.util.logging.Level;
-
 import oracle.kubernetes.operator.utils.Domain;
 import oracle.kubernetes.operator.utils.ExecResult;
 import oracle.kubernetes.operator.utils.Operator;
@@ -44,6 +44,8 @@ public class ItSitConfig extends BaseTest {
   private static String mysqltmpDir = "";
   private static String configOverrideDir = "";
   private static String mysqlYamlFile = "";
+  private static String domainYaml;
+  private static String JDBC_RES_SCRIPT;
 
   /**
    * This method gets called only once before any of the test methods are executed. It does the
@@ -82,6 +84,11 @@ public static void staticPrepare() throws Exception {
       // create weblogic domain with configOverrides
       domain = createSitConfigDomain();
       Assert.assertNotNull(domain);
+      domainYaml =
+          BaseTest.getUserProjectsDir()
+              + "/weblogic-domains/"
+              + domain.getDomainUid()
+              + "/domain.yaml";
       // copy the jmx test client file the administratioin server weblogic server pod
       ADMINPODNAME = domain.getDomainUid() + "-" + domain.getAdminServerName();
       TestUtils.copyFileViaCat(
@@ -96,6 +103,7 @@ public static void staticPrepare() throws Exception {
           domain.getDomainNs());
       KUBE_EXEC_CMD =
           "kubectl -n " + domain.getDomainNs() + "  exec -it " + ADMINPODNAME + "  -- bash -c";
+      JDBC_RES_SCRIPT = TEST_RES_DIR + "/sitconfig/scripts/create-jdbc-resource.py";
     }
   }
 
@@ -217,6 +225,7 @@ public void testCustomSitConfigOverridesForDomain() throws Exception {
     boolean testCompletedSuccessfully = false;
     String testMethod = new Object() {}.getClass().getEnclosingMethod().getName();
     logTestBegin(testMethod);
+    transferTests();
     ExecResult result =
         TestUtils.exec(
             KUBE_EXEC_CMD
@@ -247,6 +256,7 @@ public void testCustomSitConfigOverridesForDomainMS() throws Exception {
     boolean testCompletedSuccessfully = false;
     String testMethod = new Object() {}.getClass().getEnclosingMethod().getName();
     logTestBegin(testMethod);
+    transferTests();
     ExecResult result =
         TestUtils.exec(
             KUBE_EXEC_CMD
@@ -282,6 +292,7 @@ public void testCustomSitConfigOverridesForJdbc() throws Exception {
     boolean testCompletedSuccessfully = false;
     String testMethod = new Object() {}.getClass().getEnclosingMethod().getName();
     logTestBegin(testMethod);
+    transferTests();
     ExecResult result =
         TestUtils.exec(
             KUBE_EXEC_CMD
@@ -315,6 +326,7 @@ public void testCustomSitConfigOverridesForJms() throws Exception {
     boolean testCompletedSuccessfully = false;
     String testMethod = new Object() {}.getClass().getEnclosingMethod().getName();
     logTestBegin(testMethod);
+    transferTests();
     ExecResult result =
         TestUtils.exec(
             KUBE_EXEC_CMD
@@ -348,6 +360,44 @@ public void testCustomSitConfigOverridesForWldf() throws Exception {
     boolean testCompletedSuccessfully = false;
     String testMethod = new Object() {}.getClass().getEnclosingMethod().getName();
     logTestBegin(testMethod);
+    transferTests();
+    ExecResult result =
+        TestUtils.exec(
+            KUBE_EXEC_CMD
+                + " 'sh runSitConfigTests.sh "
+                + fqdn
+                + " "
+                + T3CHANNELPORT
+                + " weblogic welcome1 "
+                + testMethod
+                + "'");
+    assertResult(result);
+    testCompletedSuccessfully = true;
+    logger.log(Level.INFO, "SUCCESS - {0}", testMethod);
+  }
+
+  /**
+   * Test to verify the configuration override after a domain is up and running. Modifies the
+   * existing config.xml entries to add startup and shutdown classes verifies those are overridden
+   * when domain is restarted.
+   *
+   * @throws Exception when assertions fail.
+   */
+  @Test
+  public void testConfigOverrideAfterDomainStartup() throws Exception {
+    Assume.assumeFalse(QUICKTEST);
+    boolean testCompletedSuccessfully = false;
+    String testMethod = new Object() {}.getClass().getEnclosingMethod().getName();
+    logTestBegin(testMethod);
+    // recreate the map with new situational config files
+    String srcDir = TEST_RES_DIR + "/sitconfig/configoverrides";
+    String dstDir = configOverrideDir;
+    Files.copy(
+        Paths.get(srcDir, "config_1.xml"),
+        Paths.get(dstDir, "config.xml"),
+        StandardCopyOption.REPLACE_EXISTING);
+    recreateCRDWithNewConfigMap();
+    transferTests();
     ExecResult result =
         TestUtils.exec(
             KUBE_EXEC_CMD
@@ -363,6 +413,183 @@ public void testCustomSitConfigOverridesForWldf() throws Exception {
     logger.log(Level.INFO, "SUCCESS - {0}", testMethod);
   }
 
+  /**
+   * This test covers the overriding of JDBC system resource after a domain is up and running. It
+   * creates a datasource , recreates the K8S configmap with updated JDBC descriptor and verifies
+   * the new overridden values with restart of the WLS pods
+   *
+   * @throws Exception when assertions fail.
+   */
+  @Test
+  public void testOverrideJDBCResourceAfterDomainStart() throws Exception {
+    Assume.assumeFalse(QUICKTEST);
+    boolean testCompletedSuccessfully = false;
+    String testMethod = new Object() {}.getClass().getEnclosingMethod().getName();
+    logTestBegin(testMethod);
+    createJdbcResource();
+    // recreate the map with new situational config files
+    String srcDir = TEST_RES_DIR + "/sitconfig/configoverrides";
+    String dstDir = configOverrideDir;
+    Files.copy(
+        Paths.get(srcDir, "jdbc-JdbcTestDataSource-1.xml"),
+        Paths.get(dstDir, "jdbc-JdbcTestDataSource-1.xml"),
+        StandardCopyOption.REPLACE_EXISTING);
+    recreateCRDWithNewConfigMap();
+    transferTests();
+    ExecResult result =
+        TestUtils.exec(
+            KUBE_EXEC_CMD
+                + " 'sh runSitConfigTests.sh "
+                + fqdn
+                + " "
+                + T3CHANNELPORT
+                + " weblogic welcome1 "
+                + testMethod
+                + "'");
+    assertResult(result);
+    testCompletedSuccessfully = true;
+    logger.log(Level.INFO, "SUCCESS - {0}", testMethod);
+  }
+
+  /**
+   * This test covers the overriding of JDBC system resource with new kubernetes secret name for
+   * dbusername and dbpassword
+   *
+   * @throws Exception when assertions fail.
+   */
+  @Test
+  public void testOverrideJDBCResourceWithNewSecret() throws Exception {
+    Assume.assumeFalse(QUICKTEST);
+    boolean testCompletedSuccessfully = false;
+    String testMethod = new Object() {}.getClass().getEnclosingMethod().getName();
+    logTestBegin(testMethod);
+    // recreate the map with new situational config files
+    String files[] = {"config.xml", "jdbc-JdbcTestDataSource-0.xml"};
+    String secretName = "test-secrets-new";
+    for (String file : files) {
+      Path path = Paths.get(sitconfigTmpDir, "configoverridefiles", file);
+      String content = new String(Files.readAllBytes(path), StandardCharsets.UTF_8);
+      content = content.replaceAll("test-secrets", secretName);
+      if (getWeblogicImageTag().contains(PS3_TAG)) {
+        content = content.replaceAll(JDBC_DRIVER_NEW, JDBC_DRIVER_OLD);
+      }
+      Files.write(
+          Paths.get(sitconfigTmpDir, "configoverridefiles", file),
+          content.getBytes(StandardCharsets.UTF_8),
+          StandardOpenOption.TRUNCATE_EXISTING);
+    }
+    String content = new String(Files.readAllBytes(Paths.get(domainYaml)), StandardCharsets.UTF_8);
+    content = content.replaceAll("test-secrets", secretName);
+    Files.write(
+        Paths.get(domainYaml),
+        content.getBytes(StandardCharsets.UTF_8),
+        StandardOpenOption.TRUNCATE_EXISTING);
+
+    TestUtils.exec("kubectl delete secret " + domain.getDomainUid() + "-test-secrets", true);
+    createNewSecret(secretName);
+    TestUtils.exec("kubectl apply -f " + domainYaml, true);
+    recreateCRDWithNewConfigMap();
+    transferTests();
+    ExecResult result =
+        TestUtils.exec(
+            KUBE_EXEC_CMD
+                + " 'sh runSitConfigTests.sh "
+                + fqdn
+                + " "
+                + T3CHANNELPORT
+                + " weblogic welcome1 "
+                + testMethod
+                + " "
+                + JDBC_URL
+                + "'");
+    assertResult(result);
+    testCompletedSuccessfully = true;
+    logger.log(Level.INFO, "SUCCESS - {0}", testMethod);
+  }
+
+  /**
+   * Create a JDBC system resource in domain
+   *
+   * @throws Exception JDBC resource creation fails
+   */
+  private void createJdbcResource() throws Exception {
+    Path path = Paths.get(JDBC_RES_SCRIPT);
+    String content = new String(Files.readAllBytes(path), StandardCharsets.UTF_8);
+    content = content.replaceAll("JDBC_URL", JDBC_URL);
+    content = content.replaceAll("DOMAINUID", DOMAINUID);
+    if (getWeblogicImageTag().contains(PS3_TAG)) {
+      content = content.replaceAll(JDBC_DRIVER_NEW, JDBC_DRIVER_OLD);
+    }
+    Files.write(
+        Paths.get(sitconfigTmpDir, "create-jdbc-resource.py"),
+        content.getBytes(StandardCharsets.UTF_8));
+    TestUtils.copyFileViaCat(
+        Paths.get(sitconfigTmpDir, "create-jdbc-resource.py").toString(),
+        "create-jdbc-resource.py",
+        ADMINPODNAME,
+        domain.getDomainNs());
+    TestUtils.exec(KUBE_EXEC_CMD + " 'wlst.sh create-jdbc-resource.py'", true);
+  }
+
+  /**
+   * Update the configOverrides configmap and restart WLS pods.
+   *
+   * @throws Exception when pods restart fail
+   */
+  private void recreateCRDWithNewConfigMap() throws Exception {
+    int clusterReplicas =
+        TestUtils.getClusterReplicas(DOMAINUID, domain.getClusterName(), domain.getDomainNs());
+
+    String patchStr = "'{\"spec\":{\"serverStartPolicy\":\"NEVER\"}}'";
+    TestUtils.kubectlpatch(DOMAINUID, domain.getDomainNs(), patchStr);
+    domain.verifyServerPodsDeleted(clusterReplicas);
+
+    String cmd =
+        "kubectl create configmap "
+            + DOMAINUID
+            + "-sitconfigcm --from-file="
+            + configOverrideDir
+            + " -o yaml --dry-run | kubectl replace -f -";
+    TestUtils.exec(cmd, true);
+
+    patchStr = "'{\"spec\":{\"serverStartPolicy\":\"IF_NEEDED\"}}'";
+    TestUtils.kubectlpatch(DOMAINUID, domain.getDomainNs(), patchStr);
+    domain.verifyDomainCreated();
+  }
+
+  private void createNewSecret(String secretName) throws Exception {
+    String cmd =
+        "kubectl -n "
+            + domain.getDomainNs()
+            + " create secret generic "
+            + domain.getDomainUid()
+            + "-"
+            + secretName
+            + " --from-literal=hostname="
+            + TestUtils.getHostName()
+            + " --from-literal=dbusername=root"
+            + " --from-literal=dbpassword=root123";
+    TestUtils.exec(cmd, true);
+  }
+
+  /**
+   * Transfer the tests to run in WLS pods
+   *
+   * @throws Exception
+   */
+  private void transferTests() throws Exception {
+    TestUtils.copyFileViaCat(
+        TEST_RES_DIR + "sitconfig/java/SitConfigTests.java",
+        "SitConfigTests.java",
+        ADMINPODNAME,
+        domain.getDomainNs());
+    TestUtils.copyFileViaCat(
+        TEST_RES_DIR + "sitconfig/scripts/runSitConfigTests.sh",
+        "runSitConfigTests.sh",
+        ADMINPODNAME,
+        domain.getDomainNs());
+  }
+
   /**
    * Verifies the test run result doesn't contain any errors and exit status is 0.
    *

integration-tests/src/test/resources/sitconfig/configoverrides/config_1.xml

@@ -0,0 +1,36 @@
+<?xml version='1.0' encoding='UTF-8'?>
+<d:domain xmlns:d="http://xmlns.oracle.com/weblogic/domain" xmlns:f="http://xmlns.oracle.com/weblogic/domain-fragment" xmlns:s="http://xmlns.oracle.com/weblogic/situational-config" >
+    <d:server>
+        <d:name>admin-server</d:name>
+        <d:connect-timeout f:combine-mode="add">120</d:connect-timeout>
+        <d:restart-max f:combine-mode="add">5</d:restart-max>
+        <d:max-message-size f:combine-mode="add">78787878</d:max-message-size>
+        <d:server-debug f:combine-mode="add">
+            <d:debug-server-life-cycle f:combine-mode="add">true</d:debug-server-life-cycle>
+            <d:debug-jmx-core f:combine-mode="replace">true</d:debug-jmx-core>
+        </d:server-debug>
+        <d:network-access-point>
+            <d:name>T3Channel</d:name>
+            <d:public-address f:combine-mode="replace">${secret:customsitconfigdomain-test-secrets.hostname}</d:public-address>
+        </d:network-access-point>
+    </d:server>
+    <d:server-template>
+        <d:name>cluster-1-template</d:name>
+        <d:max-message-size f:combine-mode="add">77777777</d:max-message-size>
+    </d:server-template>
+    <d:shutdown-class f:combine-mode="add">
+        <d:name>ShutdownClass-0</d:name>
+        <d:target>admin-server,cluster-1</d:target>
+        <d:deployment-order>6</d:deployment-order>
+        <d:class-name>AddedShutdownClassOne</d:class-name>
+    </d:shutdown-class>
+    <d:startup-class f:combine-mode="add">
+        <d:name>StartupClass-0</d:name>
+        <d:target>admin-server,cluster-1</d:target>
+        <d:deployment-order>5</d:deployment-order>
+        <d:class-name>AddedStartupClassOne</d:class-name>
+        <d:failure-is-fatal>false</d:failure-is-fatal>
+        <d:load-before-app-deployments>true</d:load-before-app-deployments>
+        <d:load-before-app-activation>true</d:load-before-app-activation>
+    </d:startup-class>
+</d:domain>